Sporadic SIGSEGV in macOS [scans] workflow in CI

[mbutrovich]

I've seen this a few different times now in the macos [scans] workflow in CI. Also reported here: #5014 (comment).

Looks like:

ParquetReadFromFakeHadoopFsSuite:
- native scan on fake fs (530 milliseconds)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=22862, tid=59915
#
# JRE version: OpenJDK Runtime Environment Zulu17.68+17-CA (17.0.20+8) (build 17.0.20+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.68+17-CA (17.0.20+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  0x0000000000000000
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/runner/work/datafusion-comet/datafusion-comet/spark/hs_err_pid22862.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#
ParquetTimestampLtzAsNtzSuite:

Recent failure: https://github.com/apache/datafusion-comet/actions/runs/30050199441/job/89355580948?pr=5014

[peterxcli]

Issue #5023 is an upstream libhdfs JNIEnv use-after-free, not a Parquet failure or a regression from #5014.

Evidence

• I downloaded the failing job’s hs_err_pid22862.log and exact native-lib-macos artifact, then symbolized it:
  • pc = 0
  • lr = hdfsThreadDestructor + 80
  • x8 = 0
• Disassembly shows hdfsThreadDestructor loads the JNI GetJavaVM function pointer into x8, then executes blr x8. The stale JNIEnv contains a null function entry, producing the jump to address zero.
• The vulnerable call is exactly (*env)->GetJavaVM. Its null checks cannot detect a freed-but-non-null JNIEnv.
• This precisely matches Apache HDFS-16021: heap-use-after-free in hdfsThreadDestructorand the earlier HDFS-14488. HDFS-16021 remains open without a fix version, and even current Hadoop trunk still contains the vulnerable destructor implementation.

The Comet path is:

fake:// → libhdfs scheme → OpenDAL HDFS → hdrs → hdfs-sys 0.3.0 → vendored libhdfs

The test explicitly registers fake as a libhdfs scheme in ParquetReadFromFakeHadoopFsSuite.scala. Native routing occurs in is_hdfs_scheme and create_hdfs_object_store. macOS explicitly enables vendored hdrs in Cargo.toml, while Cargo.lock pins hdfs-sys 0.3.0.

Recommended fix

Carry or upstream the HDFS-16021 patch: record whether libhdfs itself attached the thread, and only call GetJavaVM/DetachCurrentThread when libhdfs owns that attachment. No released hdfs-sys upgrade currently includes it.

Assist by LLM