fix: preserve exec permissions on jspawnhelper in bundled JDK ([#1537…

[ZanDev32]

Fix: 3D/OpenGL sketches crash with UnsatisfiedLinkError on Linux — jspawnhelper missing exec permission

Closes #1537

Summary

3D/OpenGL sketches (P3D, P2D, OBJ-loading, anything using JOGL) crashed on launch with UnsatisfiedLinkError. The cause was lib/jspawnhelper losing its executable permission during the includeJdk Gradle copy task. Without the exec bit, the bundled JDK could not fork+exec child processes, which broke JOGL's temp-directory validation and prevented native libraries from loading.

The crash

Warning: Caught Exception while retrieving executable temp base directory:
java.io.IOException: Could not determine a temporary executable directory
    at com.jogamp.common.util.IOUtil.getTempDir(IOUtil.java:1401)
    at com.jogamp.common.util.cache.TempFileCache.<clinit>(TempFileCache.java:84)
    at com.jogamp.common.os.Platform$1.run(Platform.java:313)
java.lang.UnsatisfiedLinkError: 'boolean jogamp.common.jvm.JVMUtil.initialize(java.nio.ByteBuffer)'
    at jogamp.common.jvm.JVMUtil.initialize(Native Method)
    at com.jogamp.common.os.Platform.<clinit>(Platform.java:290)
    at com.jogamp.opengl.GLProfile.<clinit>(GLProfile.java:151)
    at processing.opengl.PSurfaceJOGL.initGL(PSurfaceJOGL.java:204)
A library used by this sketch relies on native code that is not available.

2D sketches (Java2D renderer) ran fine because they never call Runtime.exec().

Root cause

jspawnhelper is a small ELF binary at lib/jspawnhelper that JDK 17's ProcessImpl execs (via vfork + execve) to spawn child processes. The includeJdk Gradle task copied the JDK into resources/jdk/ but ran its file-permission loop at configuration time, before the copy happened, on an empty directory. The loop set writable and readable but never executable. Since the directory was empty at config time, no files were affected. The Gradle Copy task's dirPermissions spec only applied to directories, not files. Result: jspawnhelper (and jexec, and everything in bin/) landed with mode 0644 instead of 0755.

When the sketch JVM tried to exec jspawnhelper, the kernel returned EACCES (error=13, Permission denied). This broke every Runtime.exec() call in the sketch process. JOGL's IOUtil.testDirExec() writes a test script to a temp directory and actually execve()s it to validate that the temp dir allows execution. With jspawnhelper non-executable, that test failed for every candidate directory, so TempJarCache never initialized, native libraries were never extracted, and the sketch crashed with UnsatisfiedLinkError.

The same issue affected the IDE runtime at lib/runtime/lib/jspawnhelper.

How I found it

I used Kimi K2.7 Code to help find the root cause. This took several wrong turns before the actual cause surfaced. After I tried everything, i finally decided to just compare the Build-In JDK with a fresh Temurin 17 from Adoptium file by file. It turn out that there are some file that have difference permission:

File	Fresh (working)	Bundled (broken)
lib/jspawnhelper	0755	0644
lib/jexec	0755	0644
lib/server/classes.jsa	0444	0644

jspawnhelper was the critical one. chmod 755 jspawnhelper on the bundled JDK immediately fixed Runtime.exec().

Why includeJdk didn't preserve the exec bit

The original code:

tasks.register<Copy>("includeJdk") {
    from(Jvm.current().javaHome.absolutePath)
    destinationDir = composeResources("jdk").get().asFile

    dirPermissions { unix("rwx------") }
    fileTree(destinationDir).files.forEach { file ->   // runs at CONFIG time, on EMPTY dir
        file.setWritable(true, false)
        file.setReadable(true, false)                 // no setExecutable()
    }
}

The fileTree(destinationDir).files.forEach block ran at Gradle configuration time, before the Copy task executed. destinationDir was empty, so the loop processed zero files. Even if it had run after the copy, it only set writable and readable, never executable. The dirPermissions spec only applied to directories.

The fix

app/build.gradle.kts — includeJdk task (root cause)

Moved the permission loop into a doLast block (runs after the copy) and re-applied the exec bit from the source JDK for files that should be executable:

doLast {
    fileTree(destinationDir).files.forEach { file ->
        val sourceFile = File(Jvm.current().javaHome.absolutePath,
            file.relativeTo(destinationDir).path)
        if (sourceFile.canExecute()) {
            file.setExecutable(true, false)
        }
        file.setWritable(true, false)
        file.setReadable(true, false)
    }
}

app/build.gradle.kts — setExecutablePermissions task (belt and suspenders)

Added **/runtime/**/bin/** and **/runtime/**/lib/** patterns so the IDE runtime's jspawnhelper is also covered:

include("**/runtime/**/bin/**")
include("**/runtime/**/lib/**")

app/src/processing/app/Platform.java — JAVA_HOME / JDK_HOME support (Extra)

Added env var checks inside the existing "If the JDK is set in the environment" fallback block, before the java.home system property fallback. This lets users override the sketch JDK when the bundled resources/jdk is absent (e.g., portable installs without a bundled JDK):

for (String envKey : new String[] { "JAVA_HOME", "JDK_HOME" }) {
    String envPath = System.getenv(envKey);
    if (envPath != null && !envPath.isEmpty()) {
        File envHome = new File(envPath);
        if (new File(envHome, "bin/java" + (Platform.isWindows() ? ".exe" : "")).canExecute()) {
            return envHome;
        }
    }
}

The bundled resources/jdk still takes priority. JAVA_HOME/JDK_HOME is only checked when the bundle is missing.

Tests

Test sketch: a P3D project that loads an .obj file (Rubik's cube model with loadShape()).

Scenario	JAVA_HOME	Expected	Result
Normal usage (jspawnhelper fixed)	not set	Sketch runs	PASS — Finished. exit 0
JAVA_HOME = system JDK 21	set, valid	Sketch runs (bundle wins)	PASS — Finished. exit 0
JAVA_HOME = invalid path	set, invalid	Sketch runs (bundle wins)	PASS — Finished. exit 0
JDK_HOME = system JDK 21	set, valid	Sketch runs (bundle wins)	PASS — Finished. exit 0

All tests run with processing cli --sketch=<path> --run automatically using Kimi K2.7 Code under my watch . The X11Util.Display shutdown messages in the output confirm JOGL/OpenGL initialized and rendered before clean exit.

Before the fix (jspawnhelper 0644), the same sketch crashed with UnsatisfiedLinkError: JVMUtil.initialize. So this pr are tested, and verified solves the issue without side effects so far.

tbh this is just a small change covering 2 files changed, 29 insertions(+), and 4 deletions(-).

Environment

• Arch Linux, kernel 7.0.5-arch1-1-g14
• Processing 4.5.5 (rev 1433)
• Bundled sketch JDK: Temurin 17.0.19+10
• System JDK: OpenJDK 21.0.11
• JOGL / gluegen 2.6.0

[SableRaf]

Hi @ZanDev32 and thanks for the PR.

Before we review it further, could you please read our AI_USAGE_POLICY.md?

Also, in the issue (#1537), you mention using AI agents to identify the problem. Could you clarify what parts of the investigation, implementation, and testing were your own work, and what parts were assisted by AI?

Please write your response yourself rather than having an AI agent generate it. You may use an LLM for proofreading or translation, but we'd like to understand your own reasoning and contribution.

[ZanDev32]

Hi @ZanDev32 and thanks for the PR.

Before we review it further, could you please read our AI_USAGE_POLICY.md?

Also, in the issue, you mention using AI agents to identify the problem. Could you clarify what parts of the investigation, implementation, and testing were your own work, and what parts were assisted by AI?

Please write your response yourself rather than having an AI agent generate it. You may use an LLM for proofreading or translation, but we'd like to understand your own reasoning and contribution.

I already updated the original PR message and rebased to match the latest commit. All of the contents in that message have been reviewed by me and tested using an actual project, both with the graphical IDE and the CLI. At the end of the day, it's just a permission problem, nothing big. I also added a fix so the app uses the system Java instead of the built-in one (this was the original solution). Actually, this "extra" feature was already implemented before in PR #1199, but it doesn't work correctly anymore on the latest version of Processing (it doesn't work on either Windows 11 or Arch Linux with Wayland on kernel 7.0.5).