Skip to content

Add native bridge APIs for JASP replay#7

Merged
boutinb merged 31 commits into
jasp-stats:masterfrom
FBartos:bridge/native-jasp-syntax-api
May 15, 2026
Merged

Add native bridge APIs for JASP replay#7
boutinb merged 31 commits into
jasp-stats:masterfrom
FBartos:bridge/native-jasp-syntax-api

Conversation

@FBartos
Copy link
Copy Markdown
Contributor

@FBartos FBartos commented May 14, 2026

Summary

  • add high-level native-backed helpers for module description, QML resolution, option preparation, dataset reads, and result decoding
  • add .jasp archive helpers for saved options and dataset extraction, including status-returning native calls and isolated subprocess reads
  • add bridge lifecycle helpers, Windows configure/install provenance, and SyntaxInterface symbol checks
  • add Desktop-created .jasp fixtures plus tests for saved/runtime option contracts, dataset extraction, module metadata, and native export coverage

Why

This moves QML parsing, dataset loading, column encoding/decoding, and .jasp archive semantics out of jaspTools and into the lower native bridge package. jaspTools can then stay focused on developer-facing orchestration instead of duplicating Desktop/native behavior.

Related PRs

Verification

  • Rscript -e ".libPaths(c('C:/Users/fbartos/AppData/Local/Temp/jaspBridgeLib_codex_20260514_mergepass', .libPaths())); library(testthat); test_file('tests/testthat/test-dataset-helpers.R'); test_file('tests/testthat/test-jasp-file-options.R'); test_file('tests/testthat/test-desktop-jasp-contract.R'); test_file('tests/testthat/test-module-options.R')" with JASP_DESCRIPTIVES_MODULE=C:/JASP-Packages/jaspDescriptives
  • git diff --check HEAD~1..HEAD

vandenman and others added 21 commits April 17, 2026 10:04
Compile the downloaded jsoncpp sources, install the built package and helper DLLs into the package library, and copy compatible MinGW runtime DLLs from a local JASP installation when available.

Validated locally with a clean temp-library install and library load on Windows.
Treat missing runtime DLLs as optional under set -e and populate src/json from JASP_SOURCE_DIR so the review-identified configure paths build correctly.
Populate json sources in the non-Windows JASP_SOURCE_DIR path and simplify install.libs.R so the copied library set is clearer without changing behavior.
Add readDatasetFromJaspFile() as a public jaspSyntax entry point for materializing the first dataset in a saved .jasp file as an R data.frame.

The implementation validates inputs, reuses the existing jaspSyntax bridge load/read path, decodes column names, and normalizes bridge-returned factor columns so numeric-like categorical data comes back as numeric/integer while text-labeled categorical data comes back as character.

The public function runs the native dataset load/read path in a short-lived Rscript subprocess instead of in the caller session. This keeps the API stable on the current branch because repeated in-process reads were triggering native transaction unwind spam and stack-imbalance behavior after loading datasets from .jasp files.

This commit also adds Rd documentation for the new function and documents the current limitation that only dataSetIndex = 1L is supported by the bridge-backed implementation.
Wrap bridge-facing calls in syntaxfunctions.cpp so native exceptions and malformed bridge responses are turned into explicit R errors instead of being silently parsed or assumed valid.

This adds a small helper for exception-safe bridge calls and a JSON parsing helper that rejects null pointers and invalid JSON. The affected entry points are parseDescription(), loadDataSetFromJaspFile(), analysisOptionsFromJaspFile(), and getVariableNames().

The change does not alter the public API shape. It makes failures in the current bridge-backed implementation easier to diagnose and safer to surface in R.
@FBartos
Copy link
Copy Markdown
Contributor Author

FBartos commented May 14, 2026

Reproduced the jaspMixedModels crash reported from the jaspTools replay path and pushed 79258ee.

Root cause was in the bridge layer, not in jaspTools orchestration:

  • Source-module QML resolution was still going through the native parseDescription() path. For source checkouts like jaspMixedModels, that instantiates a DynamicModule just to find the QML file, and in this replay path it destabilized the subprocess before the real analysis could run.
  • readDatasetFromJaspFile() returned a plain data.frame. That meant jaspTools::extractDatasetFromJASPFile() lost the original .jasp archive as the source of truth, so loadAnalysisDataset() re-imported the data frame and had to infer Desktop column types. For Larks and Owls.jasp, Subject has 130 levels and is needed as a nominal random factor; the data-frame import path can classify it differently from the saved Desktop dataset.

The fix keeps the source of truth in jaspSyntax:

  • parseModuleDescription() now resolves source checkouts directly from Description.qml/DESCRIPTION, with native bridge fallback for installed/binary module layouts.
  • Datasets returned by readDatasetFromJaspFile() carry .jasp provenance attributes: archive path, dataset index, dimensions, and names.
  • loadAnalysisDataset() reuses loadDataSetFromJaspFile() when that provenance is intact, so real .jasp replay preserves Desktop dataset typing instead of re-importing a flattened data.frame.
  • Added unit coverage for the provenance path and updated the source-module metadata expectation to use the package DESCRIPTION version.

Verification on the local Windows checkout:

  • Reinstalled jaspSyntax from C:/JASP-Packages/jaspSyntax.
  • test-dataset-helpers.R: 75 passes.
  • jaspSyntax::readDefaultAnalysisOptions('C:/JASP-Packages/jaspMixedModels', 'MixedModelsGLMM') now returns normally with module jaspMixedModels, version 0.96.4, and inst/qml/MixedModelsGLMM.qml.
  • The exact jaspMixedModels reproduction now succeeds:
    • dataset <- jaspTools::extractDatasetFromJASPFile('examples/Larks and Owls.jasp') carries source C:/JASP-Packages/jaspMixedModels/examples/Larks and Owls.jasp.
    • jaspTools::runAnalysis('MixedModelsGLMM', dataset, opts) returns a result list with results,status,typeRequest,state instead of crashing.

One unrelated note: test-module-options.R still exits nonzero in the existing direct jaspSyntax::loadDataSet(data.frame(...)) native path. That is separate from this .jasp replay fix; the MixedModels replay no longer uses that path when the dataset came from a .jasp archive.

@FBartos
Copy link
Copy Markdown
Contributor Author

FBartos commented May 14, 2026

Senior follow-up pass pushed in 06117aa and b30247b.

What changed here:

  • Hardened Description.qml parsing so one-line Analysis { ...; func: "..." } entries are handled correctly. This covers real module styles such as jaspAnova without pushing module metadata parsing back into jaspTools.
  • Strengthened .jasp dataset provenance with archive signature plus data hash. This prevents same-shape/same-name mutated data frames from silently replaying the original archive dataset.
  • Kept strict native decoding focused on encoded bridge names only. Raw data-frame names like x, x.scale, or normal module column names now pass through unchanged instead of being sent to the native strict decoder.
  • Fixed source-checkout subprocess DLL ordering: explicit JASP_BUILD_DIR first, then the checkout DLL dirs, then inherited PATH. That preserves deliberate Desktop-build overrides while avoiding accidental resolution against unrelated global Qt DLLs.
  • Added failure diagnostics for source subprocess DLL resolution and included the child exit status when no result RDS is produced.

Verification run locally:

  • testthat::test_file("tests/testthat/test-dataset-helpers.R"): 85 pass, 0 fail.
  • testthat::test_file("tests/testthat/test-module-options.R"): 49 pass, 0 fail.
  • jaspSyntax::readDatasetFromJaspFile("C:/JASP-Packages/jaspMixedModels/examples/Larks and Owls.jasp", 1L) through the public subprocess path returns 260 x 9 with the expected column names.
  • Source-checkout subprocess replay still works when the child inherits a bad global Qt path first, as long as JASP_BUILD_DIR points at the patched Desktop build.
  • End-to-end jaspTools::runAnalysis("MixedModelsGLMM", dataset, opts) now completes with status = "complete" on the patched stack.

Related PRs for the full bridge:

@FBartos
Copy link
Copy Markdown
Contributor Author

FBartos commented May 14, 2026

Follow-up on the focused-test warning investigation:

  • Added shutdownNative() and call it from .onUnload() plus an onexit namespace finalizer. This pairs with Desktop's new terminal syntaxBridgeShutdown() so package/session teardown deletes the Qt engine/application state instead of leaving QThreadStorage noise at process exit.
  • Fixed the Windows runtime bundling order. configure.win already selected build/R-Interface/libR-InterfaceNoRInside.dll explicitly, but the later transitive dependency bundling searched the Desktop build root first and could overwrite it with a stale root DLL. The R-Interface build subdir now wins before the root build dir.

That stale-DLL overwrite was important: it could mix a fresh SyntaxInterface.dll with an old libR-InterfaceNoRInside.dll, producing misleading stack-imbalance diagnostics. With the corrected binary pairing and the Desktop-side parse-only startup changes, the focused warning scans are clean.

Validation with a fresh temporary-library install:

  • install confirmed libR-InterfaceNoRInside.dll copied from C:/JASP-Packages/jasp-desktop/build/R-Interface
  • test-module-options.R: 49 pass
  • test-dataset-helpers.R: 85 pass
  • full jaspSyntax/tests/testthat: 256 pass, 1 skip
  • real jaspMixedModels QML parse for MixedModelsGLMM: 76 options returned

No GridLayout, qml:, QThreadStorage, QObject, Timers cannot, or stack imbalance patterns remained in the captured logs.

FBartos and others added 6 commits May 14, 2026 22:40
On macOS, objdump -T exits with code 0 for Mach-O files but emits only
a warning ("this operation is not currently supported for this file
format") with no actual symbol data. The fallback to -t never fires,
so the export check reports all symbols as missing.

Add an explicit *.dylib case that calls nm -g directly, which works
correctly on macOS for reading exported symbols.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When no Qt dirs are in PATH, lapply returns an empty list and
unlist(list()) returns NULL (not character(0)). normalizePath(NULL)
then calls path.expand(NULL) which fails with "invalid 'path'
argument".

Wrap all three unlist(lapply(...)) calls in as.character() so the
result is always a character vector, even when lapply produces an
empty list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boutinb boutinb merged commit 0085f61 into jasp-stats:master May 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants