feat: Support shared scripts

[ferenc-csaky]

Key Changes

• Rename mounted dir from build to workspace inside Docker container
• Introduce new optional CLI option -r, --project-root to explicitly set relative project root to the mount point
• If project root is not set, it gets inferred from the given package file(s), and it will be the deepest common subdir (e.g. a/b/c/d/package.json, a/b/c/package.json, a/b/package.json ->a/b)
• Introduce shared config option under main (see details below)
• Add root prefix for import paths (IMPORT root.my.import.script;), which will always try to resolve the given path against the project root folder, even if the defining script is in a submodule
• Moved over SQRL script processing to a new SqrlPreprocessor, which now deals with the shared scripts, and the Mustache template resolution

Shared script concept

We can now setup packages that we plan to share between multiple projects. Most probably these will be data catalogs. It can be arranged as regular SQRL project, with the restriction the it requires a package.json named config in its root:

banking-shared
├── application_updates.jsonl
├── applications.jsonl
├── customers.jsonl
├── loan_types.jsonl
├── package.json
└── sources.sqrl

That package JSON can be minimal, and may contain an information about the project and define config variables if the defined shared script content has any of them:

{
  "version": "1",
  "information": {
    "description": "Common table definitions for banking test use cases"
  },
  "script": {
    "config": {
      "monitorInterval": "10s"
    }
  }
}

Then, we can add banking-shared to our regular SQRL project's package like:

{
...
"script": {
  "main": "loan.sqrl",
  "shared": {
    "banking-catalog": { // name for the imported script, will be used when referenced in SQRL
      "path": "../banking-shared" // path for the shared script's dir
      "config": { "monitorInterval": "5s" } // optional template variable overrides
    }
  },
  ...
}

The key name under shared will define the name of the shared module, which it can be imported by.
A shared script IMPORT will look like this:

# In loan.sqrl
IMPORT `banking-catalog`.sources;

# OR
IMPORT root.`banking-catalog`.sources;

Important

In most cases the root prefix is optional, except when the caller script is in a subdirectory, it has to be present to be able to import from the build root.

TODO

• Fix any failing tests
• Add test case for shared script template config and shared template config override
• Update docs and README

[codecov]

Codecov Report

❌ Patch coverage is 37.69231% with 162 lines in your changes missing coverage. Please review.
✅ Project coverage is 15.95%. Comparing base (71145de) to head (f48e5a7).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...in/java/com/datasqrl/planner/SqlScriptPlanner.java	0.00%	23 Missing ⚠️
...main/java/com/datasqrl/cli/AbstractCompileCmd.java	4.34%	21 Missing and 1 partial ⚠️
...java/com/datasqrl/cli/BaseOsProcessManagerCmd.java	26.92%	17 Missing and 2 partials ⚠️
...rc/main/java/com/datasqrl/util/DirectoryUtils.java	0.00%	14 Missing ⚠️
.../src/main/java/com/datasqrl/packager/Packager.java	14.28%	12 Missing ⚠️
.../main/java/com/datasqrl/loaders/ModuleLoaders.java	65.71%	11 Missing and 1 partial ⚠️
...datasqrl/packager/preprocess/SqrlPreprocessor.java	81.39%	5 Missing and 3 partials ⚠️
...ain/java/com/datasqrl/config/ScriptConfigImpl.java	27.27%	8 Missing ⚠️
.../src/main/java/com/datasqrl/compile/DagWriter.java	0.00%	7 Missing ⚠️
...m/datasqrl/packager/FilePreprocessingPipeline.java	37.50%	4 Missing and 1 partial ⚠️
... and 16 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2161      +/-   ##
============================================
+ Coverage     15.44%   15.95%   +0.50%     
- Complexity      975     1016      +41     
============================================
  Files           618      619       +1     
  Lines         17837    17925      +88     
  Branches       2177     2193      +16     
============================================
+ Hits           2755     2860     +105     
+ Misses        14812    14782      -30     
- Partials        270      283      +13     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.