Skip to content

feat(insight): add FileExporter - #700

Draft
wangyb-A wants to merge 2 commits into
mainfrom
workflow-insight-file-exporter
Draft

feat(insight): add FileExporter#700
wangyb-A wants to merge 2 commits into
mainfrom
workflow-insight-file-exporter

Conversation

@wangyb-A

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

Related: #679 (does not close it)

Description

Adds a FileExporter to the Workflow Insight plugin so insight records can be written to the local filesystem, aligned with the JavaScript SDK's file-exporter contract where practical.

The exporter has two write modes:

  • NDJSON (default): appends one compact JSON object per line to a UTC-date-partitioned file, {directory}/{YYYY-MM-DD}.ndjson. When emittedAt is missing it falls back to today's UTC date.
  • JSON: writes one pretty-printed (2-space, LF line breaks) file per execution, {directory}/{sanitize(executionName ?? executionArn)}.json, overwriting on each write. sanitize maps [^a-zA-Z0-9._-] to _.

Operation formats mirror the JS operationsFormat option:

  • ARRAY (default): operations as an array.
  • BY_NAME: operations grouped under operationsByName.
  • BOTH: the operations array plus an appended operationsByName key. The key is appended last (after all record fields and any truncation markers) to match JS {...record, operationsByName} field order.

A builder validates a required non-blank directory and a positive maxRecordSizeBytes (which has no default, matching JS). The exporter uses only java.nio.file (no new dependency); truncation and per-exporter Throwable isolation are handled centrally in WorkflowInsight.emit, so no plugin/core change was needed. Path names are normalized and asserted to stay inside the configured directory (traversal defense in depth); IO failures surface as UncheckedIOException.

EFS / shared-mount caveat (documented in the README): NDJSON Files.write(..., APPEND) is atomic for a single writer on local disk but is NOT guaranteed atomic on a shared NFS/EFS mount written by more than one Lambda environment concurrently — concurrent writers can interleave partial lines or overwrite each other. The README recommends JSON mode (one file per execution, no shared append) or a single writer per NDJSON file (e.g. a per-environment subdirectory).

Additive shared changes (public, @Experimental, no signatures changed — same pattern as the S3/CloudWatch exporters consuming the record):

  • WorkflowInsightRecord: added a public emittedAt() accessor and toBothWireMap() (builds on toWireMap() and appends operationsByName last).
  • Json: added stringifyPretty() (2-space DefaultPrettyPrinter with \n breaks, matching JSON.stringify(x, null, 2)).

Demo/Screenshots

N/A — backend library change with no user-visible UI.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Unit Tests

Yes. FileExporterTest (15 tests, real @TempDir IO, no mocked filesystem):

  • NDJSON append / compact single-line / UTC-date partitioning
  • JSON pretty-printing / overwrite-on-rewrite
  • Execution-ARN fallback + filename sanitization
  • BY_NAME and BOTH operation formats
  • Nested directory creation
  • emittedAt fallback to today
  • maxRecordSizeBytes null and set
  • Builder validation (blank directory, non-positive size)
  • flush no-op
  • traversalShapedEmittedAtIsRejectedByResolveChild: emittedAt = "../../etc/passwd" drives a traversal-shaped NDJSON name; asserts IllegalStateException ("escapes the configured directory") and no file written outside directory
  • concurrentNdjsonAppendsWriteCompleteParseableLines: 8 threads × 25 exports (200 total) against one shared NDJSON file; asserts exactly 200 non-blank lines, each parseable as a complete WorkflowInsight JSON object (no interleaving/truncation on local disk)

Full reactor mvn clean install -DskipITs: BUILD SUCCESS, 1956 tests, 0 failures, 0 errors; spotless:check clean.

Integration Tests

No. The plugin emit path (truncation + per-exporter isolation) is already covered by existing WorkflowInsight tests; the new exporter is covered end-to-end against real temp directories via the unit suite above.

Examples

No new example added — the README FileExporter usage subsection documents both modes, safe naming, the no-default size cap, and the EFS-vs-local persistence guidance.

Alex Wang added 2 commits September 9, 2026 23:39
Port the JS FileExporter to the Java insight-plugin. Writes each
Workflow Insight record to a writable directory (EFS mount, S3 File
Gateway, or /tmp) with no extra dependency, using only java.nio.file.

Two modes mirror JS: NDJSON (default) appends one compact JSON line
per emission to a date-partitioned {YYYY-MM-DD}.ndjson keyed off the
record's emittedAt day; JSON writes one pretty-printed (2-space) file
per execution, overwriting on update. operationsFormat selects ARRAY
(default), BY_NAME, or BOTH renderings. Builder validates a required
directory and a positive maxRecordSizeBytes (no default cap, matching
JS). File names are sanitized to [a-zA-Z0-9._-] and the resolved path
is verified to stay within the configured directory.

Adds a public emittedAt() accessor and a toBothWireMap() rendering to
WorkflowInsightRecord, and a stringifyPretty() helper to Json. No
change to SDK core or WorkflowInsight. Tests use real temporary
directories, not mocked IO. README documents both modes.
Document non-atomic concurrent NDJSON append on shared EFS/NFS and
recommend JSON mode or a single writer; append operationsByName last in
toBothWireMap() to match the JS spread order and correct its comment;
add a traversal-shaped emittedAt guard test and a deterministic
concurrent NDJSON append test (real @tempdir IO).
@wangyb-A

Copy link
Copy Markdown
Contributor Author

Intent: Add a filesystem FileExporter to the Workflow Insight plugin so insight records can be written locally in NDJSON (date-partitioned append) or JSON (one pretty file per execution) mode, aligned with the JS SDK's file-exporter contract, using only JDK-native IO and additive @Experimental accessors.
Not a goal: Not closing #679; no core/plugin emit changes (truncation and per-exporter isolation stay central); no new dependencies; no wired end-to-end integration test beyond real-temp-dir unit coverage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant