Skip to content

feat(insight): add SQSExporter - #701

Draft
wangyb-A wants to merge 2 commits into
mainfrom
workflow-insight-sqs-exporter
Draft

feat(insight): add SQSExporter#701
wangyb-A wants to merge 2 commits into
mainfrom
workflow-insight-sqs-exporter

Conversation

@wangyb-A

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

Related: #679

Description

Adds SQSExporter to the Workflow Insight plugin, so insight records can be shipped to an Amazon SQS queue. It is modeled on the existing S3Exporter and CloudWatchLogsExporter and mirrors the official JavaScript SQSExporter contract where practical. No change was made to the SDK core, WorkflowInsight, or WorkflowInsightRecord.

Standard vs FIFO queues. The queue type is detected from the URL: a queueUrl ending in .fifo is treated as FIFO. For a standard queue the SendMessage request carries no MessageGroupId and no MessageDeduplicationId. For a FIFO queue both are always set.

FIFO id hashing, bounds, and dedup. SQS caps both ids at 128 characters. The MessageGroupId is the explicit messageGroupId (else the executionArn) used verbatim when it is 128 characters or shorter, otherwise a deterministic 64-character SHA-256 hex digest of that raw value, so grouping stays stable per execution. The MessageDeduplicationId is always a 64-character SHA-256 digest of executionArn \0 emittedAt \0 renderedBody. Two different snapshots at the same emittedAt differ by body, so they get different dedup ids and both survive the SQS 5-minute dedup window; an exact retry has the same body and the same id, so SQS de-duplicates it. Hashing uses the JDK java.security.MessageDigest, no new dependency.

Operation formats. operationsFormat mirrors the JS enum: ARRAY (default), BY_NAME, and BOTH. render() produces the matching wire map. BOTH is built inside the exporter from the record's public API, so WorkflowInsightRecord needed no new method.

Optional dependency. The sqs Maven dependency is declared <optional>true</optional> with its version inherited from the imported AWS SDK BOM, matching the s3 and cloudwatchlogs treatment, so default Lambda-log consumers do not inherit it.

IAM docs. The README documents the required sqs:SendMessage permission and FIFO queue setup alongside the usage and options.

Size headroom. maxRecordSizeBytes defaults to 256000 and bounds the rendered JSON body only. SQS counts message attributes toward the same 256 KB quota, so the default is left below the quota to leave attribute headroom. build() rejects a non-positive maxRecordSizeBytes (which previously disabled truncation silently).

Demo/Screenshots

N/A -- backend-only change, no user-visible UI.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Unit Tests

Yes. SQSExporterTest has 13 tests covering: standard-queue requests omit FIFO fields; FIFO default group id is the executionArn and dedup id is derived from executionArn/emittedAt/body; an explicit messageGroupId is honored; a long group id is bounded to a stable SHA-256 hash; rapid same-timestamp snapshots get distinct dedup ids while exact retries de-duplicate; build() rejects a non-positive maxRecordSizeBytes; and BY_NAME and BOTH render shapes still derive the bounded dedup id from emittedAt. Insight-plugin reactor: 82 tests, 0 failures. Full reactor mvn install: BUILD SUCCESS, 1954 tests, 0 failures, 31 skipped (cloud tests disabled by default).

Integration Tests

None added. The exporter is an isolated component fully covered by mocked-client unit tests, consistent with S3ExporterTest and CloudWatchLogsExporterTest, which are also unit-only.

Examples

No new example added.

Alex Wang added 2 commits September 9, 2026 23:38
Add SQSExporter to the Workflow Insight plugin, porting the JS
SQSExporter contract to the Java plugin hook surface. Each emission is
sent as one SQS SendMessage with the rendered record JSON as the body.

- queueUrl (required, builder-validated); FIFO detected by a .fifo
  suffix. FIFO messages set MessageGroupId (default: executionArn, or a
  configured messageGroupId) and MessageDeduplicationId
  (executionArn:emittedAt); standard queues carry neither.
- operationsFormat ARRAY (default) | BY_NAME | BOTH, mirroring the JS
  OperationsFormat; BOTH is built from the record's public wire map and
  OperationsIndex so WorkflowInsight/WorkflowInsightRecord are unchanged.
- status and functionName string message attributes; 256 KB default
  maxRecordSizeBytes (SQS message limit).
- Real AWS SDK v2 SqsClient with a client-injection test seam; the sqs
  module is an optional dependency so default (Lambda-log) consumers do
  not inherit it.
- Request-construction and error unit tests via a mocked SqsClient, plus
  customer README usage, queue setup, and sqs:SendMessage IAM docs.
Address review SHOULD-FIX items on the SQS insight exporter:

- Bound FIFO MessageGroupId and MessageDeduplicationId to SQS's
  128-char limit. The group id is used verbatim when it fits and is
  otherwise a deterministic SHA-256 hex digest (64 chars) of the raw
  value, keeping per-execution grouping stable.
- Derive the dedup id from executionArn + emittedAt + rendered body so
  distinct rapid ON_CHANGE snapshots stay distinct within SQS's 5-min
  dedup window while exact retries still de-duplicate.
- Reject non-positive maxRecordSizeBytes at build time (previously it
  silently disabled truncation).
- Add FIFO BY_NAME and BOTH coverage so emittedAt extraction stays
  pinned across render shapes.
- Document that maxRecordSizeBytes bounds the rendered body and leaves
  headroom for SQS message attributes.

Hashing uses only java.security.MessageDigest; no new dependency.
Standard-queue behavior unchanged.
@wangyb-A

Copy link
Copy Markdown
Contributor Author

Intent: Add an SQSExporter to the Workflow Insight plugin so insight records can be delivered to an Amazon SQS queue (standard and FIFO), mirroring the official JavaScript exporter contract where practical, with SQS-safe bounded FIFO group/dedup ids.
Not a goal: Changing the SDK core, WorkflowInsight, or WorkflowInsightRecord; adding integration/end-to-end tests; or making the sqs dependency non-optional.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant