feat(insight): add SQSExporter - #701
Draft
wangyb-A wants to merge 2 commits into
Draft
Conversation
added 2 commits
September 9, 2026 23:38
Add SQSExporter to the Workflow Insight plugin, porting the JS SQSExporter contract to the Java plugin hook surface. Each emission is sent as one SQS SendMessage with the rendered record JSON as the body. - queueUrl (required, builder-validated); FIFO detected by a .fifo suffix. FIFO messages set MessageGroupId (default: executionArn, or a configured messageGroupId) and MessageDeduplicationId (executionArn:emittedAt); standard queues carry neither. - operationsFormat ARRAY (default) | BY_NAME | BOTH, mirroring the JS OperationsFormat; BOTH is built from the record's public wire map and OperationsIndex so WorkflowInsight/WorkflowInsightRecord are unchanged. - status and functionName string message attributes; 256 KB default maxRecordSizeBytes (SQS message limit). - Real AWS SDK v2 SqsClient with a client-injection test seam; the sqs module is an optional dependency so default (Lambda-log) consumers do not inherit it. - Request-construction and error unit tests via a mocked SqsClient, plus customer README usage, queue setup, and sqs:SendMessage IAM docs.
Address review SHOULD-FIX items on the SQS insight exporter: - Bound FIFO MessageGroupId and MessageDeduplicationId to SQS's 128-char limit. The group id is used verbatim when it fits and is otherwise a deterministic SHA-256 hex digest (64 chars) of the raw value, keeping per-execution grouping stable. - Derive the dedup id from executionArn + emittedAt + rendered body so distinct rapid ON_CHANGE snapshots stay distinct within SQS's 5-min dedup window while exact retries still de-duplicate. - Reject non-positive maxRecordSizeBytes at build time (previously it silently disabled truncation). - Add FIFO BY_NAME and BOTH coverage so emittedAt extraction stays pinned across render shapes. - Document that maxRecordSizeBytes bounds the rendered body and leaves headroom for SQS message attributes. Hashing uses only java.security.MessageDigest; no new dependency. Standard-queue behavior unchanged.
Contributor
Author
|
Intent: Add an |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Issue Link, if available
Related: #679
Description
Adds
SQSExporterto the Workflow Insight plugin, so insight records can be shipped to an Amazon SQS queue. It is modeled on the existingS3ExporterandCloudWatchLogsExporterand mirrors the official JavaScriptSQSExportercontract where practical. No change was made to the SDK core,WorkflowInsight, orWorkflowInsightRecord.Standard vs FIFO queues. The queue type is detected from the URL: a
queueUrlending in.fifois treated as FIFO. For a standard queue theSendMessagerequest carries noMessageGroupIdand noMessageDeduplicationId. For a FIFO queue both are always set.FIFO id hashing, bounds, and dedup. SQS caps both ids at 128 characters. The
MessageGroupIdis the explicitmessageGroupId(else theexecutionArn) used verbatim when it is 128 characters or shorter, otherwise a deterministic 64-character SHA-256 hex digest of that raw value, so grouping stays stable per execution. TheMessageDeduplicationIdis always a 64-character SHA-256 digest ofexecutionArn \0 emittedAt \0 renderedBody. Two different snapshots at the sameemittedAtdiffer by body, so they get different dedup ids and both survive the SQS 5-minute dedup window; an exact retry has the same body and the same id, so SQS de-duplicates it. Hashing uses the JDKjava.security.MessageDigest, no new dependency.Operation formats.
operationsFormatmirrors the JS enum:ARRAY(default),BY_NAME, andBOTH.render()produces the matching wire map.BOTHis built inside the exporter from the record's public API, soWorkflowInsightRecordneeded no new method.Optional dependency. The
sqsMaven dependency is declared<optional>true</optional>with its version inherited from the imported AWS SDK BOM, matching thes3andcloudwatchlogstreatment, so default Lambda-log consumers do not inherit it.IAM docs. The README documents the required
sqs:SendMessagepermission and FIFO queue setup alongside the usage and options.Size headroom.
maxRecordSizeBytesdefaults to256000and bounds the rendered JSON body only. SQS counts message attributes toward the same 256 KB quota, so the default is left below the quota to leave attribute headroom.build()rejects a non-positivemaxRecordSizeBytes(which previously disabled truncation silently).Demo/Screenshots
N/A -- backend-only change, no user-visible UI.
Checklist
Testing
Unit Tests
Yes.
SQSExporterTesthas 13 tests covering: standard-queue requests omit FIFO fields; FIFO default group id is theexecutionArnand dedup id is derived fromexecutionArn/emittedAt/body; an explicitmessageGroupIdis honored; a long group id is bounded to a stable SHA-256 hash; rapid same-timestamp snapshots get distinct dedup ids while exact retries de-duplicate;build()rejects a non-positivemaxRecordSizeBytes; andBY_NAMEandBOTHrender shapes still derive the bounded dedup id fromemittedAt. Insight-plugin reactor: 82 tests, 0 failures. Full reactormvn install: BUILD SUCCESS, 1954 tests, 0 failures, 31 skipped (cloud tests disabled by default).Integration Tests
None added. The exporter is an isolated component fully covered by mocked-client unit tests, consistent with
S3ExporterTestandCloudWatchLogsExporterTest, which are also unit-only.Examples
No new example added.