fix(ingestion): reduce MaxKeys to avoid uncatchable payload limit failure#1885
Merged
Conversation
…lure The reprocessing workflow's S3 ListObjectsV2 call with MaxKeys=1000 can return a response just under the 256KB Step Functions payload limit. The state reports success, but Step Functions aborts the entire execution at the state transition boundary when internal metadata pushes the payload over the limit. This ExecutionFailed event is not tied to a specific state, making it uncatchable by addCatch(States.DataLimitExceeded). Reduce startMaxKeysValue from 1000 to 750 to provide ~25% headroom below the limit, preventing this edge case.
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
otaviomacedo
approved these changes
May 27, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Mostly used AI. This is a config change.
Summary
MaxKeysfor the reprocessing workflow's S3 ListObjectsV2 call from 1000 to 750Problem
The reprocessing workflow has a retry mechanism that catches
States.DataLimitExceededand retries with fewer keys. However, there's an edge case where S3 returns a response just under 256KB — the state reports success, but Step Functions aborts the entire execution at the state transition boundary when internal metadata pushes the total payload over the limit. ThisExecutionFailedevent is not tied to a specific state, soaddCatchnever fires.This was observed on 12/10 where
Try1000reported success but the execution was immediately aborted by Step Functions.Fix
Reduce
startMaxKeysValuefrom 1000 to 750 andminMaxKeysValuefrom 100 to 150. This is a temporary mitigation — a follow-up PR will make a longer term fix.Test plan
projen buildto regenerate)