Skip to content

fix(ingestion): reduce MaxKeys to avoid uncatchable payload limit failure#1885

Merged
cdklabs-automation merged 2 commits into
mainfrom
fix/reduce-maxkeys-payload-limit
May 27, 2026
Merged

fix(ingestion): reduce MaxKeys to avoid uncatchable payload limit failure#1885
cdklabs-automation merged 2 commits into
mainfrom
fix/reduce-maxkeys-payload-limit

Conversation

@iankhou

@iankhou iankhou commented May 26, 2026

Copy link
Copy Markdown
Contributor

Mostly used AI. This is a config change.

Summary

  • Reduces the starting MaxKeys for the reprocessing workflow's S3 ListObjectsV2 call from 1000 to 750
  • Provides ~25% headroom below the 256KB Step Functions payload limit to prevent an uncatchable edge case

Problem

The reprocessing workflow has a retry mechanism that catches States.DataLimitExceeded and retries with fewer keys. However, there's an edge case where S3 returns a response just under 256KB — the state reports success, but Step Functions aborts the entire execution at the state transition boundary when internal metadata pushes the total payload over the limit. This ExecutionFailed event is not tied to a specific state, so addCatch never fires.

This was observed on 12/10 where Try1000 reported success but the execution was immediately aborted by Step Functions.

Fix

Reduce startMaxKeysValue from 1000 to 750 and minMaxKeysValue from 100 to 150. This is a temporary mitigation — a follow-up PR will make a longer term fix.

Test plan

  • Verify snapshot tests pass (will need projen build to regenerate)
  • Deploy and observe the next daily reprocessing run succeeds

…lure

The reprocessing workflow's S3 ListObjectsV2 call with MaxKeys=1000 can
return a response just under the 256KB Step Functions payload limit. The
state reports success, but Step Functions aborts the entire execution at
the state transition boundary when internal metadata pushes the payload
over the limit. This ExecutionFailed event is not tied to a specific
state, making it uncatchable by addCatch(States.DataLimitExceeded).

Reduce startMaxKeysValue from 1000 to 750 to provide ~25% headroom
below the limit, preventing this edge case.
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@iankhou iankhou self-assigned this May 26, 2026
@cdklabs-automation cdklabs-automation added this pull request to the merge queue May 27, 2026
Merged via the queue into main with commit b16aa99 May 27, 2026
10 checks passed
@cdklabs-automation cdklabs-automation deleted the fix/reduce-maxkeys-payload-limit branch May 27, 2026 07:53
@iankhou

iankhou commented May 27, 2026

Copy link
Copy Markdown
Contributor Author
Screenshot 2026-05-27 at 17 06 56 It worked!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants