Skip to content

Document step behavior when the Lambda invocation times out mid-step (no StepFailed recorded, retry budget not consumed) #281

Description

@yaythomas

Summary

The documentation does not explicitly describe what happens when the Lambda
runtime is terminated while a step is still executing — for example, when a
step's work takes longer than the function's invocation timeout (up to the
15-minute Lambda maximum), the sandbox crashes, or the SDK is unable to reach
the checkpoint API.

In this situation:

  • The step never completes, so no StepFailed event is recorded in the
    execution history — only an InvocationCompleted event with an error
    (e.g. Sandbox.Timedout).
  • Because the step is not marked as failed, the step's retry strategy /
    retry budget is not consumed.
  • Lambda retries the invocation, replays the execution, and (with default
    at-least-once step semantics) re-runs the same incomplete step from the
    beginning.
  • If the step deterministically takes longer than the invocation timeout,
    this repeats indefinitely until the durable execution's ExecutionTimeout
    is reached or the execution is stopped manually.

This is surprising if you expect a step retry strategy (e.g. "max 3
attempts") to bound how many times a step's code can run. That bound only
applies to failures the SDK observes and checkpoints — not to invocation-level
interruptions.

What the docs cover today

  • The step operation and error-handling pages describe
    StepInterruptedError for at-most-once steps that Lambda interrupted
    before the result was checkpointed.
  • The idempotency best-practices page explains at-most-once vs
    at-least-once semantics and mentions that a process crash or invocation
    timeout can cause a step to run again.

What's missing is an explicit, discoverable treatment of the
runaway-step failure mode above and the recommended mitigations.

Proposed documentation changes

  1. Describe the failure mode explicitly, ideally in
    patterns/best-practices/step-design.md (or a dedicated section in the
    error-handling docs): when the runtime is terminated mid-step (invocation
    timeout, sandbox crash, inability to checkpoint), no StepFailed event
    is recorded, retry budgets are not consumed, and the step restarts on the
    next invocation. Include the concrete consequence: a step whose work
    always exceeds the invocation timeout will loop until ExecutionTimeout.

  2. Document the mitigations:

    • Keep step work well under the invocation timeout. A single step's
      execution must fit inside one invocation; split long-running work into
      smaller steps, or move it out of the function entirely (e.g.
      invoke, callbacks).
    • Use at-most-once-per-retry semantics for steps that might be
      interrupted, so the interruption is detected (StepInterruptedError /
      StepInterruptedException) and subject to the retry strategy instead
      of silently restarting.
    • Implement an in-step timeout so the step fails (and checkpoints the
      failure) before the sandbox is terminated:
      • TypeScript: race the work against a timeout promise and use an
        AbortController to cancel it.
      • Java: DurableFuture.anyOf(longStep, context.wait(...)) — see the
        WaitExample in the Java SDK.
      • Python / C#: equivalent cancellation patterns.
  3. Cross-link the new content from:

    • sdk-reference/operations/step.md (retry strategy section — clarify
      what the retry budget does and does not bound)
    • sdk-reference/error-handling/retries.md
    • patterns/best-practices/idempotency.md
    • getting-started/key-concepts.md (Timeouts section — note that the
      function timeout also bounds how long any single step may run)

Why this matters

Without this guidance, a single misbehaving step can consume the entire
execution timeout in silent retry loops, with nothing in the step-level
history indicating a failure. Users currently have to discover the pattern
themselves (e.g. by reading execution history at startup and detecting
consecutive InvocationCompleted errors with no durable progress).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions