feat: add distributed map operation

[nvasiu]

Summary

Adds the distributed map operation (ctx.distributed_map) to the Python SDK:
A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.

Changes

concurrency/models.py

• DistributedMapSummary: what ctx.distributed_map returns, describes run's overall outcome.
• DistributedMapResult: returned from ctx.distributed_map when inline result collection is enabled. Contains individual map run item outcomes.
• DistributedMapResultItem and DistributedMapItemError: represent a single item's result / error

config.py

• DistributedMapConfig: optional settings for distributed map
• DistributedMapSource: describes where map run items come from (inline list, S3, or a custom reader)
• DistributedMapProcessor: describes the Lambda that processes items and how outcomes are reported back
• ProcessorRetryConfig: configures how failing items are retried
• DistributedMapCompletionConfig: defines item failure thresholds for marking the overall map run failed
• SuccessDestination, FailureDestination, DistributedMapDestinationConfig, DistributedMapDestination: for routing successful and failed item records to S3

context.py

• ctx.distributed_map: the entry point a customer calls to run a distributed map

distributed_map_helpers.py

• Authoring wrappers: let a customer write a plain function and have it work as a processor Lambda without hand-writing the item or batch protocol, including durable-execution variants and a reader
• Currently placed in a separate top level file, can be moved elsewhere.

operation/distributed_map.py

• The executor: drives the operation so the caller's function suspends while the run executes and resumes with the finished outcome, and surfaces a clear error if the operation itself fails

lambda_service.py

• Carries the operation and its results to and from the backend service

state.py

• Stores the run's outcome in the durable execution state so it persists across suspend and resume

exceptions.py

• DistributedMapError: the error a customer catches when a run or an item fails

__init__.py

• Makes the distributed-map types importable by customers as public API

Tests

tests/operation/distributed_map_test.py
tests/context_test.py

• Core operation unit tests: executor, config/argument validation, wire round trips, result types

tests/e2e/distributed_map_int_test.py

• End to end ctx.distributed_map tests, mocking backend responses: suspend / resume, collect results, throw on failure

tests/distributed_map_helpers_test.py

• Authoring wrapper tests: checking that they process items, report failures, reject bad inputs

tests/e2e/distributed_map_helpers_int_test.py

• End to end authoring wrapper tests.

TODO

• When the model changes and distributed map implementation are complete in the durable service backend, these SDK changes need to be verified against them.

Future Tasks

• Add distributed map to the local emulator (in the testing package).
  • When this is done, we can add full end to end tests using the emulator.
• Add distributed map examples to the examples package.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[yaythomas]

this won't work against installed botocore yet. the checkpoint goes through the boto3 lambda client, which rejects unknown structure members at param-validation time, so every ctx.distributed_map call fails client-side with ParamValidationError until a botocore release ships the distributed-map model.

bump the boto3 minimum pin once this available. so the failure mode becomes an install-time constraint instead of a runtime one.

[yaythomas]

any other changes necessary here to wire the events in for plugin? are there specific events?

[nvasiu]

Looks like no operation type specific hooks anywhere, DISTRIBUTED_MAP will call the same hooks as invoke/step.

But there are experimental result/error fields on OperationInfo that won't get populated for dmaps. Do we want to do that as part of this PR?