Skip to content

Implement shared-memory communication #7

@mcopik

Description

@mcopik

Serverless functions are becoming more powerful, and AWS Lambda now supports using up to 6 vCPUs per function. We could benefit by running multiple "workers" as threads on a single function and letting them communicate locally. We cannot use processes within a single function as that would require shmem, which is not supported on AWS Lambda.

  • Introduce a memory-based communication channel.
  • Adjust the function launcher to support a hierarchy of functions - local and external ones.
  • Implement in collectives and P2P communication the selection of external and local channels based on the recipient.

This might require making collectives to be natively multi-protocol first #6.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions