Skip to content

Add pluggable storage providers and optional AWS hybrid foundation - #2

Merged
kcskribbl merged 6 commits into
mainfrom
feat/secure-pluggable-blob-storage
Sep 20, 2026
Merged

kcskribbl merged 6 commits into
mainfrom
feat/secure-pluggable-blob-storage

Conversation

@kcskribbl

@kcskribbl kcskribbl commented Sep 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Draft implementation following the discussion in #1. Adds segregated,
pluggable Azure, S3, DynamoDB and private IPFS/Cluster adapters while retaining
memory/file storage and the existing encrypted patient-record service.

src/storage/
  core/                 BlobStore contract and validation
  config/               Generic registry, configuration and runtime secrets
  adapters/
    memory/  file/  azure/  s3/  dynamodb/  ipfs/

Each provider owns its construction in provider.ts. Additional typed
factories can be registered without changing the service or central factory.
Public imports remain available through src/index.ts.

Included

  • Atomic conditional creation and ETag-backed numeric compare-and-swap for Azure/S3; explicit conflict, missing-object, malformed-data and transport failure behavior.
  • Strongly consistent DynamoDB metadata with conditional writes, strict version checks, a 350-KiB payload bound and a matching key-partitioned infrastructure table.
  • Configurable backend selection wired into the local server and a synthetic enroll/append/reconnect example; unused providers are not constructed, and dependency cycles fail closed.
  • Private-IPFS raw-CID integrity checks, bounded authenticated transport, observed replica-threshold checks, and atomic pointer publication through a separately configured metadata BlobStore.
  • Optional AWS composition: EC2 -> private Kubo/Cluster -> encrypted EBS; DynamoDB metadata; synchronous SSE-KMS S3 backup before pointer publication. Failed backups block publication. Explicit CID-verified restoration never silently substitutes for failed reads.
  • Runtime Azure/AWS identity chains; optional IPFS authorization from Key Vault, Secrets Manager or restricted secret mounts. HTTPS/TLS defaults, emulator-only loopback exceptions, redacted provider failures, and secret-file exclusions.
  • Review-only CloudFormation foundation: private/versioned S3, encrypted/PITR DynamoDB, scoped runtime IAM, retained storage/keys, and optional private EC2/EBS with IMDSv2. Compute is disabled by default; approved AMI and private-cluster setup remain operator prerequisites.
  • Account-free contract, SDK HTTP-transport, concurrency, malformed-input, access-denial, tamper and cleartext-leakage tests.
  • Actual Azurite integration harness; isolated three-peer Linux Docker IPFS fixture with temporary file metadata/backups, verified restore, and an existing-block node-loss probe; opt-in real-provider integration suite.
  • Account-free PR CI and a separate manual, main-only cloud smoke workflow using protected-environment OIDC, not stored cloud credentials.
  • Setup: storage quickstart; extension guide: provider layout; AWS prerequisites, configuration and recovery: AWS runbook.

Validation and limitations

  • npm test: 252 tests passed, including 44 DynamoDB adapter tests, 10 backup/recovery tests and 14 AWS infrastructure policy checks.
  • npm run typecheck and npm run build: passed.
  • npm run test:integration:azure: 7 tests passed against actual Azurite, using temporary synthetic credentials and cleaned-up local storage.
  • Configured file/memory examples and wrong-factor example completed.
  • npm audit --omit=dev: no reported runtime-dependency vulnerabilities.
  • Linux CI: 8 private-IPFS integration tests passed, including configured backup restoration, plus byte-for-byte retrieval after stopping the ingress Kubo/Cluster node. The real three-peer fixture remains private through internal Docker networking and loopback-only API relays. Local Windows Docker execution is not claimed.
  • Final CI run passed on 1cf6145: 252 default tests, typecheck/build, 7 Azurite tests, and the private-IPFS suite/node-loss probe.
  • Infrastructure tests parse JSON and check selected invariants; they are not CloudFormation service validation, deployment or live IAM/KMS simulation.
  • Real Azure/AWS account, IAM, KMS, private-network and OIDC tests have not been run: no accounts/credentials were supplied.

Important boundaries for review

  • This does not fully resolve Consider use a distributed file system (peer 2 peer) #1's no-single-point-of-failure goal. Replicated IPFS content still requires a durable atomic key-to-CID index. File metadata is local-test-only; independent index consensus, automatic ingress failover, offline merge, governance, funding and long-term recovery remain separate design work. This PR references rather than closes Consider use a distributed file system (peer 2 peer) #1.
  • These are selectable adapters, not automatic cross-cloud mirroring. Existing local data is not migrated. Use a separate namespace for the IPFS index.
  • Future activation requires provisioning private storage, scoped IAM/RBAC/KMS permissions, networking and workload-identity trust, then filling nonsecret configuration. The AWS node additionally requires a reviewed prebuilt AMI, external secret provisioning, an authenticated private HTTPS proxy and other private peers. This is not a turnkey cluster installer or an "add access keys and deploy" promise.
  • KMS protects infrastructure encryption only; it does not replace patient/clinician factors or receive reconstructed record keys. A single EC2/EBS node remains one failure domain. Metadata recovery is separate from content restoration, and old-snapshot rollback resistance remains future work.
  • Configure required reviewers and main-only deployment rules for the storage-integration environment before enabling the cloud smoke workflow. No long-lived cloud credentials should be placed in GitHub.
  • The smoke workflow uses a separate OIDC role and synthetic-integration/; do not reuse the EC2-only foundation role, whose object scope is records/.
  • Full dependency audit still reports pre-existing development-tool advisories (Happy DOM/Vitest) and Azurite development-only transitive UUID advisories. No forced, unrelated major tool upgrades are included; review these separately before enabling privileged CI. The account-free PR workflow has no cloud identity.
  • Research POC only: synthetic fixtures, no real patient data, and no claim of clinical, regulatory, or production readiness.

Please keep this PR in draft while reviewing concurrency, metadata exposure,
operator/network assumptions, IAM policies and live-provider results.

Karthik Chandrasekaran added 6 commits September 19, 2026 08:31
Add runtime identities, secret-source configuration, storage contract and integration tests, isolated emulators, examples and draft deployment guidance. Refs #1.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fc4430e2-35f5-48b8-bd0f-5dc6fad2a0ff
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fc4430e2-35f5-48b8-bd0f-5dc6fad2a0ff
Keep the swarm on an internal bridge and expose fixture APIs through loopback-only relays.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fc4430e2-35f5-48b8-bd0f-5dc6fad2a0ff
Report only aggregate healthy-peer and freespace counts, never daemon logs or identities.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fc4430e2-35f5-48b8-bd0f-5dc6fad2a0ff
Add DynamoDB metadata, synchronous encrypted-content backups and explicit recovery, runtime Secrets Manager references, and an opt-in EC2/EBS/S3/KMS foundation. Keep provider SDKs outside the record service and preserve dual unlock.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fc4430e2-35f5-48b8-bd0f-5dc6fad2a0ff
Keep emulator backups in fixture-owned temporary files and clarify the separate federated smoke-test role and prefix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fc4430e2-35f5-48b8-bd0f-5dc6fad2a0ff
@kcskribbl kcskribbl changed the title Add secure Azure/S3 adapters and experimental private IPFS storage Add pluggable storage providers and optional AWS hybrid foundation Sep 19, 2026
@kcskribbl
kcskribbl marked this pull request as ready for review September 20, 2026 02:03
@kcskribbl
kcskribbl merged commit e81ded4 into main Sep 20, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Consider use a distributed file system (peer 2 peer)

1 participant