Skip to content

fix(k8s): give each bridge its own CA secret - #17

Merged
vercel-eddie merged 1 commit into
mainfrom
ca-secret-not-per-bridge
Oct 1, 2026
Merged

vercel-eddie merged 1 commit into
mainfrom
ca-secret-not-per-bridge

Conversation

@vercel-eddie

Copy link
Copy Markdown
Collaborator

InjectCA named the CA secret bridge-ca, and Rename only appends the device ID, so every bridge from one machine shared bdg-bridge-ca-<device>:

  • Each create replaced the others' CA. InjectCA generates a new CA per create and upserts it over the shared secret.
  • Concurrent creates raced. Both found it missing and created it, so the second failed with secrets "bdg-bridge-ca-<device>" already exists. That's what broke the managed-drives e2e when it brought two bridges up at once.
  • One remove deleted it for everyone. Each create relabels the secret with its own vercel.sh/bridge-deployment, so removing whichever bridge created it last deletes it. The device's other bridges keep running, but their pods can't restart, because the secret volume is gone. I hit this in venus: removing a test bridge deleted bdg-bridge-ca-39twy3, which three live bridges still mount.

Change

The secret is named after the bridge, bdg-bridge-ca-<bridge>-<device>, so it carries only its own bridge's labels and remove deletes only that one. Nothing else refers to the secret by name. The proxy reads the CA from its mount, and the devcontainer gets it from the proxy's metadata.

Rollout

  • Existing bridges keep mounting the old shared secret until they're recreated. New bridges never touch it.
  • A shared secret that's already been deleted only matters if a bridge pod restarts. Recreating that bridge fixes it.

Testing

  • TestInjectCA_SecretPerBridge runs InjectCA, Rename, InjectLabels and RewriteRefs for two bridges from one device. It checks that each gets its own secret, labeled with its own bridge's deployment and mounted by it. Without the fix, both bridges get bdg-bridge-ca-dev123.
  • go vet, and go test ./pkg/k8s/resources/ ./pkg/commands/ pass.

🤖 Generated with Claude Code

InjectCA named the secret "bridge-ca", and Rename only appends the device
ID, so every bridge from a device shared bdg-bridge-ca-<device>:

- each create generated a new CA and replaced the one the device's other
  bridges had mounted;
- two concurrent creates both found it missing and raced to create it, so
  the loser failed with AlreadyExists;
- each create relabeled it with its own bridge's labels, so removing that
  bridge deleted the secret out from under the others, whose pods then
  can't restart.

The secret is now named after the bridge (bdg-bridge-ca-<bridge>-<device>),
so it carries only its own bridge's labels and is removed with it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vercel-eddie
vercel-eddie merged commit 8517be9 into main Oct 1, 2026
6 checks passed
@vercel-eddie
vercel-eddie deleted the ca-secret-not-per-bridge branch October 1, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant