Skip to content

fix: own the auth-token Secret by the Token, not the JWT key - #248

Open
giuliocalzo wants to merge 1 commit into
SlinkyProject:mainfrom
giuliocalzo:fix/token-secret-owner-reference
Open

fix: own the auth-token Secret by the Token, not the JWT key#248
giuliocalzo wants to merge 1 commit into
SlinkyProject:mainfrom
giuliocalzo:fix/token-secret-owner-reference

Conversation

@giuliocalzo

Copy link
Copy Markdown
Contributor

Summary

BuildTokenSecret passes the JWT signing key Secret to BuildSecret as the owner, so the generated credential's controller ownerReference points at another Secret rather than the Token that requested it.

jwtSecret := &corev1.Secret{}
if err := b.client.Get(ctx, token.JwtKey(), jwtSecret); err != nil {
	return nil, err
}

o, err := b.BuildSecret(opts, jwtSecret)

This cuts both ways:

  • Deleting a Token leaves a live, valid Slurm JWT behind with no owning resource, invisible to anyone auditing Token objects.
  • Deleting or rotating the JWT signing key garbage-collects every credential derived from it.

It also makes Owns(&corev1.Secret{}) in SetupWithManager dead code, because the owner lookup can never resolve back to a Token. An out-of-band Secret deletion therefore goes unnoticed until the refresh requeue at 80% of the token lifetime — or forever when spec.refresh is false, since that path schedules no requeue at all.

The fix passes the Token as the owner. That also removes the client.Get above, which existed only to supply that owner; the signing key material is already resolved through refResolver.GetSecretKeyRef.

Adopting existing Secrets

Secrets written by earlier versions cannot be repaired by the normal sync path. SyncObject is called create-only from the Token Secret step, so it returns before the patch block, and even with shouldUpdate=true it skips Immutable Secrets — which is exactly what spec.refresh: false produces. Those credentials would stay mis-owned indefinitely.

adoptSecret repairs them by patching only the owner references, which remains legal on an immutable Secret because immutability covers data, not metadata.

Adoption is deliberately narrow: it only touches Secrets whose controller owner is this Token's signing key. spec.secretRef can name any Secret, so a name collision with an unrelated pre-existing Secret must not cause the operator to seize it and later garbage-collect it.

Checklist

  • I have read the
    CONTRIBUTING.md
    and the
    Code of Conduct.
  • New or existing tests cover these changes (where applicable).
  • Documentation is updated if user-visible behavior changes.

Breaking Changes

None for users. On upgrade, existing auth-token Secrets are re-pointed from the signing key Secret to their Token. This changes their garbage-collection behavior to the intended one: they are removed when the Token is deleted, and they survive deletion of the signing key.

Testing Notes

internal/controller/token/suite_test.go never started a manager, so its specs passed without a controller running. The spec that looked like it covered this behavior used token.JwtKey() (the signing key) where it meant token.SecretKey() (the generated credential), so it never touched the generated Secret at all. This PR starts a manager and corrects that key, then adds specs for ownership, recreation after out-of-band deletion, adoption of an immutable mis-owned Secret, and the no-hijack guard. Each new spec was confirmed to fail when its corresponding fix is reverted.

Also verified end to end on a kind cluster, comparing an unpatched build against the patched one using identical manifests. A Token Secret created by the unpatched build was adopted at operator startup, and a refresh: false Token behaved as follows:

Scenario Unpatched Patched
Owner on create Secret/<jwt-key> Token/<name>, uid matches CR
Pre-existing mis-owned Secret never repaired adopted at startup
Credential deleted out-of-band absent >90s, 0 reconciles recreated in ~3s
Signing key deleted credential destroyed credential survives
Token deleted live JWT orphaned collected in ~3s

Token reconcile counts stayed flat over an idle period afterwards and adoption logged once per affected Secret, so the adoption patch is idempotent and does not hot-loop.

Additional Context

Fixing the owner also revives the Owns(&corev1.Secret{}) watch as a side effect. The early return on DeletionTimestamp in token_sync.go is left as-is, since owner-reference garbage collection now covers cleanup and no finalizer is needed.

BuildTokenSecret passed the JWT signing key Secret to BuildSecret as the
owner, so the generated credential's controller ownerReference pointed at
another Secret rather than the Token that requested it.

This cut both ways. Deleting a Token left a live, valid Slurm JWT behind
with no owning resource, while deleting or rotating the signing key
garbage-collected every credential derived from it. It also made the
Owns(&corev1.Secret{}) watch dead code, because the owner lookup could
never resolve to a Token: an out-of-band Secret deletion went unnoticed
until the refresh requeue at 80% of the token lifetime, or forever when
spec.refresh is false and no requeue is scheduled.

Existing Secrets cannot be repaired by the normal sync, which is
create-only here and skips immutable Secrets outright, so adopt them
explicitly. Adoption is limited to Secrets owned by the Token's own
signing key, since spec.secretRef can name any Secret and we must not
take ownership of a resource this controller did not create.

The token suite never started a manager, so its specs passed without a
controller running, and the spec that looked like it covered this used
JwtKey() (the signing key) where it meant SecretKey() (the credential).
Start a manager and fix the key so the suite exercises the controller.

Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant