fix: own the auth-token Secret by the Token, not the JWT key - #248
Open
giuliocalzo wants to merge 1 commit into
Open
fix: own the auth-token Secret by the Token, not the JWT key#248giuliocalzo wants to merge 1 commit into
giuliocalzo wants to merge 1 commit into
Conversation
BuildTokenSecret passed the JWT signing key Secret to BuildSecret as the
owner, so the generated credential's controller ownerReference pointed at
another Secret rather than the Token that requested it.
This cut both ways. Deleting a Token left a live, valid Slurm JWT behind
with no owning resource, while deleting or rotating the signing key
garbage-collected every credential derived from it. It also made the
Owns(&corev1.Secret{}) watch dead code, because the owner lookup could
never resolve to a Token: an out-of-band Secret deletion went unnoticed
until the refresh requeue at 80% of the token lifetime, or forever when
spec.refresh is false and no requeue is scheduled.
Existing Secrets cannot be repaired by the normal sync, which is
create-only here and skips immutable Secrets outright, so adopt them
explicitly. Adoption is limited to Secrets owned by the Token's own
signing key, since spec.secretRef can name any Secret and we must not
take ownership of a resource this controller did not create.
The token suite never started a manager, so its specs passed without a
controller running, and the spec that looked like it covered this used
JwtKey() (the signing key) where it meant SecretKey() (the credential).
Start a manager and fix the key so the suite exercises the controller.
Signed-off-by: Giulio Calzolari <gcalzolari@nvidia.com>
giuliocalzo
requested review from
SkylerMalinowski,
alanmutsch,
catblade,
vivian-hafener and
wickberg
as code owners
August 28, 2026 13:20
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
BuildTokenSecretpasses the JWT signing key Secret toBuildSecretas the owner, so the generated credential's controllerownerReferencepoints at another Secret rather than theTokenthat requested it.This cuts both ways:
Tokenleaves a live, valid Slurm JWT behind with no owning resource, invisible to anyone auditingTokenobjects.It also makes
Owns(&corev1.Secret{})inSetupWithManagerdead code, because the owner lookup can never resolve back to aToken. An out-of-band Secret deletion therefore goes unnoticed until the refresh requeue at 80% of the token lifetime — or forever whenspec.refreshisfalse, since that path schedules no requeue at all.The fix passes the
Tokenas the owner. That also removes theclient.Getabove, which existed only to supply that owner; the signing key material is already resolved throughrefResolver.GetSecretKeyRef.Adopting existing Secrets
Secrets written by earlier versions cannot be repaired by the normal sync path.
SyncObjectis called create-only from theTokenSecret step, so it returns before the patch block, and even withshouldUpdate=trueit skipsImmutableSecrets — which is exactly whatspec.refresh: falseproduces. Those credentials would stay mis-owned indefinitely.adoptSecretrepairs them by patching only the owner references, which remains legal on an immutable Secret because immutability coversdata, not metadata.Adoption is deliberately narrow: it only touches Secrets whose controller owner is this
Token's signing key.spec.secretRefcan name any Secret, so a name collision with an unrelated pre-existing Secret must not cause the operator to seize it and later garbage-collect it.Checklist
CONTRIBUTING.md
and the
Code of Conduct.
Breaking Changes
None for users. On upgrade, existing auth-token Secrets are re-pointed from the signing key Secret to their
Token. This changes their garbage-collection behavior to the intended one: they are removed when theTokenis deleted, and they survive deletion of the signing key.Testing Notes
internal/controller/token/suite_test.gonever started a manager, so its specs passed without a controller running. The spec that looked like it covered this behavior usedtoken.JwtKey()(the signing key) where it meanttoken.SecretKey()(the generated credential), so it never touched the generated Secret at all. This PR starts a manager and corrects that key, then adds specs for ownership, recreation after out-of-band deletion, adoption of an immutable mis-owned Secret, and the no-hijack guard. Each new spec was confirmed to fail when its corresponding fix is reverted.Also verified end to end on a kind cluster, comparing an unpatched build against the patched one using identical manifests. A
TokenSecret created by the unpatched build was adopted at operator startup, and arefresh: falseToken behaved as follows:Secret/<jwt-key>Token/<name>, uid matches CRTokendeletedToken reconcile counts stayed flat over an idle period afterwards and adoption logged once per affected Secret, so the adoption patch is idempotent and does not hot-loop.
Additional Context
Fixing the owner also revives the
Owns(&corev1.Secret{})watch as a side effect. The early return onDeletionTimestampintoken_sync.gois left as-is, since owner-reference garbage collection now covers cleanup and no finalizer is needed.