Skip to content

🐛 OCPBUGS-123707: prevent ClusterCatalog catalog rollbacks - #2965

Open
tmshort wants to merge 1 commit into
operator-framework:mainfrom
tmshort:OCPBUGS-123707-catalog-version-guard
Open

tmshort wants to merge 1 commit into
operator-framework:mainfrom
tmshort:OCPBUGS-123707-catalog-version-guard

Conversation

@tmshort

@tmshort tmshort commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Summary

References OCPBUGS-123707.

  • Records the OCI config label olm.operatorframework.io/catalog-version in cached catalog metadata and ClusterCatalog status.
  • Rejects a different catalog digest when its version is missing or not greater than the last accepted version.
  • Preserves the accepted version across unavailable content so retries continue to reject a rollback.
  • Adds controller, cache, and puller coverage for the rollback guard and metadata lifecycle.

Validation

  • make test-unit

Summary by CodeRabbit

  • New Features
    • Catalog images can include a publication version that is shown in the catalog’s resolved image status.
    • When a version has been recorded, a different image digest must have a higher version to replace the currently served catalog.
    • Older images without a publication version remain supported until a version has been recorded.
  • Bug Fixes
    • Catalogs are prevented from serving content from a different image digest when its version is equal to or lower than the recorded version.

Record a positive catalog publication version from the OCI image config label alongside cached catalog content and in ClusterCatalog status.

Reject a newly resolved digest when the currently accepted catalog is versioned and the new catalog has no version or a non-increasing version. Preserve the last accepted version when content is unavailable so subsequent reconciles continue to reject the rollback.

Add controller, cache, and puller coverage for version parsing, metadata persistence and garbage collection, and rollback rejection.

Validation: make test-unit
Signed-off-by: Todd Short <tshort@redhat.com>
@openshift-ci

openshift-ci Bot commented Oct 1, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tmshort for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify

netlify Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for olmv1 ready!

Name Link
🔨 Latest commit e970625
🔍 Latest deploy log https://app.netlify.com/projects/olmv1/deploys/6abe646705074700088ac5fb
😎 Deploy Preview https://deploy-preview-2965--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@tmshort tmshort changed the title OCPBUGS-123707: prevent ClusterCatalog catalog rollbacks 🐛 OCPBUGS-123707: prevent ClusterCatalog catalog rollbacks Oct 1, 2026
@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 68fc10ee-8286-4e91-8524-19d44749d8aa

📥 Commits

Reviewing files that changed from the base of the PR and between 18dfd75 and e970625.

📒 Files selected for processing (17)
  • api/v1/clustercatalog_types.go
  • applyconfigurations/api/v1/resolvedimagesource.go
  • applyconfigurations/internal/internal.go
  • docs/api-reference/olmv1-api-reference.md
  • helm/olmv1/base/catalogd/crd/experimental/olm.operatorframework.io_clustercatalogs.yaml
  • helm/olmv1/base/catalogd/crd/standard/olm.operatorframework.io_clustercatalogs.yaml
  • internal/catalogd/controllers/core/clustercatalog_controller.go
  • internal/catalogd/controllers/core/clustercatalog_controller_test.go
  • internal/shared/util/image/cache.go
  • internal/shared/util/image/cache_test.go
  • internal/shared/util/image/fakes.go
  • internal/shared/util/image/pull.go
  • internal/shared/util/image/pull_test.go
  • manifests/experimental-e2e.yaml
  • manifests/experimental.yaml
  • manifests/standard-e2e.yaml
  • manifests/standard.yaml

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.


📝 Walkthrough

Walkthrough

The change reads catalog publication versions from OCI image metadata, stores them with cached images, and exposes them in ResolvedImageSource. Reconciliation uses the versions to reject a different image digest unless its version is greater than the recorded version.

Changes

Catalog publication version

Layer / File(s) Summary
Expose catalogVersion in API and schemas
api/v1/clustercatalog_types.go, applyconfigurations/api/v1/resolvedimagesource.go, applyconfigurations/internal/internal.go, helm/olmv1/base/catalogd/crd/*, manifests/*.yaml, docs/api-reference/olmv1-api-reference.md
ResolvedImageSource and its apply configuration add the optional catalogVersion field. The generated schemas and API reference describe its source and the version requirement for replacing served content with a different digest.
Read and persist image versions
internal/shared/util/image/cache.go, internal/shared/util/image/pull.go, internal/shared/util/image/fakes.go, internal/shared/util/image/cache_test.go, internal/shared/util/image/pull_test.go
The cache reads versions from the OCI label and persists them by image digest. CatalogPuller returns the cached version with pulled content. Tests cover parsing, persistence, garbage collection, and pull errors.
Check versions during reconciliation
internal/catalogd/controllers/core/clustercatalog_controller.go, internal/catalogd/controllers/core/clustercatalog_controller_test.go
Reconciliation rejects a different digest when its version is not greater than the currently recorded positive version. It records accepted versions in status and cached state, and retains the last accepted source when content is unavailable. Tests cover accepted versions, rollbacks, and repeated reconciliation.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant ClusterCatalogController
  participant CatalogPuller
  participant CatalogVersionCache
  participant ClusterCatalogStatus
  ClusterCatalogController->>CatalogPuller: Pull catalog content and version
  CatalogPuller->>CatalogVersionCache: Look up version for canonical image reference
  CatalogVersionCache-->>CatalogPuller: Return cached catalog version
  CatalogPuller-->>ClusterCatalogController: Return content, reference, and version
  ClusterCatalogController->>ClusterCatalogController: Compare versions when image digests differ
  alt Version accepted
    ClusterCatalogController->>ClusterCatalogStatus: Record resolved reference and catalog version
  else Rollback rejected
    ClusterCatalogController->>ClusterCatalogStatus: Set progressing status and retain accepted source
  end
Loading

Suggested reviewers: perdasilva, pedjak

Merge Risk: ⚪ Minimal · up to e9706

The publication-version guard behaves as documented, including when a catalog source changes. No identified issue needs resolution before merging.

Security Architecture Review

Security architecture risk: 🟡 Moderate · up to e9706

Rollback protection can lose track of a newly served catalog when saving its accepted version fails. A later retry may then accept an older publication. Existing image-trust controls still limit which content can be accepted.

Retained concerns

  • Medium · security · inferred: Served content and its durable accepted version have different commit points. Reconciliation replaces serving storage before saving status, and a status-update failure does not undo that replacement. For example, if status records version 10, version 20 is published, and its status update fails, a later candidate at version 15 can pass the guard against version 10. The higher in-memory version does not participate in that comparison. This can undermine rollback protection during recovery without bypassing the existing image policy.
Security review details

Security Blast Radius

  • inferred — The identified recovery gap affects a catalog's served content and its consumers. Cache metadata is isolated by catalog owner and digest. Exploitation requires an older candidate to become reachable through that catalog's configured source and remain acceptable under existing image-trust controls; cross-catalog access, credential exposure, and installed-workload compromise were not established.

Security Findings and Attack Paths

  • inferred — A publisher or source editor able to make an older trusted digest available could combine that candidate with a failed accepted-version status update. Because the next comparison uses the older persisted version rather than the highest content already published, the candidate may pass despite being older than content previously served. This sequence is source-derived and was not reproduced by execution.

Trust Boundaries and Controls

  • observed — The concrete implementation binds version lookup to the same catalog owner and canonical digest returned by Pull. Pull failures return before metadata acceptance, invalid label values fail cache storage, and a different digest is rejected against a recorded positive version unless its version increases.

Resilience and Maintainability Implications

  • inferred — The cache content and sidecar have separate write points, but catalogd's startup cleanup removes both before rebuilding them. Consequently, an interrupted sidecar write alone does not establish a production restart bypass. This counterevidence does not resolve publication that occurs before its accepted version is durably saved.

Hardening Proposals

  • proposed — Define a durable publication transition that prevents a newly served version from exceeding the recorded rollback floor. Consider persisting a pending or accepted high-water mark before activation, with explicit recovery semantics, and validate status conflicts, interruption, and changing candidates across retries.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 5.26% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 19 functions across 10 files. (7 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the bug, issue reference, and primary change: preventing ClusterCatalog catalog rollbacks.
Description check ✅ Passed The description summarizes the change, explains its purpose, references the issue, and records unit-test validation. It omits the reviewer checklist, but the required change information is mostly comp…
Full details: Docstring Coverage

Explanation

Docstring coverage is 5.26% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 19 functions across 10 files. (7 skipped: 7 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@grokspawn

Copy link
Copy Markdown
Contributor

I think this approach cannot proceed because it negatively impacts custom ClusterCatalogs from users which express a catalog-version. In this mode, a rollback may be desired/required and the feature provides no capability to override the new behavior.

/hold

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 1, 2026
// greater version before its content can replace the currently served catalog.
// +kubebuilder:validation:Minimum:=1
// +optional
CatalogVersion int64 `json:"catalogVersion,omitempty"`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have to record this in the API. It could just be stored and read from the cache. Something we should discuss more.

Also, I think monotonic increase checks should only apply if the spec's reference is unchanged.


fsys, canonicalRef, unpackTime, err := r.ImagePuller.Pull(ctx, catalog.Name, catalog.Spec.Source.Image.Ref, r.ImageCache)
catalogPuller, ok := r.ImagePuller.(imageutil.CatalogPuller)
if !ok {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't type assert here. That breaks the abstraction. We may need to think about extending the source interface or adding a new, optional SourcePolicy interface/implementation that abstractly validates an update. That would also keep this function much cleaner.

@grokspawn

Copy link
Copy Markdown
Contributor

Cribbing from ongoing slack convo for notes:

  1. needs to have optionality/override mechanisms so it does not impact folks where monotonic reversion is acceptable/desirable
  2. should not be activated by the presence of the attribute which it leverages to identify progression; instead some ClusterCatalog attribute should control feature enforcement for the specified catalog; optionality/override is then explicit
  3. the label that is leveraged to implement the progression should be documented to be available to all users

// +kubebuilder:validation:XValidation:rule="self.find('(@.*:)') != \"\" ? self.find(':.*$').matches(':[0-9A-Fa-f]*$') : true",message="digest is not valid. the encoded string must only contain hex characters (A-F, a-f, 0-9)"
Ref string `json:"ref"`
// catalogVersion is the monotonic version assigned to the catalog publication by the image publisher.
// It is read from the OCI image config label "olm.operatorframework.io/catalog-version".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need something new here. I think we just need to look at the build timestamp that already exists in OCI image configs (at least until we end up using a custom OCI artifact, but that's another whole thing that's out-of-scope here)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants