Skip to content

fix: set expectation before creating target to prevent race condition#31

Closed
AnnaYue wants to merge 1 commit intomainfrom
fix/race-condition-target-creation
Closed

fix: set expectation before creating target to prevent race condition#31
AnnaYue wants to merge 1 commit intomainfrom
fix/race-condition-target-creation

Conversation

@AnnaYue
Copy link
Copy Markdown
Contributor

@AnnaYue AnnaYue commented Mar 20, 2026

  • Move expectation setting before target creation to prevent race condition where multiple reconciles try to create the same target
  • Delete expectation if create fails
  • Handle AlreadyExists error by getting and returning the existing target

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

  • N
  • Y

2. What is the scope of this PR (e.g. component or file name):

3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Other

4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

  • N
  • Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

  • Unit test
  • Integration test
  • Benchmark (add benchmark stats below)
  • Manual test (add detailed scripts or steps below)
  • Other

6. Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

- Move expectation setting before target creation to prevent race condition
  where multiple reconciles try to create the same target
- Delete expectation if create fails
- Handle AlreadyExists error by getting and returning the existing target
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a race condition in target creation by reordering when expectations are set relative to the actual creation attempt. The changes include:

  1. Moving expectation setting to occur BEFORE target creation (instead of after)
  2. Adding logic to handle AlreadyExists errors by retrieving and returning the existing target
  3. Attempting to clean up expectations when creation fails

Changes:

  • Added necessary imports for error checking (apierrors) and kubernetes types (types)
  • Modified the CreateTarget function to gracefully handle AlreadyExists errors
  • Reordered expectation setting to prevent race conditions in concurrent reconciles
  • Added expectation deletion when target creation fails (with a critical bug in implementation)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
xcontrol/target_control.go Added AlreadyExists error handling to retrieve and return existing targets instead of failing
synccontrols/sync_control.go Moved expectation setting before target creation and added expectation cleanup on creation failure

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +593 to +594
// delete expectation if create failed
r.cacheExpectations.DeleteExpectations(clientutil.ObjectKeyString(xsetObject))
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When target creation fails, DeleteExpectations is called with only the xset object key, which will delete ALL expectations for that xset. However, if some targets were successfully created before this one failed (since SlowStartBatch returns after first error), those targets' expectations will also be deleted. This can cause the reconciliation logic to not correctly track when those successfully created targets are added to the cache. The fix should delete only the specific target's expectation that failed, not all expectations for the xset.

Suggested change
// delete expectation if create failed
r.cacheExpectations.DeleteExpectations(clientutil.ObjectKeyString(xsetObject))
// delete expectation for this specific target if create failed
r.cacheExpectations.DeleteExpectations(clientutil.ObjectKeyString(xsetObject), r.targetGVK, target.GetNamespace(), target.GetName())

Copilot uses AI. Check for mistakes.
@AnnaYue AnnaYue closed this Mar 20, 2026
@AnnaYue AnnaYue deleted the fix/race-condition-target-creation branch March 20, 2026 09:26
@github-actions github-actions bot locked and limited conversation to collaborators Mar 20, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants