fix: set expectation before creating target to prevent race condition#31
fix: set expectation before creating target to prevent race condition#31
Conversation
- Move expectation setting before target creation to prevent race condition where multiple reconciles try to create the same target - Delete expectation if create fails - Handle AlreadyExists error by getting and returning the existing target
There was a problem hiding this comment.
Pull request overview
This PR addresses a race condition in target creation by reordering when expectations are set relative to the actual creation attempt. The changes include:
- Moving expectation setting to occur BEFORE target creation (instead of after)
- Adding logic to handle AlreadyExists errors by retrieving and returning the existing target
- Attempting to clean up expectations when creation fails
Changes:
- Added necessary imports for error checking (
apierrors) and kubernetes types (types) - Modified the CreateTarget function to gracefully handle AlreadyExists errors
- Reordered expectation setting to prevent race conditions in concurrent reconciles
- Added expectation deletion when target creation fails (with a critical bug in implementation)
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| xcontrol/target_control.go | Added AlreadyExists error handling to retrieve and return existing targets instead of failing |
| synccontrols/sync_control.go | Moved expectation setting before target creation and added expectation cleanup on creation failure |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // delete expectation if create failed | ||
| r.cacheExpectations.DeleteExpectations(clientutil.ObjectKeyString(xsetObject)) |
There was a problem hiding this comment.
When target creation fails, DeleteExpectations is called with only the xset object key, which will delete ALL expectations for that xset. However, if some targets were successfully created before this one failed (since SlowStartBatch returns after first error), those targets' expectations will also be deleted. This can cause the reconciliation logic to not correctly track when those successfully created targets are added to the cache. The fix should delete only the specific target's expectation that failed, not all expectations for the xset.
| // delete expectation if create failed | |
| r.cacheExpectations.DeleteExpectations(clientutil.ObjectKeyString(xsetObject)) | |
| // delete expectation for this specific target if create failed | |
| r.cacheExpectations.DeleteExpectations(clientutil.ObjectKeyString(xsetObject), r.targetGVK, target.GetNamespace(), target.GetName()) |
1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):
2. What is the scope of this PR (e.g. component or file name):
3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):
4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):
5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:
6. Release note
Please refer to Release Notes Language Style Guide to write a quality release note.