Avoid full repository scans for hotfile replication jobs#8144
Open
greenc-FNAL with Copilot wants to merge 12 commits into
Open
Avoid full repository scans for hotfile replication jobs#8144greenc-FNAL with Copilot wants to merge 12 commits into
greenc-FNAL with Copilot wants to merge 12 commits into
Conversation
Copilot
AI
changed the title
[WIP] Fix hotfile replication migration tasks in INITIALIZING state
Avoid full repository scans for hotfile replication jobs
Jul 8, 2026
Contributor
|
This PR has been built successfully with tests on a local node; CI in-progress. |
There was a problem hiding this comment.
Pull request overview
This PR optimizes hotfile-triggered migration jobs to avoid the generic startup path that scans the entire pool repository, keeping hotfile replication scoped to the specific file that crossed the threshold.
Changes:
- Switch hotfile-triggered jobs to start from an already-resolved
CacheEntry(no full repository scan). - Add
Job.start(Iterable<CacheEntry>)to support synchronous startup with preselected entries, while keeping the existing asyncstart()behavior for scan-based jobs. - Add a regression test asserting hotfile startup does not invoke
Repository.iterator().
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| modules/dcache/src/test/java/org/dcache/pool/migration/MigrationModuleTest.java | Adds a regression test to ensure hotfile job startup does not iterate the repository. |
| modules/dcache/src/main/java/org/dcache/pool/migration/MigrationModule.java | Uses the new Job.start(Iterable<CacheEntry>) overload for hotfile-triggered jobs. |
| modules/dcache/src/main/java/org/dcache/pool/migration/Job.java | Factors initialization bookkeeping and introduces a synchronous start path that enqueues known entries without scanning. |
Motivation: The javadoc for the synchronous Job.start(Iterable<CacheEntry>) variant was brief and did not clearly explain the method's purpose, typical callers, or the synchronous blocking behavior. Modification: Expanded the javadoc to clarify that this method bypasses the generic repository scan, is typically used by hot-file replication, and blocks the calling thread until all entries are examined and queued. Added @param and @see tags for better cross-referencing with the asynchronous start() method. Result: Developers reading the code now have a clearer understanding of when and why to use this synchronous startup variant versus the asynchronous one. Target: master Requires-notes: no Requires-book: no
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2fc16bb to
5e80381
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hotfile replication jobs were being created through the generic migration startup path, which performs a repository-wide initialization scan before the job can leave
INITIALIZING. On large pools, that made single-file hotfile jobs unnecessarily expensive and could leave some of them stuck in that startup state.Hotfile startup path
CacheEntryinstead of invoking the generic full-repository initialization path.Job initialization API
Job.start(Iterable<CacheEntry>)overload for callers that already know the exact entries to enqueue.start()behavior for normal migration jobs that still need a repository scan.Initialization handling
Regression coverage
Example of the behavioral change: