worker: Clone index repository in the background #7461
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Turbo87
added
C-internal 🔧
LawnGnome
approved these changes
Nov 7, 2023
|
☔ The latest upstream changes (presumably #7459) made this pull request unmergeable. Please resolve the merge conflicts. |
… background This commit moves us back to the previous behavior, where the index repository is cloned on startup. But compared to the previous behavior, the cloning is now happening in a background thread so that any background jobs that don't need the repository can run uninterrupted.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are currently delaying the execution of any tasks until the index repository is cloned. This was useful while we were mostly running tasks that needed the repository anyway, but with the introduction of the sparse index, this is no longer the case.
This PR adjusts the code to lazily clone the index when requested, but then also fire off a background thread to start that operation once the background worker has started.
This should allow us to process sparse index updates while the git repository is still being cloned.
Note that due to the way the background worker is implemented this is still not entirely true. While the repository is being cloned a
Mutexis being held. At some point all five background worker threads might be handlingsync_to_git_indexjobs with all of them waiting to lock that mutex. At this point no other jobs can run anymore since the workers are busy. This can be solved in the future by having multiple queues, with a dedicated queue for jobs that involve the git repository.