You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/release/rollups.md
+69-28Lines changed: 69 additions & 28 deletions
Original file line number
Diff line number
Diff line change
@@ -42,10 +42,13 @@ queue has been merged.
42
42
43
43
2. Run the following command in the pull request thread:
44
44
45
-
```
46
-
@bors r+ rollup=never p=5
47
-
````
48
-
where 5 is the number of PRs contained in your rollup.
45
+
```
46
+
@bors r+ rollup=never p=5
47
+
````
48
+
49
+
Tools and subtree syncs should use `p=5`, like rollups, so they interleave between rollups, as
50
+
tools and subtree syncs are typically more tricky to fix than rebasing PRs.
51
+
49
52
3. If the rollup fails, use the logs rust-log-analyzer
50
53
provides to bisect the failure to a specific PR and
51
54
`@bors r-` the PR. If the PR is running, you need to do `@bors r- retry`. Otherwise,
@@ -63,15 +66,22 @@ queue has been merged.
63
66
64
67
## Selecting Pull Requests
65
68
66
-
The queue is sorted by rollup status. In general, a good rollup contains a bunch of `maybe` (unmarked) PRs, and a large pile of `always` PRs. You can include one or two `iffy` PRs if you are confident that they will pass.
67
-
If you have a lot of time, you can also make a rollup just from `iffy` PRs (if there are enough of them) and weed out the failures one by one.
68
-
A rollup should never include `rollup=never` PRs.
69
+
The queue is sorted by rollup status. In general:
70
+
71
+
- A good rollup contains a bunch of `maybe` (unmarked) PRs, and a large pile of `always` PRs. You
72
+
can include one or two `iffy` PRs to amortize the cost of testing full CI.
73
+
- If you have a lot of time, you can also make a rollup just from `iffy` PRs (if there are enough of
74
+
them) and weed out the failures one by one.
75
+
- A rollup must **never** include `rollup=never` PRs.
69
76
70
77
The actual absolute size of the rollup can depend based on experience and current length of the queue.
71
78
People new to making rollups might start with including 1 `iffy`, 2 `maybe`s, and 4 `always`s. Usually 6-8 PRs per rollup is a good compromise.
72
79
There is rarely a need to roll up more than 10 PRs at once (unless there are >30 PRs waiting the queue), keep in mind that we also try to minimize regressions per merge.
73
80
74
-
Don't try to make mega-rollups (15-20 PRs that merge half or more of the entire queue all at once) to keep the number of perf or bug regressions per merge as low as possible and keep potential regressions [bisectable].
81
+
Limit the size of rollups, *even if* the queue is backed up. Large rollups run the risk of failing
82
+
or merge conflicts, and smaller rollups keep potential regressions [bisectable]. On average, rollups
83
+
are 7 PRs large, often varying from 5 to 10 depending on the number of `rollup=always` PRs that can
84
+
be included.
75
85
76
86
Don't hesitate to downgrade the rollup status of a PR! If your intuition tells you that a `rollup=always` PR has some chances for failures, mark it `rollup=maybe` or better `rollup=iffy`. A lot of the unmarked `maybe` PRs are categorized as such because the reviewer may not have considered rollupability, so it's always worth picking them with a critical eye. Similarly, if a PR causes your rollup to fail, it's worth considering changing its rollup status.
77
87
@@ -84,38 +94,69 @@ If an `iffy` PR keeps failing in a rollups, it should be marked `never` to preve
84
94
It should be noted which runner the PR failed on, to run this runner as a `try-job` job and make sure it succeeds there before another merge is attempted (example on syntax [here]).
85
95
In general, if possible, try to test a failed PR via a handful of carefully selected try-jobs instead of having to run the full battery of all 60 runners on if it's likely a PR may fail again.
86
96
87
-
To not have `never` or `iffy` PRs stuck in the queue indefinitely, it is recommended to alternate between rollup and non-rollup prs, so one `never`, one rollup, one `iffy`, one `rollup`, one `never` etc, until most of the `never`s are merged.
88
-
If you are the only person making rollups, you can also leave a couple of `never`/`iffy`s for a time where you know nobody will be doing rollups actively, or for weekends which generally see a lower number of PR approvals.
97
+
To not have `never` or `iffy` PRs stuck in the queue indefinitely, it is recommended to alternate
98
+
between rollup and non-rollup prs, so one `never`, one rollup, one `iffy`, one `rollup`, one `never`
99
+
etc, until most of the `never`s are merged. You can selectively bump the priority on certain
100
+
`iffy`/`never` PRs to `p=1` (or even `p=5`) if you want to interleave them between rollups.
89
101
90
-
Try to be fair with rollups: Rollups are a way for things to jump the queue. For `rollup=maybe` PRs, try to include the oldest one (at the top of the section) so that newer PRs aren't jumping the queue over older PRs entirely. You don't have to include every PR older than PRs included in your rollup, but try to include the oldest. Similar to the perspective around `iffy`, it's useful to look at a rollup as a way for other PRs to piggyback on the CI cycle of the oldest PR in queue.
91
-
Very old (several months) or very large PRs that are extremely prone to merge conflicts may also be given a slight priority bump (`p=1`) to finally get them out of the queue without having to rebase them repeatedly.
92
-
Ultimately, we want to keep the number of regressions per merge at a minimum while also minimizing the amount of time between approval and the final merge of a PR, to avoid unnecessary merge conflicts and rebases.
102
+
If you are the only person making rollups, you can also leave a couple of `never`/`iffy`s for a time
103
+
where you know nobody will be doing rollups actively, or for weekends which generally see a lower
104
+
number of PR approvals.
93
105
106
+
Try to be fair with rollups: Rollups are a way for things to jump the queue. For `rollup=maybe` PRs,
107
+
try to include the oldest one (at the top of the section) so that newer PRs aren't jumping the queue
108
+
over older PRs perpetually. You don't have to include every PR older than PRs included in your
109
+
rollup, but try to include the oldest. Similar to the perspective around `iffy`, it's useful to look
110
+
at a rollup as a way for other PRs to piggyback on the CI cycle of the oldest PR in queue.
111
+
112
+
Very old (several months) or very large PRs that are extremely prone to merge conflicts may also be
113
+
given a priority bump (`p=5`) to finally get them out of the queue without having to rebase them
114
+
repeatedly, to have the same priority as rollups or sandwich between rollups.
115
+
116
+
Ultimately, we want to keep the number of regressions per merge at a minimum while also minimizing
117
+
the amount of time between approval and the final merge of a PR, to avoid unnecessary merge
118
+
conflicts and rebases.
94
119
95
120
## Failed rollups
121
+
96
122
If the rollup has failed, run the `@bors retry` command if the
97
123
failure was spurious (e.g. due to a network problem or a timeout).
98
-
There may be a matching `CI-spurious-fail-.*` label that you can use to tag the PR as such, to help discover common fail patterns.
99
-
If it wasn't spurious, find the offending PR and throw it out by copying a link to the rust-logs-analyzer comment,
100
-
and leaving a comment like `Failed in <link_to_comment>, @bors r-`.
101
-
In case the log-analyzer does not give any meaningful output, you can directly open the ci-logs (the `(web)` link), find the point where the error was thrown
102
-
and directly copy the URL to the respective line in the log output.
103
-
Hopefully, the author or reviewer will give feedback to get the PR fixed or confirm that it's not
104
-
at fault. The failed rollup PR should then be closed.
124
+
There may be a matching `CI-spurious-fail-.*` label that you can use to tag the PR as such, to help
125
+
discover common fail patterns.
126
+
127
+
If it wasn't spurious, find the offending PR and return it to the author to be fixed by copying a
128
+
link to the `rust-log-analyzer` comment, and leaving a comment like
129
+
130
+
```text
131
+
Failed in <link_to_comment>.
132
+
@bors r-`.
133
+
```
134
+
135
+
In case `rust-log-analyzer` does not give any meaningful output, you can directly open the ci-logs
136
+
(the `(web)` link), find the point where the error was thrown and directly copy the URL to the
137
+
respective line in the log output. Hopefully, the author or reviewer will give feedback to get the
138
+
PR fixed or confirm that it's not at fault. The failed rollup PR should then be closed.
105
139
106
140
Once you've removed the offending PR, recreate your rollup without it (see 1.).
107
141
Merge one batch of PRs by throwing out the failures one by one instead of adding new PRs to it, as this may introduce additional points of failure.
108
142
109
-
Sometimes however, it is hard to find the offending PR. If so, use your intuition
110
-
to avoid the PRs that you suspect are the problem and recreate the rollup.
111
-
Another strategy is to raise the priority of the PRs you suspect,
112
-
mark them as `rollup=never` (or `iffy`) and let bors test them standalone to dismiss
113
-
or confirm your hypothesis, or split the rollup into 2 smaller ones until are certain of the failure cause. If a PR was found to be the cause and other PRs were "wrongfully" `iffy`'d, they can of course be reprioritised as `maybe` again.
143
+
Sometimes it is hard to find the offending PR. There are usually three strategies to figure out the
144
+
offending PR:
145
+
146
+
1.**Avoid the problem**: Use your intuition to avoid the PRs that you suspect are the problem and
147
+
recreate the rollup.
148
+
2.**Test suspected PRs alone**: Raise the priority of the PRs you suspect, mark them as
149
+
`rollup=never` (or `iffy`) and let bors test them standalone to dismiss or confirm your
150
+
hypothesis.
151
+
3.**Divide and conquer**: Split rollup into 2 smaller ones until you are certain of the failure
152
+
cause. If a PR was found to be the cause and other PRs were "wrongfully" marked `iffy`, they can
153
+
of course be marked as `maybe` again with `@bors rollup=maybe` or `@bors rollup-`.
154
+
155
+
If a PR in a rollup continues to fail you can run the `@bors rollup=never` command to ensure the PR
156
+
gets tested independently, since it's likely it will fail again in the future.
114
157
115
-
If a PR in a rollup continues to fail you can run the `@bors rollup=never` command to
0 commit comments