Enforce in bootstrap that check must have stage at least 1 #143048

Kobzol · 2025-06-26T08:01:58Z

This PR is another step towards https://rust-lang.zulipchat.com/#narrow/channel/326414-t-infra.2Fbootstrap/topic/Proposal.20to.20cleanup.20stages.20and.20steps.20after.20the.20redesign/with/523586917, this time dealing with x check.

It enforces the invariant that:

We check std stage N with rustc stage N
We check everything else stage N with rustc stage N - 1

It creates a single function that prepares a proper build compiler for checking something, and also adds snapshot tests for various common check steps. Some obsolete code was also removed.

The default check stage also becomes 1, for all profiles. I tested manually that x check std with download-ci-rustc still works and doesn't build rustc locally.

Subsumes #139170.

r? @ghost

Kobzol · 2025-06-26T08:04:11Z

CC @jieyouxu (one of many heaps of bootstrap fixup commits I have in store 😆)

bors · 2025-06-26T10:22:00Z

☔ The latest upstream changes (presumably #141899) made this pull request unmergeable. Please resolve the merge conflicts.

Kobzol · 2025-06-26T14:31:27Z

Cross-compilation checks are now doing too much work, I need to take a look at how we can reduce that, because x check --stage1 with cross-compilation is no longer a thing.

bors · 2025-07-01T03:25:36Z

☔ The latest upstream changes (presumably #143254) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot · 2025-07-01T15:44:03Z

This PR modifies src/bootstrap/defaults.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

Kobzol · 2025-07-01T15:44:13Z

r? @jieyouxu

rustbot · 2025-07-01T15:44:16Z

jieyouxu is not on the review rotation at the moment.
They may take a while to respond.

bors · 2025-07-01T22:49:32Z

☔ The latest upstream changes (presumably #143287) made this pull request unmergeable. Please resolve the merge conflicts.

jieyouxu

I like this a lot in general, it does feel much more self-consistent and coherent.

I left some questions/discussions for things that I wanted to double-check, but overall looks good!

src/bootstrap/src/core/build_steps/check.rs

jieyouxu · 2025-07-02T05:44:55Z

src/bootstrap/src/core/builder/mod.rs

+        self.stage.or(self
+            .built_by
+            .map(|compiler| if self.name == "std" { compiler.stage } else { compiler.stage + 1 }))


Discussion: this makes me feel a bit uneasy, but I guess the logic here is: anything that's not the standard library should be considered a non-library artifact, and should be considered a product of the stage n (compiler, standard library) pair (and so gets numbered as the next stage)? I.e. reading back https://hackmd.io/QXj6LqntQTKPgm3uBuE-aA again, the core model is

stage corresponds to what gets built, not what is used to build it

(The following is me trying to make sure I understand this)

Let's call the (compiler, library) pair¹²
$$\mathbf{CLP}_{\mathrm{stage}} := \left\langle \text{compiler}_{\mathrm{stage}}, \text{library}_{\mathrm{stage}} \right\rangle$$
where the separation between stages is delineated by which $\mathrm{compiler}$ gets built. Then, by design, because the standard library is "special" and tightly coupled with the compiler, we fix the immediate standard library built by $\mathrm{compiler}_{\mathrm{stage}}$ to be of the same $\mathrm{stage}$, i.e. $\mathrm{library}_{\mathrm{stage}}$.

With the stage 0 redesign, we now have a nice straightforward "derivation chain" ($\longrightarrow$; artifacts can be used to build new artifacts) property where
$$\left\langle \text{compiler}_{\mathrm{stage}}, \text{library}_{\mathrm{stage}} \right\rangle \longrightarrow \left\langle \text{compiler}_{\mathrm{stage + 1}}, \text{library}_{\mathrm{stage + 1}} \right\rangle$$
actually holds even for $\mathrm{stage} = 0$.

However, for:

bootstrap tools
Tools that are unconditionally built by the stage 0 $\mathbf{CLP}$, i.e. always depends on $\mathbf{CLP}_{0}$. Their stage numbering will be called $\mathrm{BootstrapTool}_{1}$, at least in the short-term.

host rustc tools
Tools that depend on rustc_private and so on $\mathrm{compiler}_{\mathrm{stage}}$. This means that they will be numbered as $\mathrm{HostRustcTool}_{\mathrm{stage} + 1}$.

std tools
Tools that depend on the whole $\mathbf{CLP}_{\mathrm{stage}}$, i.e. both $\mathrm{compiler}_{\mathrm{stage}}$, and $\mathrm{library}_{\mathrm{stage}}$. Similiarly, this means that they will be numbered as $\mathrm{StdTool}_{\mathrm{stage} + 1}$.

They then follow the general "what gets built" model (?).

Footnotes

I can't come up with a better name, it's not exactly a toolchain, neither is it a "complete" sysroot I think ↩

Not to be confused with the Chilean Peso. ↩

src/bootstrap/src/core/builder/tests.rs

src/bootstrap/src/core/build_steps/check.rs

jieyouxu · 2025-07-02T08:00:08Z

src/bootstrap/src/core/build_steps/check.rs

            builder.std(build_compiler, host);
-            builder.std(build_compiler, target);
+            builder.ensure(Std::new(build_compiler, target));


Question: wait, why is this different? For $\mathrm{stage} \ge 1$, isn't this equivalent? I.e.

rust/src/bootstrap/src/core/builder/mod.rs

Lines 1393 to 1409 in f51c987

if compiler.stage == 0 {

if target != compiler.host {

panic!(

r"It is not possible to build the standard library for `{target}` using the stage0 compiler.

You have to build a stage1 compiler for `{}` first, and then use it to build a standard library for `{target}`.

",

compiler.host

)

}

// We still need to link the prebuilt standard library into the ephemeral stage0 sysroot

self.ensure(StdLink::from_std(Std::new(compiler, target), compiler));

} else {

// This step both compiles the std and links it into the compiler's sysroot.

// Yes, it's quite magical and side-effecty.. would be nice to refactor later.

self.ensure(Std::new(compiler, target));

}

At this point, wouldn't we have already rejected invoking ./x check flow with $\mathrm{stage} = 0$?

Equivalent to what? :) I changed it just to this:

let build_compiler = builder.compiler(builder.top_stage, host); builder.ensure(Std::new(build_compiler, target)); build_compiler

because I remembered that stdlib's proc macros and build scripts are always compiled using the stage0 libstd. We only need the host stdlib when compiling host code of other modes (yeah, I know..).

Except! We actually need the host stdlib when checking the tool itself (not stdlib). Aaaaargh.

Equivalent to what?

NGL, I don't know what I meant by this myself -- when I click at this comment, it's outdated against something, and I can't tell what it was...

src/bootstrap/src/core/builder/tests.rs

jieyouxu · 2025-07-02T08:05:01Z

src/ci/docker/host-x86_64/mingw-check-1/Dockerfile

@@ -48,7 +48,7 @@ ENV SCRIPT \
           python3 ../x.py build src/tools/build-manifest && \
           python3 ../x.py test --stage 0 src/tools/compiletest && \
           python3 ../x.py check compiletest --set build.compiletest-use-stage0-libtest=true && \
-           python3 ../x.py check --stage 1 --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \
+           python3 ../x.py check --stage 2 --target=i686-pc-windows-gnu --host=i686-pc-windows-gnu && \


Suggestion: actually, can you add a comment for this specifically that we're exercising cross-compile here, so --stage 1 isn't sufficient?

…eyouxu Add bootstrap check snapshot tests Split off from rust-lang#143048, so that we get a baseline of how check behaved before we make changes to it. Note that the output of the check snapshot tests is suboptimal in many places, as we're missing information about stages and the build compiler. That will be changed in rust-lang#143048. r? `@jieyouxu`

Rollup merge of #143316 - Kobzol:bootstrap-check-tests, r=jieyouxu Add bootstrap check snapshot tests Split off from #143048, so that we get a baseline of how check behaved before we make changes to it. Note that the output of the check snapshot tests is suboptimal in many places, as we're missing information about stages and the build compiler. That will be changed in #143048. r? `@jieyouxu`

…fault check stage for it

… tools

rust-log-analyzer · 2025-07-03T16:16:58Z

The job pr-check-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#11 DONE 2.3s

#12 [ 5/11] RUN npm install es-check@6.1.1 eslint@8.6.0 typescript@5.7.3 -g
#12 5.930 npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
#12 5.948 npm WARN deprecated @types/minimatch@6.0.0: This is a stub types definition. minimatch provides its own type definitions, so you do not need this installed.
#12 5.970 npm WARN deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported
#12 5.981 npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
#12 6.018 npm WARN deprecated @humanwhocodes/config-array@0.9.5: Use @eslint/config-array instead
#12 6.045 npm WARN deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported
#12 6.051 npm WARN deprecated @humanwhocodes/object-schema@1.2.1: Use @eslint/object-schema instead
---
#16 3.030 Building wheels for collected packages: reuse
#16 3.031   Building wheel for reuse (pyproject.toml): started
#16 3.245   Building wheel for reuse (pyproject.toml): finished with status 'done'
#16 3.246   Created wheel for reuse: filename=reuse-4.0.3-cp310-cp310-manylinux_2_35_x86_64.whl size=132719 sha256=d2a2565e7037ad3883fb9337653f2e25bbb588534fbef3697286cbc26d1bf634
#16 3.247   Stored in directory: /tmp/pip-ephem-wheel-cache-hb5rgpkr/wheels/3d/8d/0a/e0fc6aba4494b28a967ab5eaf951c121d9c677958714e34532
#16 3.249 Successfully built reuse
#16 3.250 Installing collected packages: boolean-py, binaryornot, tomlkit, reuse, python-debian, markupsafe, license-expression, jinja2, chardet, attrs
#16 3.651 Successfully installed attrs-23.2.0 binaryornot-0.4.4 boolean-py-4.0 chardet-5.2.0 jinja2-3.1.4 license-expression-30.3.0 markupsafe-2.1.5 python-debian-0.1.49 reuse-4.0.3 tomlkit-0.13.0
#16 3.651 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
#16 DONE 3.7s

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 26, 2025

Kobzol force-pushed the bootstrap-check-stage-1 branch from f06e41e to aed5f23 Compare June 26, 2025 08:03

This comment has been minimized.

Sign in to view

bors added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jun 26, 2025

Kobzol force-pushed the bootstrap-check-stage-1 branch from aed5f23 to a36ee24 Compare June 26, 2025 14:29

This comment has been minimized.

Sign in to view

Kobzol mentioned this pull request Jul 1, 2025

Cannot check rustc from master branch on Windows 10: build failure in rustc_codegen_gcc #142754

Open

Kobzol force-pushed the bootstrap-check-stage-1 branch from a36ee24 to 22fb548 Compare July 1, 2025 15:43

Kobzol marked this pull request as ready for review July 1, 2025 15:44

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 1, 2025

rustbot assigned jieyouxu Jul 1, 2025

This comment has been minimized.

Sign in to view

jieyouxu removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 2, 2025

Kobzol mentioned this pull request Jul 2, 2025

Add bootstrap check snapshot tests #143316

Merged

jieyouxu reviewed Jul 2, 2025

View reviewed changes

Kobzol added 3 commits July 3, 2025 17:37

Make default check stage be 1, and error out on checking with stage 0

ee719f7

Remove custom_stage override from check::Std and make 1 be the de…

870bb86

…fault check stage for it

Make build compiler explicit in check::Rustc and check::Std

4b11385

Kobzol added 5 commits July 3, 2025 17:55

Fixup check of rust-analyzer, codegen backends, compiletest and other…

32fa3b5

… tools

Implement CoverageDump checking through the tool_check_step macro

de6a380

Unify selection of build compiler for checking in a single function

aaf6016

Add change tracker entry

f94cc79

Fix CI

fae5ba2

Kobzol force-pushed the bootstrap-check-stage-1 branch from 22fb548 to fae5ba2 Compare July 3, 2025 15:59

	if compiler.stage == 0 {
	if target != compiler.host {
	panic!(
	r"It is not possible to build the standard library for `{target}` using the stage0 compiler.
	You have to build a stage1 compiler for `{}` first, and then use it to build a standard library for `{target}`.
	",
	compiler.host
	)
	}

	// We still need to link the prebuilt standard library into the ephemeral stage0 sysroot
	self.ensure(StdLink::from_std(Std::new(compiler, target), compiler));
	} else {
	// This step both compiles the std and links it into the compiler's sysroot.
	// Yes, it's quite magical and side-effecty.. would be nice to refactor later.
	self.ensure(Std::new(compiler, target));
	}

Enforce in bootstrap that check must have stage at least 1 #143048

Are you sure you want to change the base?

Enforce in bootstrap that check must have stage at least 1 #143048

Conversation

Kobzol commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Jun 26, 2025

Uh oh!

This comment has been minimized.

bors commented Jun 26, 2025

Uh oh!

Kobzol commented Jun 26, 2025

Uh oh!

This comment has been minimized.

bors commented Jul 1, 2025

Uh oh!

rustbot commented Jul 1, 2025

Uh oh!

Kobzol commented Jul 1, 2025

Uh oh!

rustbot commented Jul 1, 2025

Uh oh!

This comment has been minimized.

bors commented Jul 1, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jieyouxu Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jieyouxu Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kobzol Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kobzol Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jieyouxu Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

rust-log-analyzer commented Jul 3, 2025

Uh oh!

Uh oh!

Kobzol commented Jun 26, 2025 •

edited

Loading

jieyouxu Jul 2, 2025 •

edited

Loading

jieyouxu Jul 2, 2025 •

edited

Loading

Kobzol Jul 3, 2025 •

edited

Loading

jieyouxu Jul 4, 2025 •

edited

Loading