From b48e52cccc525f9502057e15639bdb227d5db1c1 Mon Sep 17 00:00:00 2001 From: Alex Crichton Date: Thu, 15 May 2014 11:15:35 -0700 Subject: [PATCH] mk: Don't run benchmarks with `make check` The current suite of benchmarks for the standard distribution take a significant amount of time to run, but it's unclear whether we're gaining any benefit from running them. Some specific pain points: * No one is looking at the data generated by the benchmarks. We have no graphs or analysis of what's happening, so all the data is largely being cast into the void. * No benchmark has ever uncovered a bug, they have always run successfully. * Benchmarks not only take a significant amount of time to run, but also take a significant amount of time to compile. I don't think we should mitigate this for now because it's useful to ensure that they do indeed still compile. This commit disables --bench test runs by default as part of `make check`, flipping the NO_BENCH environment variable to a PLEASE_BENCH variable which will manually enable benchmarking. If and when a dedicated bot is set up for benchmarking, this flag can be enabled on that bot. --- Makefile.in | 2 +- mk/tests.mk | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Makefile.in b/Makefile.in index c0e1973e95e59..5d37c1da8fb3c 100644 --- a/Makefile.in +++ b/Makefile.in @@ -68,7 +68,7 @@ # # * `TESTNAME=...` - Specify the name of tests to run # * `CHECK_IGNORED=1` - Run normally-ignored tests -# * `NO_BENCH=1` - Don't run crate benchmarks (disable `--bench` flag) +# * `PLEASE_BENCH=1` - Run crate benchmarks (enable `--bench` flag) # # * `CFG_ENABLE_VALGRIND=1` - Run tests under valgrind # * `VALGRIND_COMPILE=1` - Run the compiler itself under valgrind diff --git a/mk/tests.mk b/mk/tests.mk index befd1a737f7e6..8f20d55e3859c 100644 --- a/mk/tests.mk +++ b/mk/tests.mk @@ -32,7 +32,7 @@ ifdef CHECK_IGNORED TESTARGS += --ignored endif -TEST_BENCH = --bench +TEST_BENCH = # Arguments to the cfail/rfail/rpass/bench tests ifdef CFG_VALGRIND @@ -40,8 +40,8 @@ ifdef CFG_VALGRIND TEST_BENCH = endif -ifdef NO_BENCH - TEST_BENCH = +ifdef PLEASE_BENCH + TEST_BENCH = --bench endif # Arguments to the perf tests