I want to run cargo clippy and cargo t for different combinations of features using both stable and some older version of cargo. Assuming Cargo.lock nor target/ exist and both toolchains are already installed, should the order of how I do this affect the overall time? If I had to guess, the following is the list of contributing factors in descending order:
Toolchain
Features
Command
For example, the following sequence is the optimal sequence in terms of minimizing the total amount of time for the whole sequence to complete:
where toolchain_1 and toolchain_2 are toolchains with one of them being the stable toolchain and the other an older version, cmd_1 and cmd_2 are commands with one being clippy and the other being t, and features_1 and features_2 are different sequences (of possibly different lengths) of features where one doesn't imply the other.
Is my guess right? Does it matter what toolchain_1 and toolchain_2 are (i.e., replacing one with stable and the other with <older_version> should have different performance than if they were swapped)? Does it matter what cmd_1 and cmd_2 are (i.e., replacing one with clippy and the other with t should have different performance than if they were swapped)? Does it matter what features_1 and features_2 are (e.g., it's better to use fewer features sooner)?
I'm trying to optimize my CI flow on a single device and am curious if the order in which I do this matters (e.g., if cargo replaces certain caches when a different toolchain is used behooving one to focus on using one toolchain before using a different one in order to re-use the cache).
No; different toolchains will not touch each other’s files at all, at least with recent versions. (I think it used to be the case that they would overwrite each other’s, but I just checked with stable and nightly and they do not cause rebuilding when interleaved.)
Overall, you can run your commands in any order and it won’t invalidate any caches or make any other significant difference, except:
The opposite! You should put --all-featuresfirst, not because of any cache invalidation considerations, but because building all of your dependencies in a single command gives the most opportunities for parallelizing those builds. (You will still see some further builds of dependencies in cases where fewer features are enabled on those dependencies.)
OK, I think I remember overwriting occurring in the past as well; but I wasn't sure if this was a Mandela Effect thing. If it is true, then seeing how I use stable and an older version, it seems that it would be better to use a single toolchain first unless I actually verify the older version isn't too old since otherwise the older version could possibly overwrite the cache.
Ah! Great point. I'm currently running features in arbitrary order, so I'll be sure to run --all-featuresfirst. From there, it sounds like the order could be arbitrary.
Note that if you choose anything else different in addition to the toolchain, like selecting a specific package, you may get a different feature subset or otherwise need to build different things. And RUSTFLAGSdoes invalidate cache always. So, in my experience, it's often the case that when one is typing cargo +nightly for some special purpose, it will need to build many packages, even if there isn’t any overwriting going on.
From my experience, it is possible, because I encountered in a project that cargo cache was messed up when multiple rustc toolchain versions gated behind features were switching, and the only solution is cargo clean and rebuild. But that is probably an issue from incremental compilation.
But as the author of another project os-checker (see its deepwiki for architecture description in English), I just suggest grouping your cmds and toolchains. os-checker runs some academic static analysis tools on several hundred os-related crates, the tools requires distinct toolchains, and in early development these check cmds (combining checkers, toolchains, --target, and --features etc) were just randomly executed in parallel (or in single thread), then I got all kinds of issues where cargo just did not work. A notable and infamous issue is Rustup (including proxies) is not safe for concurrent use · Issue #988 · rust-lang/rustup · GitHub
Finally, these grouped commands are just sequentially run in single thread and cargo cache is cleaned once all cmds on the same checker are done: it's the safest strategy and I'm satisfied.
Thanks for the input. Your crate looks neat. It seems to be a more robust version of a CLI tool I'm writing which only focuses on features and not other aspects like platforms for both stable and whatever MSRV is defined in Cargo.toml. I've been affected by a fair number of crates (including popular ones) that don't even compile when certain features are enabled, and I didn't want that to happen to my own crates[1].
Obviously features have an exponential effect on the quantity of tests that need to be run, so it's not surprising that a fair amount of crates aren't actually tested for each possible set of features. My crate is designed primarily for local use and only to be run on a single machine, so the concurrency issues of rustup don't affect me but it's useful knowledge to have nonetheless. Not to mention it needs to run on platforms that don't support rustup (e.g., OpenBSD); although on such platforms you lose the ability to run the tests using the MSRV and stable toolchains but instead only whatever toolchain cargo uses.
Anyway, my strategy is to do everything with stable first then the MSRV. Within a given toolchain, I'll run cargo clippy then cargo t for each set of features ensuring that the first set of features corresponds to --all-features.
Compilation errors due to compile_error are exempt since it's not uncommon for crates to require/forbid certain features. ↩︎
