Extremely slow compile times in test suite

I'm looking for some help debugging why the compile times in the mongodb crate's test suite are so slow. For reference, a single incremental change can take a minute or longer, even when running just cargo check --tests. The crate itself compiles really quickly--the issue is only with the tests.

The output of compiling with -Zself-profile indicates that nearly the entire time (~1m) is spent in evaluate_obligation, but I can't figure what this implies or how to possibly reduce this.

A few possible culprits that I've investigated so far:

  • the tests make heavy use of #[cfg_attr(...)] to conditionally select either #[tokio::test] or #[async_std::test]. Removing all usages of this and defaulting to just #[tokio::test] has an insignificant impact on compile times
  • the tests also make heavy use of a function that accepts a closure that returns a future. According to this thread, such functions can cause type length / compilation time issues, but converting it to a macro that calls a function directly didn't seem to change much. Perhaps that suffers from a similar issue?
  • We generate a lot of builders via the typed-builder crate. Perhaps using them in the tests leads to long compilation times due to the generated builders having lots of generic parameters.
  • this compiler issue related to async and compilation times, though someone posted their profile output and evaluate_obligation is not a factor in it.
  • most of our tests are in src/test rather than tests, not sure if this has any impact

Anyways, I'm not sure where to go from here, and I was wondering if anyone else has encountered similar issues in the past or knows how I could further debug this to identify the root cause. Thanks!

Here are the top few results from the self-profile output:

+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+
| Item                                             | Self time | % of total time | Time     | Item count | Incremental load time |
+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+
| evaluate_obligation                              | 58.34s    | 85.300          | 58.39s   | 110480     | 0.00ns                |
+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+
| expand_crate                                     | 3.54s     | 5.179           | 3.59s    | 1          | 0.00ns                |
+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+
| typeck                                           | 2.74s     | 4.011           | 61.39s   | 1159       | 71.43ms               |
+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+
| self_profile_alloc_query_strings                 | 979.78ms  | 1.433           | 980.49ms | 1          | 0.00ns                |
+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+
| hir_lowering                                     | 204.80ms  | 0.299           | 204.80ms | 1          | 0.00ns                |
+--------------------------------------------------+-----------+-----------------+----------+------------+-----------------------+

There's probably some accidentally-quadratic check in there. It has happened before.

File a rust-lang issue about this, and developers will hopefully guide you how to narrow it.

Filed https://github.com/rust-lang/rust/issues/87012, thank you!

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.