Consider something like this:
pub BigEnum {
.. 2000 entries ... }
impl BigEnm {
pub fn to_i32(&self) -> { ... 2000 line match statement }
pub const all: [BigEnum; 2000] = [ ... list all 2000 cominations ... ]
Then, suppose this crates is included 125 times in the final crate (think 3 levels of diamond shape crate dependency, each with branching factor of 5).
Are there some weird interactions where this might cause a 5s increase in cargo build time ? (release mode)
===
Context: I have a separate crate that builds a textured atlas. I decided on the "brilliant" idea of: hey, for ultra type safe, let's have the atlas program generate a "Texture_Atlas_Enum.rs" which statically lists out all the valid textures embedded in the atlas.
This enum turns out to have ~ 2000 entries, and appears to have exploded the compile time.
Not sure about the enum size, but this should not have any difference at all - crate is compiled only once anyway (if you're using it as a dependency and not via include!ing, of course).
I am not using include!, it is one texture_atlas_gen.rs file, which is in one crate; and the crate is included many times (as a dependency).
I know very little about the Rust compile process, so this might come off as stupid: even if the crate is only compiled once, is it possible, on the wasm32-unknown-unknown target, that it is linked more than once, and then resolving the multiple copies during linking causes compile time increase ?
Please correct me if I'm wrong, as this is based on hearsay, not any hard evidence: code gen units can NOT split individual files right ? So one down side with 'giant' files is that even with incremental = true, codegen-units = 256, a single giant file can increase the overall codegen time. (I'm looking at the output of cargo build --release --timings, and it's codegen dominated).
That's not my understanding of how codegen units work.
The idea is that the compiler will parse and typecheck all your files, then try to split all the items up into n buckets so they can be compiled by LLVM in parallel. I'm sure there are heuristics to the way this splitting happens to improve efficiency (e.g. a copy of each generic function must be passed to each codegen unit that uses it, so maybe try to put all the functions that use Vec::<String>::push() in the same codegen unit), but in general I doubt the distribution of codegen units can be relied on.
It might be a good idea to use the measureme tool and rustc's self-profiling functionality to see which parts of the compilation process are slow. There's a good chance codegen units have nothing to do with this situation, and profiling your build will give you concrete data instead of us just guessing as to what is going on.
1 Like
Michael-F-Bryan:
It might be a good idea to use the measureme tool and rustc's self-profiling functionality to see which parts of the compilation process are slow. There's a good chance codegen units have nothing to do with this situation, and profiling your build will give you concrete data instead of us just guessing as to what is going on.
I ran:
cargo +nightly rustc -- -Z self-profile
~/.cargo/bin/summarize summarize client_w-1133789.mm_profdata
I am NOT sure if I am using this tool correctly, because the timings the tool output does NOT add up to the actual runtime. I have also dropped all the lines that took less than 100 micro seconds.
This particular seems to only report timings for the top most crate instead of a summary of all the dependencies (useless in my case, as the top most crate just includes other crates). Is there a way to have it summarize the entire 'cargo build' process?
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| Item | Self time | % of total time | Time | Item count | Incremental result hashing time |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| run_linker | 209.07ms | 63.809 | 209.07ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| LLVM_module_codegen_emit_obj | 20.73ms | 6.326 | 20.73ms | 11 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_exported_symbols | 19.14ms | 5.841 | 19.14ms | 104 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| LLVM_passes | 12.31ms | 3.758 | 12.31ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_register_crate | 7.69ms | 2.347 | 31.84ms | 104 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| codegen_module | 5.79ms | 1.767 | 16.33ms | 10 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| upstream_monomorphizations | 5.77ms | 1.760 | 29.69ms | 1 | 3.99ms |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| exported_symbols | 4.98ms | 1.520 | 63.11ms | 105 | 4.91ms |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| free_global_ctxt | 2.96ms | 0.904 | 2.96ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| incr_comp_intern_dep_graph_node | 2.25ms | 0.686 | 3.33ms | 8487 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| expand_crate | 1.80ms | 0.549 | 10.47ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| implementations_of_trait | 1.70ms | 0.518 | 2.42ms | 1352 | 1.03ms |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| codegen_crate | 1.57ms | 0.480 | 18.58ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_impl_trait_ref | 1.52ms | 0.464 | 1.52ms | 446 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| LLVM_module_optimize | 1.23ms | 0.375 | 1.23ms | 11 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| incr_comp_encode_dep_graph | 1.10ms | 0.336 | 1.10ms | 8534 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| layout_of | 1.04ms | 0.318 | 11.61ms | 209 | 123.29µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| trait_impls_of | 987.54µs | 0.301 | 3.43ms | 13 | 147.69µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| expand_proc_macro | 947.66µs | 0.289 | 947.66µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| param_env | 845.04µs | 0.258 | 3.61ms | 190 | 70.77µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| LLVM_module_codegen | 814.38µs | 0.249 | 21.54ms | 11 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| fn_abi_of_instance | 756.59µs | 0.231 | 8.00ms | 87 | 94.16µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_item_attrs | 743.45µs | 0.227 | 743.45µs | 49 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| eval_to_allocation_raw | 699.23µs | 0.213 | 1.75ms | 96 | 36.02µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_optimized_mir | 682.45µs | 0.208 | 682.45µs | 10 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| upstream_monomorphizations_for | 680.59µs | 0.208 | 30.38ms | 40 | 653.88µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| predicates_of | 572.13µs | 0.175 | 2.65ms | 240 | 260.24µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| symbol_name | 567.49µs | 0.173 | 1.02ms | 83 | 27.44µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| link_binary | 542.22µs | 0.165 | 210.21ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| link_rlib | 492.28µs | 0.150 | 492.28µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| optimized_mir | 489.60µs | 0.149 | 1.35ms | 13 | 366.70µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| self_profile_alloc_query_strings | 482.44µs | 0.147 | 482.44µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| mir_borrowck | 467.49µs | 0.143 | 2.32ms | 6 | 2.34µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| specialization_graph_of | 417.18µs | 0.127 | 2.99ms | 2 | 64.72µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| typeck | 386.39µs | 0.118 | 1.17ms | 6 | 20.25µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| evaluate_obligation | 382.16µs | 0.117 | 2.35ms | 37 | 11.21µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| predicates_defined_on | 366.78µs | 0.112 | 1.41ms | 240 | 80.39µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_type_of | 330.99µs | 0.101 | 330.99µs | 186 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| impl_trait_ref | 312.81µs | 0.095 | 1.96ms | 446 | 151.61µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| generate_crate_metadata | 312.20µs | 0.095 | 40.82ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| eval_to_const_value_raw | 294.61µs | 0.090 | 3.72ms | 186 | 42.95µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| explicit_predicates_of | 264.28µs | 0.081 | 618.18µs | 240 | 114.63µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_implementations_of_trait | 258.32µs | 0.079 | 258.32µs | 1352 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_mir_for_ctfe | 251.50µs | 0.077 | 251.50µs | 85 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| parse_crate | 250.28µs | 0.076 | 250.28µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| monomorphization_collector_graph_walk | 247.81µs | 0.076 | 38.31ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| mir_drops_elaborated_and_const_checked | 247.08µs | 0.075 | 325.06µs | 5 | 1.24µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_explicit_predicates_of | 244.35µs | 0.075 | 244.35µs | 230 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| late_resolve_crate | 233.12µs | 0.071 | 241.18µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| check_well_formed | 229.75µs | 0.070 | 885.06µs | 9 | 1.90µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| mir_for_ctfe | 229.58µs | 0.070 | 506.28µs | 87 | 162.26µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| crates | 225.67µs | 0.069 | 226.24µs | 1 | 217.98µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| def_span | 224.07µs | 0.068 | 382.35µs | 199 | 130.17µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| panic_in_drop_strategy | 219.16µs | 0.067 | 277.71µs | 97 | 190.04µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| generics_of | 210.54µs | 0.064 | 427.04µs | 227 | 99.09µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| impl_parent | 209.55µs | 0.064 | 375.93µs | 414 | 73.70µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_fn_sig | 206.97µs | 0.063 | 206.97µs | 77 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| inferred_outlives_of | 194.96µs | 0.060 | 342.31µs | 240 | 73.34µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| opt_def_kind | 189.45µs | 0.058 | 324.16µs | 286 | 67.33µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| compute_debuginfo_type_name | 186.18µs | 0.057 | 192.53µs | 290 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| write_allocator_module | 183.49µs | 0.056 | 183.49µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| normalize_projection_ty | 178.03µs | 0.054 | 418.75µs | 10 | 7.42µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| encode_query_results_for | 174.04µs | 0.053 | 174.04µs | 61 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| hir_crate | 169.72µs | 0.052 | 191.39µs | 1 | 482.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| item_attrs | 168.10µs | 0.051 | 931.32µs | 49 | 138.16µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_adt_def | 160.46µs | 0.049 | 370.78µs | 28 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| type_of | 155.08µs | 0.047 | 582.54µs | 192 | 46.09µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| mir_shims | 149.29µs | 0.046 | 568.09µs | 7 | 14.23µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| mir_built | 145.84µs | 0.045 | 248.31µs | 6 | 27.42µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| dependency_formats | 141.74µs | 0.043 | 751.57µs | 1 | 844.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| constness | 138.99µs | 0.042 | 222.05µs | 191 | 41.41µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| resolve_instance | 137.64µs | 0.042 | 4.61ms | 48 | 17.24µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| incr_comp_prepare_load_dep_graph | 134.45µs | 0.041 | 134.45µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| fn_sig | 133.78µs | 0.041 | 413.86µs | 80 | 57.29µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| reachable_non_generics | 130.92µs | 0.040 | 447.22µs | 5 | 108.52µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| codegen_select_candidate | 126.29µs | 0.039 | 990.44µs | 8 | 4.24µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| conservative_is_privately_uninhabited | 120.37µs | 0.037 | 216.64µs | 183 | 38.73µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| crate_hash | 111.94µs | 0.034 | 155.48µs | 105 | 18.95µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| postorder_cnums | 109.80µs | 0.034 | 110.44µs | 1 | 12.62µs |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_generics_of | 107.53µs | 0.033 | 107.53µs | 217 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| configure_and_expand | 106.22µs | 0.032 | 11.05ms | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| copy_all_cgu_workproducts_to_incr_comp_cache_dir | 104.49µs | 0.032 | 104.49µs | 1 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| metadata_decode_entry_def_span | 102.08µs | 0.031 | 102.08µs | 188 | 0.00ns |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
Below 100 us. Dropped
Total cpu time: 327.650835ms
+----------------------------+---------------+
| Item | Artifact Size |
+----------------------------+---------------+
| codegen_unit_size_estimate | 1010 bytes |
+----------------------------+---------------+
| crate_metadata | 8720 bytes |
+----------------------------+---------------+
| dep_graph | 334015 bytes |
+----------------------------+---------------+
| linked_artifact | 5309916 bytes |
+----------------------------+---------------+
| object_file | 123864 bytes |
+----------------------------+---------------+
| query_cache | 46298 bytes |
+----------------------------+---------------+
| work_product_index | 621 bytes |
+----------------------------+---------------+
I figured out what went wrong.
I have this enum, which has ~ 2000 entries.
I also have my own procedural macro for poor-mans-serde.
My poor-mans-serde expands this 2000 entry enum into:
match self {
opt1: => do_write(...)?,
opt2 => do_write(...)?,
opt3 => do_write(...)?,
}
This is a 2000 * N line function handed off to llvm. Then we have another 2000*N line function handed off to llvm for reading.
LLVM did not like this very much.
@Cerber-Ursi , @Michael-F-Bryan : Thanks for your suggestions through this.
3 Likes
system
January 17, 2023, 6:00pm
8
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.