Can you try rustup run stable cargo build. That should avoid the startup time of rustup showing in the graph. This is a fixed cost for every rustc invocation when rustup is used as wrapper to choose the right toolchain. Also try cargo +nightly rustc -- -Ztime-passes to show where exactly rustc spends time.
Note that in the cargo timings chart, Rust library crates get split into two spans, one light blue for the parsing, type checking, etc, and one purple for code generation (going from intermediate representation to machine code); but binary (and cdylib) crates display all the time in blue. So, the time being shown is parsing, checking, code generation, and linking — no distinction is made.
This is largely an accident: the reason it can produce the distinction for libraries is because of “pipelined” compilation where rustc tells cargo as soon as the crate metadata is available and downstream crates can start being compiled, but for binaries and cdylibs there are no such intermediate outputs, so Cargo has no timing information about stages of that compilation. In order to have that information, someone would need to invent a protocol for rustc to produce it in a machine-readable way for Cargo to copy into the chart.