"error: build failed" apparently without reason

For a while now we have one CI build task that fails about every other time, and it does so without giving any hint of what is wrong. I never observed it on my local machine, only on Azure pipelines, and it only affects one particular crate, so I think it must be some build script spuriously failing. But for the life of me I cannot figure out which one it is, because there is absolutely no clue in the console output.

The command I’m using is

cargo build --release -vv

How can I get more debug output to show what exactly triggers the error? I’m using 1.59.

It looks like it is the Azure script that crash and not Cargo, if I understand correctly.

The log start with that:

[command]/usr/bin/bash /root/agents/agent-00/_work/_temp/88d1c5fa-c46c-4fcd-8d42-413a6b27f986.sh

Can you check with another target or toolchain?

Thanks for looking into this!

This is the relevant part of the pipeline definition:

      - job:
        displayName: make docs
        pool:
          name: Native
        steps:
          - checkout: self
            clean: true
            path: "s"
          - bash: |
              set -e
              echo "Building cosmos-release"
              cd rust/release
              cargo build --release -vv
              echo "Building docs"

Since there is no “finished” message from the cargo run and “Building docs” is not printed, I assume that it is indeed cargo that prints error: build failed (I think we can exclude that this comes from bash itself). We’ve had this issue since a while, across several Rust versions (from approx. 1.54 to 1.59; earlier versions cannot be excluded because make docs hasn’t been around forever).

Another point to consider is that the dependency tree is 156 crates large, so it definitely looks like cargo is still at the beginning when this happens.

Is it possible for some build.rs or proc_macro to crash the build like this without a proper error being reported?

I guess it would be possible for a build.rs to use a command that doesn't exist or a command that will crash for some reasons, but you should always get a relevant error.

Proc macros produce the same compilation error that you could have after an expand as far as I know. However, I looked a bit at the dependencies that you have in the compilation and there is not so much build.rs except the ones from dtolnay in whom I trust a lot.

Into the cargo book, the error 101 mean that he failed to complete. That doesn't help a lot... it can also be an OOM, you could check the memory allocated to your target VM.

That's not a solution, but all the warnings in the convert-case create are also suspicious.

I'm sorry if I can't help you more!! :smile:

Hmm, the machine has 128GiB memory, so that’s probably not the issue. And I can run the build on that machine successfully when I log in. Would be really helpful if I could get more output from cargo as to what it is doing …

You can try to set the environment variable CARGO_TERM_VERBOSE to true. I doubt that will return more than the -vv option.

Yeah, I’ll try that. Thanks for your help!