Compiler optimizations are different for crate itself vs dependency?

I've been using no-panic crate/hack to get higher assurances about code behavior.
Here is the command:

RUSTFLAGS="-Cllvm-args=--inline-threshold=5000 -C embed-bitcode -C lto -Z dylib-lto -C target-cpu=native" cargo build --all-targets --release --features no-panic

It works pretty well for crates individually (by individual I mean its tests and benchmarks), but I'm having issues with the same crate that was built successfully if it is used as a dependency in another crate (by that I mean another crate's tests and benchmarks).

All of a sudden, compiler is unable to eliminate some parts of the code and no longer able to prove that panics can't happen, which is concerning for performance reasons too.

--inline-threshold=5000 is already quite high and getting it higher and higher isn't really practical.

I know this is not a very popular thing to do in Rust ecosystem, at least not yet, but I find it quite fascinating.

Any pointers as to why that might be the case in case I'm missing something?

It could be caused by something as simple as the different crate name or -Cmetadata argument perturbing sort order somewhere causing a different function to be inlined first which in the end turns out to be a worse option.

My understanding was that the crate is compile as a library first, then other libraries and binaries it depends to are compiled and then everything is linked together.

So with no-panic relying on compiler being able to eliminate clearly unnecessary code, I don't understand how something deep in the dependency tree that is neither generic nor inlined could be affected.

Something is clearly different, but it doesn't make sense to me why it would be the case, let alone how to fix it.