Building for target-cpu when the machine does not support it

I'm trying to set up CI build in a way that it builds for --target-cpu=skylake-avx512. The problem is that I can get SIGILL occasionally on the CI machine during build scripts execution. It probably happens when the machine doesn't support skylake-avx512 itself. It seems like the target-cpu flag from RUSTFLAGS or cargo config got passed to both target and build configs. That sounds weird to me. Is it possible to set RUSTFLAGS only for the target binary?

If you explicitly say --target=... as well, cargo will separate its use of RUSTFLAGS.