I'm trying to create a release binary (as small as possible, but with backtrace info) + a separate .dwp debug info artifact.
Below are my config files. .cargo/config.toml
The size is much much bigger than with strip = "debuginfo" enabled, but with strip = "debuginfo", the size is reasonable, but lldb and gdb can't find the debug symbols with dwp, even when you mannually add-symbol-file xxx.dwp for it.
Full debug info is good, to profile you program, analyze a core dump and replay a rr trace, even for release build, so here is split dwarf for linux platform, which also significantly speed compiling.
Let's admit one thing: Windows *.pdb does do thing right, split debug info should be default, and debug info is alse needed for release build. What a mess for linux world, objcopy --strip-all --add-gnu-debuglink=<DBG_FILE> <BASE_FILE> is a lie, cheating us for so many years, wasting so much time and electric power.
So my wish is
keep all debug info in dwp file, not the binary file.
keep stack trace info in binary file by default, but provide options to not keep it or leave it in dwp file.
generate debug info and keep it by default for release build.
At least, we should provide options for it, developer experience matters, especially for rust programmers.
split-debuginfo = "unpacked" and split-debuginfo = "packed" uses DWARF's split debuginfo functionality on systems other than Windows and macOS. It works as follows: When compiling a crate the debuginfo is split into two parts: A part that ends up in .dwo files which contains all address independent information and a part that ends up in the object files themself which contains all the address dependent information and references the .dwo files. When linking the debuginfo inside the object files is relocated and added to the final executable. In addition if you use split-debuginfo = "packed" then all .dwo files are combined into a single .dwp file with things like strings deduplicated. What is essential to be aware of however is that the .dwp file is useless without the parts of the debuginfo that strip = "debuginfo" removes. The .dwp file does not contain the addresses of functions, so the debugger wouldn't know which part of the debuginfo it needs to apply to the current function unless you keep the debuginfo inside the executable.
Unix linkers have no awareness of debuginfo at all. They cannot produce separate debuginfo files themself. That is why if you want to move all debuginfo out of the executable, not just the address independent half, you have to link with all debuginfo inside the executable and then move the debuginfo to a separate file after linking. The MSVC linker does directly emit debuginfo on the other hand.
This is not something we can do ourself. You will have to convince the people who write linkers to add native support for putting debuginfo into a separate file to their linker.
This sounds like a good idea, perhaps something that the wild linker could consider (at some point in the future)? Cc @davidlattimore
I don't see a reason why we should be beholden to the Unix legacy where it doesn't make sense. And in many areas rust and tooling written in Rust have started to question such assumptions.
If you care about binary sizes, never use debug = "full" (and debug = true). It's an extremely bloated overkill. It only useful for tracking values of individual variables inside an interactive debugger.
Fully detailed backtrace information is already added at debug = "limited" level.
Backtraces with simplified function names also work with debug = "line-tables-only", and this gives much smaller symbol sizes.
I couldn't get DWP/DWO to work. I don't know if I'm doing something wrong, or is it just poorly supported by the tooling. For ELF executables, the old trick of extracting ELF sections (the classic .debug files) seems to work best.
I don't recommend relying on Cargo's split-debuginfo, especially that this setting is not compatible across-platforms. It will backfire or completely break your builds on other platforms.
gnu-debuglink is hard to embed properly, because if you're not careful, it may embed absolute path to the debug symbols file inside your target temporary directory. Even if you embed it well, it's still not reliable due to using executable's filename. Using .note.gnu.build-id is much more robust, since it creates debug symbol files named after the hash of the binary data.
objcopy --compress-debug-sections=zlib helps a lot. You can't use zstd, because it's not supported by default in Rust executables, but even zlib more than halves the debug info size.
It's a common common misconception: debug = "full" is not XOR with binary size.
If you strip binary with debug = "full" and then strip the binary with debug = "none", they are toally identical files, debug = "full" won't make you binay a byte bigger.
Why don't think binary size for a low level programming is very important? What about the embeded world?
I want to express one point: split-debuginfo = “packed” is not that userful in practice, though the ideas behind it is cool and make linker faster.
Yeah, I totally agree with you, “For ELF executables, the old trick of extracting ELF sections (the classic .debug files) does to work best.": As for as I know, It's the only workable way to distrubute you debug info and keep you binary file as small as it should be, and it works well with debuginfod.
Thank you for explaining the very clear details, I realize that it's not just a Rust related problem.
But one thing to mention, split-debuginfo = "packed" is cool, it's a dwarf standard, and the Rust community put effort to implement it, it makes linker faster(so the entire compilation time), but it is not that usefull in practice.
We have to use split-debuginfo = "unpacked" and do the old trick of extracting ELF sections (the classic .debug files), and it works well with debuginfod.
Like the details you describe: split-debuginfo = "packed" will leave some part debug info in final elf file and much debug info in dwp(dwo) file, but this way you can't take the some part out from the final elf file, right? It does make the final elf file much much bigger, noone will distribute release file and debug info like that, and it's unclear if it works well with debuginfod.
The old tricks works well, but we have to pay for it: longer linker time, duplicate and big intermediate files.
The split dwarf standard just doesn't solve the problem.
Not quite. I also had to use it for cache line contention profiling in the past (perf c2c). I suspect some other types of memory layout profiling might also need it.