Nix users -- why aren't debug symbols working?

help
1

Hi all,

I started this thread at the NixOS discourse a couple months ago, and it hasn't gotten any attention, so I thought I'd try here as well.

In short, I'm using nix's buildRustPackage with buildType = "debug"; and dontStrip = true;, and yet when I run rust-lldb on the resulting binary, I'm not seeing human-readable names (EDIT: and my function name breakpoints don't work).

// src/main.rs
fn a_function() -> u32 {
    dbg!("this is from debug!");
    4
}

fn main() {
    println!("{}", a_function());
    println!("Hello, world!");
}

# flake.nix
{
  inputs.nixpkgs.url = "github:nixos/nixpkgs";
  outputs = {nixpkgs, ...}: let
    system = "aarch64-darwin";
  in {
    packages.${system}.default = let
      pkgs = import nixpkgs {inherit system;};
    in
      pkgs.rustPlatform.buildRustPackage {
        pname = "foo";
        buildType = "debug";
        dontStrip = true;
        version = "0.0.1";
        src = ./.;
        cargoHash = "sha256-p1UJmkerUpvbkZKZYPpqMixWdL3mHSwYsiHo14AUih8=";
      };
  };
}

Running with cargo build and then rust-lldb seems fine:

$ cargo build
$ rust-lldb -o 'break set --name a_function' -o run ./target/debug/foo
(lldb) command script import "/Users/n8henrie/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/etc/lldb_lookup.py"
(lldb) command source -s 0 '/Users/n8henrie/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/etc/lldb_commands'
Executing commands in '/Users/n8henrie/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/etc/lldb_commands'.
(lldb) type synthetic add -l lldb_lookup.synthetic_lookup -x ".*" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)String$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?str$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?\\[.+\\]$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::ffi::([a-z_]+::)+)OsString$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Vec<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)VecDeque<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Rc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Arc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Cell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Ref<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefMut<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefCell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^core::num::([a-z_]+::)*NonZero.+$" --category Rust
(lldb) type category enable Rust
(lldb) target create "./target/debug/foo"
Current executable set to '/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4zz7jkEjQF/foo/target/debug/foo' (arm64).
(lldb) break set --name a_function
Breakpoint 1: where = foo`foo::a_function::h6de5a9fc4db6253c + 20 at main.rs:3:5, address = 0x0000000100000b28
(lldb) run
Process 65944 stopped
* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100000b28 foo`foo::a_function::h6de5a9fc4db6253c at main.rs:3:5
   1    // src/main.rs
   2    fn a_function() -> u32 {
-> 3        dbg!("this is from debug!");
   4        4
   5    }
   6
   7    fn main() {
Target 0: (foo) stopped.
Process 65944 launched: '/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4zz7jkEjQF/foo/target/debug/foo' (arm64)
(lldb)

In contrast, running nix build, I get Breakpoint 1: no locations (pending).,

$ nix build
$ rust-lldb -o 'break set --name a_function' -o run ./result/bin/foo
(lldb) command script import "/Users/n8henrie/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/etc/lldb_lookup.py"
(lldb) command source -s 0 '/Users/n8henrie/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/etc/lldb_commands'
Executing commands in '/Users/n8henrie/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/etc/lldb_commands'.
(lldb) type synthetic add -l lldb_lookup.synthetic_lookup -x ".*" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)String$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?str$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?\\[.+\\]$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::ffi::([a-z_]+::)+)OsString$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Vec<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)VecDeque<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Rc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Arc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Cell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Ref<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefMut<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefCell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^core::num::([a-z_]+::)*NonZero.+$" --category Rust
(lldb) type category enable Rust
(lldb) target create "./result/bin/foo"
Current executable set to '/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4zz7jkEjQF/foo/result/bin/foo' (arm64).
(lldb) break set --name a_function
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) run
warning: (arm64) /nix/store/bp7fq96ciy906kf8xkq8hmjhgcjncx1q-foo-0.0.1/bin/foo(0x0000000100000000) address 0x0000000100000000 maps to more than one section: foo.__TEXT and foo.__TEXT
warning: (arm64) /nix/store/bp7fq96ciy906kf8xkq8hmjhgcjncx1q-foo-0.0.1/bin/foo(0x0000000100000000) address 0x0000000100050000 maps to more than one section: foo.__DATA_CONST and foo.__DATA_CONST
warning: (arm64) /nix/store/bp7fq96ciy906kf8xkq8hmjhgcjncx1q-foo-0.0.1/bin/foo(0x0000000100000000) address 0x0000000100054000 maps to more than one section: foo.__DATA and foo.__DATA
warning: (arm64) /nix/store/bp7fq96ciy906kf8xkq8hmjhgcjncx1q-foo-0.0.1/bin/foo(0x0000000100000000) address 0x0000000100000000 maps to more than one section: foo.__TEXT and foo.__TEXT
warning: (arm64) /nix/store/bp7fq96ciy906kf8xkq8hmjhgcjncx1q-foo-0.0.1/bin/foo(0x0000000100000000) address 0x0000000100050000 maps to more than one section: foo.__DATA_CONST and foo.__DATA_CONST
warning: (arm64) /nix/store/bp7fq96ciy906kf8xkq8hmjhgcjncx1q-foo-0.0.1/bin/foo(0x0000000100000000) address 0x0000000100054000 maps to more than one section: foo.__DATA and foo.__DATA
[src/main.rs:3] "this is from debug!" = "this is from debug!"
4
Hello, world!
Process 66626 exited with status = 0 (0x00000000)
Process 66626 launched: '/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.4zz7jkEjQF/foo/result/bin/foo' (arm64)
(lldb)

The buildType = "debug"; and dontStrip seem to be doing something, but still has way less output than the cargo build version:

$ cargo build
$ objdump --syms ./target/debug/foo  | wc -l
2242
$
$ # normal release build / *without* `buildType = "debug"`
$ nix build && objdump --syms ./result/bin/foo  | wc -l
1262
$
$ # adding `buildType = "debug";`:
$ nix build && objdump --syms ./result/bin/foo  | wc -l
1272
$
$ # adding `dontStrip = true;` as well:
$ nix build && objdump --syms ./result/bin/foo  | wc -l
1375

Are there any nix + rust users that can help me build a rust binary with nix that can be debugged with rust-lldb?

FWIW, I'm seeing the same behavior with rust-gdb on x86_64-linux.

TIA for any suggestions!

2 Likes
2

Actually, rust-gdb seems to work if I use break foo::a_function (thanks Problems with debugging programs and GDB - #3 by dinfuehr)

Still no luck with rust-lldb though, which is tough as I'm primarily a Mac user.

3
4

I'm also unable to get lldb to work with this simple flake to compile a "hello world" C++ script:

{
  description = "Example attempt to debug C++";

  outputs = {
    self,
    nixpkgs,
  }: let
    system = "aarch64-darwin";
    pkgs = import nixpkgs {
      inherit system;
    };
  in {
    packages.${system}.default = pkgs.stdenv.mkDerivation {
      name = "foo";
      src = ./foo.cpp;
      dontUnpack = true;
      dontStrip = 1;
      buildPhase = ''
        clang++ -g -o foo $src
      '';
      installPhase = ''
        mkdir -p $out
        cp -r * $out
      '';
    };
  };
}

The foo.cpp is literally just a hello world; compiling directly with the clang command above allows lldb to recognize the symbols, whereas compiling with nix results in the same kind of gibberish I'm seeing with rust.

So it seems like perhaps this is a bigger clang(?) nix issue and not Rust specific.

5

I don't use nix but let's discuss troubleshooting techniques:

You want to find what factors are relevant and what aren't. So far you have found that nix + rust + lldb and nix + clang + lldb cause issues. There are many combinations you haven't reported trying. For example:

  • nix + gcc + lldb?
  • What about a different base OS (eg nix on NixOS vs nix as a dev env on a different base OS)?
  • What about architecture, you mentioned Mac so x86-64 vs Aarch64?

You should make a matrix or spread sheet with the different combinations (gcc/clang/rustc, host OS, nix/non-nix, architecture) to make sure you don't miss anything. This can help narrow down which factors are actually important to investigate further.

Then consider looking into other tools that read debug info, (objdump, pahole, nm, rr, valgrind, ...) and see if they also behave strangely. Some of those come in multiple versions: eu-readelf, binutils readelf, llvm-readelf (I think it has its own version, llvm-objcopy definitely exists at least). This could help narrow down if it is a specific reading library (e.g. llvm) causing issues.

6

Thanks for your response -- sorry that I haven't put all relevant context here, I was mostly hoping some rust + nix on aarch64-darwin users might chime in.

I've done most of the above, with my results posted mostly at the nixpkgs issue; in short, the behavior seems specific to nix + macOS (gdb works on x86_64 Linux even when compiled by nix, I only have an aarch64-darwin machine to test). I'm also trying to add details as they come up, unfortunately I don't have much time to spend debugging this.

7

I figured out the C++ issue -- looks like it was probably related to the binary missing a UUID: rust debug symbols don't work on aarch64-darwin · Issue #262131 · NixOS/nixpkgs · GitHub

Unfortunately that didn't fix the issue with rust debugging. It looks like some symbols are missing:

Looking at this:

fn a_function() -> u32 {
    7
}

fn main() {
    println!("Hello, nix world!");
    println!("{}", a_function());
}

I get the following with a native cargo-built binary:

$ objdump -S target/debug/foo | grep a_function
00000001000014b0 <__ZN3foo10a_function17h8e2c0e9f3fe54aedE>:
fn a_function() -> u32 {
    println!("{}", a_function());
   1000014ec:   97fffff1        bl      1000014b0 <__ZN3foo10a_function17h8e2c0e9f3fe54aedE>

but this with nix:

$ objdump -S result/bin/foo | grep a_function
0000000100002118 <__ZN3foo10a_function17h19020d20e390a0d3E>:
   100002154:   97fffff1        bl      100002118 <__ZN3foo10a_function17h19020d20e390a0d3E>

The missing plaintext fn a_function() -> u32 { seems to suggest a problem in generating the debug symbols I would think.

8

It seems that the issue may ultimately be related to the nix build process, which compiles the project in a sandboxed temporary directory, and then copies the resulting binary back to the working directory.

Specifically, it seems like there are a bunch of "/path/to/rcgu.o" paths that are embedded into the executable that are no longer valid by the time I run rust-lldb (they refer to the temporary build directory, which is already gone).

Now to see if I can do anything about it...

9

Still working on this. I'm surprised to see that apparently the intersection of nix / rust / aarch64-darwin users seems pretty small.

10

Still an issue on a recent nixpkgs-unstable. Still wonder if this is related to lldb looking for things in the build environment: rust debug symbols don't work on aarch64-darwin · Issue #262131 · NixOS/nixpkgs · GitHub

11

Made some progress today -- looks like lldb is looking for debug stuff from the nix build directory (which is cleaned up after a successful build). More details and repro: rust debug symbols don't work on aarch64-darwin · Issue #262131 · NixOS/nixpkgs · GitHub

12

PR: rustPlatform.buildRustPackage: provide debug symbols on darwin by n8henrie · Pull Request #327136 · NixOS/nixpkgs · GitHub

Workaround uses split-debuginfo=packed to provide the .dSYM files for debugging on MacOS.

Still doesn't address the wrong source-map issue: rust-lldb finds wrong source map (from build directory) · Issue #282900 · NixOS/nixpkgs · GitHub but I was able to write a helper function that sorts this out acceptably: rust-lldb finds wrong source map (from build directory) · Issue #282900 · NixOS/nixpkgs · GitHub