How Do I Use the GDB to Debug the Release Service Process?

Hi, I'm learning to debug rust with GDB. But there are difficulties.
My scene is:
Start a service process, and then use GDB to debug the process while the process is uninterrupted.

My program is endlessly looped:

use std::thread::sleep;
use std::time::Duration;

fn main() {
    let mut count = 0;
    loop {
        sleep(Duration::from_secs(1));
        count += 1;
    }
}

Then I build the program and execute it.

cargo build --release
./target/release/bin

Debug the process in the GDB. But the results didn't seem right, and I executed commands like breakpoints and variable output that were not what I wanted to see:

root@szxphis00500:/home/temp/release_debug# gdb -q
(gdb) attach 69096
Attaching to process 69096
Reading symbols from /home/temp/release_debug/target/release/release_debug...done.
Reading symbols from /lib/libachk.so...done.
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/librt-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug/.build-id/68/f36706eb2e6eee4046c4fdca2a19540b2f6113.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.27.so...done.
done.
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.27.so...done.
done.
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.27.so...done.
done.
0x00007f3ab47ded21 in __GI___nanosleep (requested_time=0x7fff6fadd9c0, remaining=0x7fff6fadd9c0) at ../sysdeps/unix/sysv/linux/nanosleep.c:28
28      ../sysdeps/unix/sysv/linux/nanosleep.c: No such file or directory.
(gdb) b main
Breakpoint 1 at 0x55deca55c456: file library/std/src/panicking.rs, line 525.
(gdb) list
23      in ../sysdeps/unix/sysv/linux/nanosleep.c
(gdb) p count
No symbol "count" in current context.
(gdb)

My question is:

  1. Why are breakpoints not printed in the main.rs file?
  2. Why does the list command output no source code?
  3. Why cannot the count variable be printed?

In a release build, symbols and debug info in general are stripped, and agressive optimizations would make it hard to map individual instructions back to source locations anyway.

It's perhaps not surprising that you should be debugging a debug build, and not a release build, instead.

I see what you mean. However, in my scenario, the release mode needs to be debugged.

Do you control the build process? If you have control over it, you can try setting the debug = 2 profile flag in Cargo.toml. This will still generate full debug information – albeit its usefulness in the presence of optimizations will be questionable.

Thank you for your reply.
There seems to be a loss of symbolic information.

I used RUSTFLAGS="-C debuginfo=2" cargo build to generate the full debug information.
Then build the release file cargo build --release.

Then I run the following command to strip the information such as the symbol table from the debug file and save it to the debug_info.dbg file:
objcopy --only-keep-debug target/debug/release_debug debug_info.dbg

Then I strip the debug information such as the symbol table in the release file:
objcopy --strip-debug target/release/release_debug

Then I ran the following command to link the two files (The debug_info.dbg and release_debug files are stored in the same directory.)
objcopy --add-gnu-debuglink=debug_info.dbg release_debug

GDB debugging shows that the debugging information is incomplete or incorrect.

root@szxphis00500:/home/temp/release_debug# gdb -q -s debug_info.dbg -e release_debug
Reading symbols from debug_info.dbg...done.
(gdb) attach 99276
Attaching to program: /home/temp/release_debug/release_debug, process 99276
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
0x00007fa2e22a2d21 in ?? ()
(gdb) bt
#0  0x00007fa2e22a2d21 in ?? ()
#1  0x0000562a44a5e750 in <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop ()
    at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/alloc/src/sync.rs:1640
#2  core::ptr::drop_in_place<alloc::sync::Arc<std::sync::mutex::Mutex<alloc::vec::Vec<u8>>>> ()
    at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ptr/mod.rs:188
#3  core::ptr::drop_in_place<core::option::Option<alloc::sync::Arc<std::sync::mutex::Mutex<alloc::vec::Vec<u8>>>>> ()
    at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ptr/mod.rs:188
#4  core::ptr::drop_in_place<core::cell::UnsafeCell<core::option::Option<alloc::sync::Arc<std::sync::mutex::Mutex<alloc::vec::Vec<u8>>>>>> ()
    at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ptr/mod.rs:188
#5  core::ptr::drop_in_place<core::cell::Cell<core::option::Option<alloc::sync::Arc<std::sync::mutex::Mutex<alloc::vec::Vec<u8>>>>>> ()
    at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ptr/mod.rs:188
#6  core::ptr::drop_in_place<core::option::Option<core::cell::Cell<core::option::Option<alloc::sync::Arc<std::sync::mutex::Mutex<alloc::vec::Vec<u8>>>>>>> ()
    at /rustc/db9d1b20bba1968c1ec1fc49616d4742c1725b4b/library/core/src/ptr/mod.rs:188
#7  std::thread::local::lazy::LazyKeyInner<T>::initialize () at library/std/src/thread/local.rs:452
#8  std::thread::local::fast::Key<T>::try_initialize () at library/std/src/thread/local.rs:609
#9  0x00007fa2e1d295e0 in ?? ()
#10 0x0000ffff00001f80 in ?? ()
#11 0x0000000000000000 in ?? ()
(gdb) b main
main                 main.rs              release_debug::main
(gdb) b main.rs:53
No line 53 in file "main.rs".
Make breakpoint pending on future shared library load? (y or [n])
(gdb) b main.rs:5
Breakpoint 1 at 0x562a44a4e184: file src/main.rs, line 5.
(gdb)

But one important change is that I can set breakpoints.

In short, my strategy is to use the debugging information carried in the debug file to debug the release file without debugging information. But it seems to be wrong.

Of course it is wrong! The release file has completely different symbol addresses from those in the debug executable, and it will also have some missing symbols which were in the debug file, since the code is optimized differently across the two profiles/resulting executables. You can't use the debug info of one binary for debugging another binary.

4 Likes

Thank you very much for your reply. When I generate the release version, I choose not to optimize, that is, opt-level = 0, so that the generated release contains the correct debugging information.

It's a huge step forward. After performing the following operations, the debugging is normal, but the call stack is still faulty.

RUSTFLAGS="-C debuginfo=2" cargo build --release
objcopy --only-keep-debug target/release/release_debug release_info.dbg
objcopy --strip-debug --strip-unneeded target/release/release_debug
cp target/release/release_debug release_debug
objcopy --add-gnu-debuglink=release_info.dbg release_debug
./release_debug

rust-gdb -q -s release_info.dbg -e release_debug
(gdb)attach PID

Results:

root@szxphis00500:/home/temp/release_debug# rust-gdb -q -s release_info.dbg -e release_debug
Reading symbols from release_info.dbg...done.
(gdb) attach 124818
Attaching to program: /home/temp/release_debug/release_debug, process 124818
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.27.so...done.
done.
0x00007fc9fcfe1d21 in ?? ()
(gdb) bt
#0  0x00007fc9fcfe1d21 in ?? ()
#1  0x0000560fe8af0550 in std::sys::unix::thread::Thread::sleep () at library/std/src/sys/unix/thread.rs:216
#2  std::thread::sleep () at library/std/src/thread/mod.rs:815
#3  0x0000000000000000 in ?? ()
(gdb) b 5
Breakpoint 1 at 0x560fe8adfac4: file src/main.rs, line 5.
(gdb) b 7
Breakpoint 2 at 0x560fe8adfacc: file src/main.rs, line 7.
(gdb) l
1       use std::thread::sleep;
2       use std::time::Duration;
3
4       fn main() {
5           let mut count = 0;
6           loop {
7               sleep(Duration::from_secs(1));
8               count += 1;
9               println!("{}", count);
10          }
(gdb) p count
No symbol 'count' in current context
(gdb) c
Continuing.

Breakpoint 2, release_debug::main () at src/main.rs:7
7               sleep(Duration::from_secs(1));
(gdb) c
Continuing.

Breakpoint 2, release_debug::main () at src/main.rs:7
7               sleep(Duration::from_secs(1));
(gdb) p count
$1 = 97
(gdb)

The following information is displayed in the call stack: ?? (). I'm not sure how this will be resolved.

If you look at the source of the stdlib, you can see that thread::sleep() calls into libc for actually sleeping, I don't think that libc is a debug version or that it ships with debug symbols. So the debugger is correct in not knowing where it is. You could try installing debug symbols for your system's libc.

By the way, if you are turning off optimizations, why are you still trying to debug the release version? If you don't need the optimizations, you may as well use the debug build.

2 Likes

Thank you for your reply.

After I upgraded my gdb from 8.1 to 10.2, I was able to debug it. (Although some interference information is displayed: No debugging symbols found).

root@szxphis00500:/home/temp/release_debug# rust-gdb -q
(gdb) attach 79011
Attaching to process 79011
Reading symbols from /home/temp/release_debug/release_debug...
Reading symbols from /home/temp/release_debug/release_info.dbg...
Reading symbols from /lib/libachk.so...
Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libgcc_s.so.1)
Reading symbols from /lib/x86_64-linux-gnu/librt.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/librt.so.1)
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...
(No debugging symbols found in /lib/x86_64-linux-gnu/libpthread.so.0)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...
(No debugging symbols found in /lib/x86_64-linux-gnu/libdl.so.2)
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...
(No debugging symbols found in /lib/x86_64-linux-gnu/libc.so.6)
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(No debugging symbols found in /lib64/ld-linux-x86-64.so.2)
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...
(No debugging symbols found in /lib/x86_64-linux-gnu/libnss_compat.so.2)
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...
(No debugging symbols found in /lib/x86_64-linux-gnu/libnss_nis.so.2)
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...
(No debugging symbols found in /lib/x86_64-linux-gnu/libnsl.so.1)
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...
(No debugging symbols found in /lib/x86_64-linux-gnu/libnss_files.so.2)
0x00007f2aa2279d21 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt
#0  0x00007f2aa2279d21 in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000562b7895a550 in std::sys::unix::thread::Thread::sleep () at library/std/src/sys/unix/thread.rs:216
#2  std::thread::sleep () at library/std/src/thread/mod.rs:815
#3  0x0000562b78949aee in release_debug::main () at src/main.rs:7
(gdb) l
1       use std::thread::sleep;
2       use std::time::Duration;
3
4       fn main() {
5           let mut count = 0;
6           loop {
7               sleep(Duration::from_secs(1));
8               count += 1;
9               println!("{}", count);
10          }
(gdb) b 5
Breakpoint 1 at 0x562b78949ac4: file src/main.rs, line 5.
(gdb) b 8
Breakpoint 2 at 0x562b78949aee: file src/main.rs, line 8.
(gdb) c
Continuing.

Breakpoint 2, release_debug::main () at src/main.rs:8
8               count += 1;
(gdb) c
Continuing.

Breakpoint 2, release_debug::main () at src/main.rs:8
8               count += 1;
(gdb)
Continuing.

Breakpoint 2, release_debug::main () at src/main.rs:8
8               count += 1;
(gdb) p count
$1 = 25

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.