Help building a debuggable rust application on FreeBSD

hartzell · January 27, 2020, 9:01pm

Quick summary

How can I build a rust application on an amd64 FreeBSD system so that
I can use a debugger to examine its core files?

Using a "dev" build, rust@1.38.0, rust-gdb and gdb@8.3.1 don't seem
to get me there.

Gory details

I'm hoping for feedback on, or other places to ask, a question about getting debugging info from core dumps from a rust application on FreeBSD.

Here's the scenario: I listen to music using the Logitech Media Server ("Slimdevices") ecosystem. There's a plugin, spotty that lets me listen to spotify, via the librespot library. The application periodically crashes due to a SIGABRT. I'm unable to get debug info from the resulting core files.

I'm working on an amd64 FreeNAS system, which runs FreeBSD 11.2 under the covers. I've installed "everything" from packages; in particular

rustc --version says: rustc 1.38.0
the core gdb is 6.1.1
the gdb from packages is 8.3.1.

I've cloned the application and built it with cargo build, which seems to give me a "debugging binary".

root@spotty-build:~/spotty/target/debug # file ./spotty
./spotty: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically
 linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, with
 debug_info, not stripped
root@spotty-build:~/spotty/target/debug #

I ran it by hand in the background and killed it via kill -ABRT <pid> to generate a trivial core file so that I can figure out how to get debugging symbols in the deployed application. I think that this is rust-related, but perhaps my gdb skills are just rusty.

If I use the system gdb it seems to have trouble with version of the Dwarf debugging info:

root@spotty-build:~/spotty/target/debug # gdb spotty /var/db/system/cores/spotty.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...Dwarf Error: wrong version in compilation unit header (is 4, should be 2) [in module /root/spotty/target/debug/spotty]

Core was generated by `./spotty -n poodle'.
Program terminated with signal 6, Aborted.
#0  0x0000000803182b2c in ?? ()
(gdb) where
#0  0x0000000803182b2c in ?? ()
#1  0x0000000803176bc1 in ?? ()
#2  0x00007fffffffad60 in ?? ()
#3  0x00000008031797cc in ?? ()
#4  0x000000000001894b in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffffadd0 in ?? ()
#7  0x0000000803180858 in ?? ()
#8  0x00007fffffffad90 in ?? ()
#9  0x00000008040162b8 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)

gdb@8.3.1 (from packages) seems to handle the DWARF debugging info version but still no joy:

root@spotty-build:~/spotty/target/debug # /usr/local/bin/gdb --version
GNU gdb (GDB) 8.3.1 [GDB v8.3.1 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
root@spotty-build:~/spotty/target/debug # /usr/local/bin/gdb spotty /var/db/system/cores/spotty.core
GNU gdb (GDB) 8.3.1 [GDB v8.3.1 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.3".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from spotty...
[New LWP 100683]
[New LWP 100623]
[New LWP 100649]

warning: Unexpected size of section `.reg-xstate/100683' in core file.
Core was generated by `./spotty -n poodle'.
Program terminated with signal SIGABRT, Aborted.

warning: Unexpected size of section `.reg-xstate/100683' in core file.
#0  0x0000000803182b2c in ?? ()
[Current thread is 1 (LWP 100683)]
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /root/spotty/target/debug/spotty.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) where
#0  0x0000000803182b2c in ?? ()
#1  0x0000000803176bc1 in ?? ()
#2  0x00007fffffffad60 in ?? ()
#3  0x00000008031797cc in ?? ()
#4  0x000000000001894b in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffffadd0 in ?? ()
#7  0x0000000803180858 in ?? ()
#8  0x00007fffffffad90 in ?? ()
#9  0x00000008040162b8 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb) quit
root@spotty-build:~/spotty/target/debug #

using rust-gdb (and setting the path so that it picks up the newer gdb) clears up the debug script issue, but I'm still not getting useful line numbers:

root@spotty-build:~/spotty # rust-gdb target/debug/spotty /var/db/system/cores/spotty.core
GNU gdb (GDB) 8.3.1 [GDB v8.3.1 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.3".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from target/debug/spotty...
[New LWP 100683]
[New LWP 100623]
[New LWP 100649]

warning: Unexpected size of section `.reg-xstate/100683' in core file.
Core was generated by `./spotty -n poodle'.
Program terminated with signal SIGABRT, Aborted.

warning: Unexpected size of section `.reg-xstate/100683' in core file.
#0  0x0000000803182b2c in ?? ()
[Current thread is 1 (LWP 100683)]
(gdb) where
#0  0x0000000803182b2c in ?? ()
#1  0x0000000803176bc1 in ?? ()
#2  0x00007fffffffad60 in ?? ()
#3  0x00000008031797cc in ?? ()
#4  0x000000000001894b in ?? ()
#5  0x0000000000000001 in ?? ()
#6  0x00007fffffffadd0 in ?? ()
#7  0x0000000803180858 in ?? ()
#8  0x00007fffffffad90 in ?? ()
#9  0x00000008040162b8 in ?? ()
#10 0x0000000000000000 in ?? ()
(gdb)

I'd appreciate any help, suggestions or ... on how to get useful information about crashed from these core files.

Thanks!

g.

hartzell · January 27, 2020, 9:08pm

One more tidbit I just noticed; the file command reports a screwy name of the application in the core file (I ran it as spotty -y poodle):

root@spotty-build:~ # file /var/db/system/cores/spotty.core
/var/db/system/cores/spotty.core: ELF 64-bit LSB core file x86-64, version 1 (FreeBSD), FreeBSD-style, from 'y -n poodle'
root@spotty-build:~ #

hartzell · January 27, 2020, 10:39pm

Another bit: if I run spotty in rust-gdb and then kill it with a SIGABRT, I do see source/line info in where output:

[...snip...]
(gdb) c
Continuing.

Thread 1 received signal SIGABRT, Aborted.
0x0000000803182b2c in ?? () from /lib/libthr.so.3
(gdb) where
#0  0x0000000803182b2c in ?? () from /lib/libthr.so.3
#1  0x0000000803176bc1 in ?? () from /lib/libthr.so.3
#2  0x0000000803180858 in ?? () from /lib/libthr.so.3
#3  0x00000000025409b4 in std::sys::unix::condvar::Condvar::wait (
    self=0x80401d020, mutex=0x80401d018)
    at /wrkdirs/usr/ports/lang/rust/work/rustc-1.38.0-src/src/libstd/sys/unix/condvar.rs:71
#4  0x0000000002540956 in std::sys_common::condvar::Condvar::wait (warning: (Internal error: pc 0x253f734 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x253f6c0 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x253f734 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0x253f734 in read in psymtab, but not in symtab.)


self=0x80401d020, warning: (Internal error: pc 0x253f734 in read in psymtab, but not in symtab.)

mutex=0x80401d018)
    at /wrkdirs/usr/ports/lang/rust/work/rustc-1.38.0-src/src/libstd/sys_common/condvar.rs:41
[...snip...]

system · April 26, 2020, 10:39pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Dwarf, debug format	6	3476	January 12, 2023
Rust debugger's question help	10	3518	January 12, 2023
What do you use to debug Rust (particularly in Emacs? Editors and IDEs	12	3265	April 18, 2023
Build rustc properly for cross targets ( Linux > FreeBSD ) help	5	1590	May 11, 2020
Beginner's question: Cannot debug with rust-gdb under openSUSE help	5	2553	January 12, 2023

Help building a debuggable rust application on FreeBSD

Quick summary

Gory details

Related Topics