Headcrab, a Rust debugger library

You are right @TomP. However, why is it so slow compared to other lang (C#, pascal etc)? For proving the capability of Rust, I tried porting a pascal code. I am definitely getting 25% improvement in release which is great. However, I am having slow debug which is almost x5 times slower than the pascal. From other sources, I could see praise for pascal in its debug performance but x5 times slower is really unacceptable in my case. Now I am having hard time in convincing my team :frowning:.

Just to clarify, is it slow both when you run the code in debug mode normally (i.e. cargo run from the command line) and under the debugger, or only under the debugger?

Rust uses lots of abstractions which only become zero-cost after optimisation. The debug mode contains next to no optimisations (deliberately, it's hard to debug optimised code) and is well known for generating slow code.

It's not uncommon that newbies come to this forum saying "my rust code is slower than this Python equivalent" because they've just done cargo run, then are blown away when they are told to use cargo run --release and the code is 1000x faster.

If this slow down only happens under the debugger, then I'd make an issue upstream.


This doesn't just happen to newbies and is actually a very valid concern. I think it is wrong to make dev mode the default for cargo build and cargo run; not just is the debuggability of Rust programs very limited in general but dev also has a number of serious issues such as:

  • it applies globally (whereas in other languages you can control the optimisation on a per file level and it is quite common to only enable debugging for the code you're actually interested to debug)
  • dev mode enables debug assertions unconditionally
  • there really is no "optimize lightly" or "optimize for best debugging experience" option like in other languages, the options are only off, normal, crazy, for size and crazy but keep size in mind
  • off is often not an option, e.g. in embedded it is quite typical that a program without optiomisations will not even fit into the flash of a MCU, if you have a tiny MCU even a LED blinky will not even link due to size problems. Also things which are "guaranteed" to be zero-cost like a compiler_fence() will suddenly turn into real code including a panicking branch: https://github.com/rust-lang/rust/issues/68208. Also trivial things will suddenly be so inefficient and slow that using some peripherals (e.g. USB) will simply not work in that setting.

Other languages do a whole better here and people are not only used to that ability but it can actually be a show-stopper for Rust usage.

Personally, I couldn't care less about classical debuggers, but there's a whole lot more attached to the ability to generate debuggable code and Rust really is very sub-par here.

1 Like

I'm curious to know what you feel is missing here.

Reaching for a "classical debugger" has been my last resort in all kind of languages, including assembler, for decades having often found that when a bug is hard proving hard to find then building for and using a debugger hides the bug, make stings worse or cannot even be used. As you point out for embedded systems.

Luckily the need to reach for such a debugger is greatly reduced in Rust as it eliminates all those insidious memory and type errors.

In a nutshell what I wrote above wrt to light optimisations. But Rust has additional challenges due to the lack of language support within the usual debuggers which is hopefully what headcrab is going to rectify. In the last few months the embedded folks have massively stepped up their game with respect to tooling, e.g. via https://probe.rs.

Now all we need is a usable debugger. :wink:

I fully agree, people around me appreciate that because they know when I reach for a debugger, the shit has just hit the fan and things are about get ugly in my vicinity... :sweat_smile:

1 Like

My personal experience with debuggers so far has been the following:

  • In Rust code println! is easier than a debugger. Pretty much only when a program hangs do I use a debugger to get a backtrace.
  • While debugging miscompilation caused by cg_clif, a debugger is pretty useful. Especially with rr to be able to follow an invalid value back to the original miscompilation.
  • In C++ code a debugger is very useful for me, as there is no #[derive(Debug)] equivalent and many of my bugs are caused by memory corruption.

Thanks @Michael-F-Bryan. Below are my stats (I am not an expert in bench-marking; I ran my code couple of times and showing averages here)

Runtime of my code with cargo run --release       is 0.08 mins
Runtime of my code with cargo run                 is 4.30 mins 
Runtime of my code with cargo build + vsdgb       is 6.40 mins
Runtime of my code with cargo build + vscode-lldb is 5.80 mins

Below is what my code does,

1) Read 1 binary file
2) Store it in a ndarray
3) process the columns in iter().for_each()
4) Store the results
5) Repeat step 1-4 for additional 10 files (Note: I will run this tool for 1000s of files and for development I usually limit it to less files

I believe @therealprof concerns are valid. I am not able to justify why dev build is so slow compared to pascal dev build or C# dev build.

This is what I am worried about. Most Rust developers are not interested in debugger and I agree to their view. Because, their work is making extra-ordinary libraries or system-level things which I could never imagine. I am a mediocre programmer who want to use Rust as an enterprise level app. Usually, Java or C# is preferred for this but Rust can be used to speed things up. Now, this app will be continuously modified with new business requirements and many people use it. Likewise, many developers will be modifying it and so it is hard to maintain quality of the app. This brings edge case bugs like Vec<_> being empty. When a bug arise, it is hard to debug with the current dev build and debugger. Some options and their challenges are:

  • I can optimize my dev build but then I will run into optimised variables
  • I can selectively optimise my crates and I will have to fiddle with the settings often. And worst, I will have to educate developers on how to selectively optimise
  • If an issue arise and it takes 5 mins for it to panic - It is unacceptable when time is precious. And, when the panic message is cryptic, I have to rerun the build multiple time to identify the bug
  • After identifying the location of the bug and when I inspect variables, debugger shows pointer and not actual value which is so annoying. On top of it, I can't evaluate the line which is available out of box in most of the languages

This again depends on the app complexity. For example, if my app has a function which is called in 5 places and that function panics, I will have to add println! in each of those 5 functions. Now, if I want to inspect all the variables in that function, I have to write println! N times. Which is not productive. I know some people will call it bad programming and I should have unit testcases to capture this; But in real world, no one is interested to spend too much time in testcases.

This is true for small libraries and system level program. But if your program has lot of if/else condition, you are not worried about type errors but more on accuracy. An example is OpenFoam which is a CFD solver written in C++. Most of the time I spend is not writing the code but on improving result's accuracy. I will have to run some iterations and then inspect the variable. After inspecting, I would realise my mistake in some if/else condition.

In summary,

  • I feel Rust is very good lang for performance and writing code
  • Dev build is so slow compared to some other langs I have tried
  • Current debugger is horrible because I can't set conditional breakpoint, evaluate expressions, ndarray display and so on. I hope HeadCrab fix all these and boost Rust user-base
  • In my opinion, proper debugger and IDE support is what needed by most academic and enterprise developers. I feel current user-base of Rust is limited only to experts
  • Println! is not a way to debug when I have several 1000s of lines in my code

Sorry for a big content. Hope you get time to read and reflect on this.

All the best to HeadCrab developers!!! Eagerly waiting for your release while waiting for someone to optimise the dev builds :frowning:


The idea is not to clutter println!() all over the map but only introspect the interesting cases. Things like an empty Vec<_> (as you mentioned) should normally not be an error condition but a totally normally occurrence; if it is unexpected for your program you might want to return an Err or if it's a grave problem even stick an assert!() to abort with a panic.

Macros like https://doc.rust-lang.org/std/macro.dbg.html or (to debug your empty Vec<_> problem)` https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.inspect are super powerful to find such problems. I definitely would not want to use a debugger for that! Better invest some time into proper error handling, e.g. via https://crates.io/crates/anyhow to create some useful error messages with a useful stack of errors and contexts.

1 Like

I use the debugger to find logic errors, something Rust's language features can't help with. In other languages, I rarely have code that I haven't single-stepped through at least once as I'm developing the algorithm. (I must have single stepped the APL Roman numeral program 1,000 times, and I still don't understand it.)

I once participated in the Parallel Tools Consortium. The first thing we did was poll users to learn what their favorite debugging tools was. The winner was "printf" by a lot. That was my strategy with Rust until I started working with a former Netflix employee. He showed me how to create trace files that would allow me to find errors without needing to re-run the program after inserting println!s.

I definitely agree with your point but as I said, the real world has more mediocre programmers like me and have less time to invest in err handling.

Also when you have a complex app, your recommendation of Err might not help. For example, when a function throws an error I will not know which function called it (backtrace is not as informative as Java or C# even with rust_backtrace=1). Hence, in the parent function, I will have to modify the error message to include that function information (like local variables state). Likewise, I have to do the same for its parent function and so on. Do you think my code will have readability if I have a match statement in each function call? I feel it will be too difficult to read and understand.

Or, if you think I am wrong and there is a Rust way of doing it clean and clear, please let me know.

Edit: I see that anyhow has the capability to set the context message on the parent function. I will try that and thanks for pointing me to it
Edit 2: I have to add the context message for each function call. It would be nice if we have one context set in each function rather than each calls. Is there such a feature?

Believe me, investing in error handling will not only make your code cleaner, faster and better to maintain; it's also a much faster way of developing than relying on a debugger.
Professionally I mostly write C code and even there (as mentioned before) a debugger is a last effort measure when everything else fails and luckily there're only like 5 days a year when that is the case; everything else is handled by good programming habits, tools like clan-static-analyzer and valgrind, continuous integration and boatloads of unittests.

Getting code right and clean in Rust is magnitudes easier than in C!

Getting back on topic... One problem generic debuggers have is that they have pretty much zero understanding of Rust. They look at the running binary and debugging symbols and then there're some plugins with some demangling and rudimentary code parsing capabilities and that's pretty much all she wrote. That makes them even less useful with Rust than for other languages. The interesting part about headcrab is that it not only has the potential to address all of those deficits but completely turn it upside down and integrate Rust features but maybe also some useful tools.

I'd love to see things like callgraphs, function call profiling, proper disassembly with symbol rematerialisation and Rust side-by-side, heap and stack size analysis, MIR output support (maybe even MIRI integration?), ...

I very much hope I'll never have to single step a Rust program ever...

1 Like

Precisely! Single-stepping is really inefficient, time-consuming, and error-prone -- it's too easy to step farther than needed. No wonder it's easier & faster to use dbg! or println! instead.

And your comments about 'generic debuggers' are also spot on. This is something I've had in mind when starting this project - while the techniques such as single-stepping aren't really helpful, the underlying principles behind them are. What I'd like to see is e.g. an ability to step through code programmatically or to set watchpoints with predicate functions written in Rust.

And then there are more interesting techniques & tools built on top of the basic debugger primitives. Dynamic tracers, deterministic debugging or omniscient debuggers can be indispensable in complex projects.


September 2020 progress report

This month, we have been working on the following features:

  • GUI example: It demonstrates how Headcrab functionality can be used to visualize the state of a running program. The demo shows backtrace and a source view, and it can be expanded to display more contextual information.

  • Code injection demonstrates how Cranelift can be used for code injection. This allows to intercept and replace functions in runtime and it has many useful applications. For example, it can be used to run code snippets in the debugger command line, or for dynamic tracing to record information about a debuggee process.

  • Command line improvements: highlighting and completion for the REPL example and inline source view.

  • Breakpoint support on Linux: functions to set user breakpoints, which is a cornerstone feature of any debugger.

You can find the full report on our website.


Do not know if debug code compilation speeds up