How do you guys do debugging in Rust?

Editors and IDEs
1

Hello,

I am often debugging rather convoluted algorithms in Rust. I usually attach the rust-lldb debugger to the running PID and start inspecting the state of the running program, and check that all variables are updated as they should. But I have many issues with how the debugger works especially regarding creating conditional breakpoints. Rusts debugging experience seems good for view the state, but not for interacting with it.

  • What is your debugging workflow in Rust?
  • What applications are you using?
  • Does anyone have a perspective to C++

I have found it difficult to debug Rust exactly the way that I think it should be done. I am wondering whether it is just me who has been unable to attain an excellent work flow.

1 Like
2

I don't know whether this is helpful, but so far I have not yet made use of any debugger when writing Rust.
When I write software I try to avoid "convoluted" algorithms, try to keep functions short and structs small.
Up until now this lead to rather good testability of the software components.
Given a good test coverage and Rust's strict type system, passing tests and the fact that everything compiles without any warning, plus a strict clippy config, has so far been a trustworthy indicator, that everything works as intended.

Given your hint at checking the changing state of data within the program, I can only say, that I generally try to avoid state if possible by implementing functional programming practices.
In places where state is handled and / or changed, the modifications are usually small and covered extensively by unit tests.

10 Likes
3

I work pretty much as @Schard already explained but when I need to debug the "flow" of a non-trivial piece of code I start by writing a test for it, then I debug the test from VSCode (small "debug" link near the test function name) and single step through it. The portion of code I need to analyze is usually small, so I don't need complex tools like conditional breakpoints. The only pain point is inspecting Rust data structures, especially enums, because there isn't a nice visualization for them.

3 Likes
4

It is helpful! When coding I just have a feeling that I am doing something in an inefficient way, and I was contemplating whether there is a better way of doing things. Below I have tried to add some more context to the current pain points when developing.

The main application is a randomized multi-threaded optimization system consisting of a series of algorithms (metaheuristics). The algorithms needs to be somewhat convoluted, as that convolution is the primary reason that the algorithms can optimize the system.

You can look at it as having a large nested struct that holds a lot of business data. This business data has a set of invariants in the struct that always have to be upheld. Finding permutations of this nested struct that have a high quality (meaning minimizing a mathematical function) is difficult as so many parts of the struct have to be internally consistent while mutating basically every conceivable part of the struct.

I am making progress but it is slow. I think that one of my issues is that I do not know before hand why the algorithms will "fail" as it is not that a function does not perform correctly, but that higher level invariants are not upheld. And because the state mutations are random AND multi-threaded (asynchronous). It is really helpful to always run the binary through a debugger so that I can always pause execution and inspect the higher level invariants.

The main issue for rust so far is that, due to the state being randomly mutated I cannot simply step through the code and find the issue, I would have to use conditional breakpoints to stop the code when some condition occur, and from there start stepping through the source code line-by-line.

5

For cases like this one I usually write some "meta" code that runs whenever the data has been modified, checks the invariants and prints a detailed dump of the program state whenever it finds something wrong. I find this method much more efficient than a debugger because I can express the conditions in the same language I use for the code. Sometimes (but I don't know how to do that in Rust because I still haven't had the necessity) I enter the debugger directly from my analysis code.

2 Likes
6

Do you mean something like?

impl AlgorithmStruct {
    fn asset_high_level_invariant(&self) -> Result<(), AssertError> {
        if self.<field>.check_invariant_x() {
            return AssertError::X
        }
        if self.<field>.check_invariant_y() {
            return AssertError::Y
        }
    }
}

And then the caller would at specific lines in the code call these methods?

fn main {
    let algorithm = AlgorithmStruct::new(...)
    
    algorithm.assert_high_level_invariant().expect("The invariants should always be upheld")

}

I just asked ChatGPT about it, you can actually do this (requires that you run the binary through a debugger to work e.g. lldb target/debug/<binary>):

impl AlgorithmStruct {
    fn asset_high_level_invariant(&self) -> Result<(), AssertError> {
        if self.<field>.check_invariant_x() {
            // If run through a debugger this will activate the breakpoint
            std::intrinsics::breakpoint()
            return AssertError::X
        }
        if self.<field>.check_invariant_y() {
            // If run through a debugger this will activate the breakpoint
            std::intrinsics::breakpoint()
            return AssertError::Y
        }
    }
}

This way you may be able to get the best of both worlds? Leverage the language's rich structure to determine the complex conditionals and then activate the breakpoint to start inspecting the state?

7

As-is, you can't, since std::intrinsics::breakpoint is unsafe. And since it doesn't have any kind of "safety" comment, we don't know how to use it soundly, so replacing it with unsafe { std::intrinsics::breakpoint() } might be not fine, too.

2 Likes
8

Yes you are correct.

I tested it, it seems that:

   1    fn main() {
   2        unsafe { std::arch::asm!("int3") }
-> 3        println!("Hello, world!");
   4    }

is a better approach, if you want to make a setup like this, which you could argue that you should not...

9

Echoing similar sentiments, dbg!() and tests are as close as I need to get to a live debugging session in practically all cases.

The primary exception is when I use a crate with some unsoundness that segfaults or misbehaves in confusing ways. The first tools I reach for to handle those situations are miri (if the dependency doesn't have FFI) or -Z sanitizer=address (if it's supported on the target platform).

Between these tools, a live debugger is the last resort. And the one time I had to use a debugger, I didn't find what I was looking for anyway, and gave up.

3 Likes
10

In C++ I would sometimes use a debugger, mainly for core dumps. In that case it would be a multiarch gdb (since u usually need to handle core dump from embedded linux systems).

In Rust I have used rust-gdb once I think. It is not often needed. And when it is, it isn't ideal as it doesn't work as well: calling into the debugee usually doesn't work, printing of enum variants doesn't work well, etc. Basically you can feel that the debugger isn't made for Rust.

There is also this project: GitHub - godzie44/BugStalker: Rust debugger for Linux x86-64 (though I haven't had the opportunity to try it out, probably will next time I need a debugger for Rust).

2 Likes
11

The sort of thing you're describing sounds like it might benefit from property testing?

In general, using the real assert!() macro tends to work better as it stops where and why you had the problem in the debugger and outside with a full stack trace (if requested)

3 Likes
12

Tests, asserts and logs are all great. When that is not enough, I use GDB.

1 Like
13

I think that this is very promising, although that said, I think that the kind of testing setup that I would have to create for the algorithms would be enormous. I can provide more details if you want?

The crate that you link is promising, especially for the mathematical problems that I try to solve and optimize, it just feels like I have to extend my project if I want to test that way. Or maybe I am just not in the habit of using proptest.

I usually use debug_assert!(function_asserting_complex_invariant()) at runtime and then, because the whole system (and individual algorithms) are randomized, it gets you somewhat the same as proptest. The main distinction that the "proptest" only covers the states that the algorithms actually calculate while optimizing.

14 
fn breakpoint<T>() -> Option<T> {
    unsafe { asm!("int3") }
    None
}

fn assert_complex_invariant(&self) -> Option<<ALGOTIHM_TYPE>> {
    // CODE 
}

fn main() -> Result<(), Error> {
    let mut algorithm = Algorithm::new();
    
    algorithm
        .assert_complex_invariant()
        .or_else(breakpoint)?;

This is what I am current doing, i.e. making a breakpoint function.

I think that Rust really needs a debugger like the BugStalker (I am not sure how extensive it is implemented).

If you could use something like BugStalker to:

  • Prints variables Debug trait
  • Allows conditional expressions on Debug fields/values (is this even possible to implement in a debugger?)

I believe that I could speed up my development of the complex algorithm parts by 1.5x to 2.0x. Mostly to inspect all the complex state without having to recompile the whole binary to insert a few dbg!()'s that may not show what you were actually looking for, and then you would have to recompile again...

15

I would put it differently. Property testing and debug_assert!() are used together. The invariant assertions are intertwined within your algorithm. The property tests live outside of it, providing stimulus and deterministic pseudorandom seeding.

Deterministic seeding is the quintessential ingredient of property testing, allowing trivial reproduction of bugs (assertion and test failures) and collecting a regression test suite. Without it, you are just randomizing inputs.

Your algorithm sounds a bit different from what you typically want to property test, though. If you could replace its PRNG with a deterministic PRNG, then the results of the tests are almost certainly worth far more than the time required to implement. If not to find all of the bugs and missed optimization opportunities, then as a peace of mind that future changes will not regress any of that hard work.

As for the intrusive breakpoint idea, yeah, it requires recompiling when you want to change something. But you save orders of magnitude more time than running with complex breakpoint conditions that are evaluated out-of-process in an exception handler. You already have a general-purpose language that knows everything there is to know about the type system.

Not only can the breakpoint condition be arbitrarily complex at almost no cost, but the script that runs in response to the condition can as well; print all of your state in any format you prefer, persist it to disk, modify it in place with other complex rules, do whatever you can think of.

You have the source code. An external debugger is most valuable when you do not.

16

Your algorithm sounds a bit different from what you typically want to property test, though. If you could replace its PRNG with a deterministic PRNG, then the results of the tests are almost certainly worth far more than the time required to implement. If not to find all of the bugs and missed optimization opportunities, then as a peace of mind that future changes will not regress any of that hard work.

Maybe this is a good idea, the code changes often, and the proptests would have to do the same. There is also the added issue of the algorithms solutions being shared between each other through pointer swapping (see GitHub - vorner/arc-swap: Support atomic operations on Arc itself), so many of the hardest bugs arise due to having randomized algorithms with asynchronous swapping of state. I cannot control the timing of swaps; you could add synchronization, but my Ph.D. is on how to avoid synchronization primitives (e.g. Mutex, GC, etc.) while optimizing, so any kind of determinism in proptest cannot be achieved for this part (or maybe I am mistaken?), as it may not catch all bugs.

Side note:
You might think that message-passing between threads would be a good idea but it causes duplication of state which is a nightmare to handle while optimizing. Refer to the "failure" of multi-agent systems

Not only can the breakpoint condition be arbitrarily complex at almost no cost, but the script that runs in response to the condition can as well; print all of your state in any format you prefer, persist it to disk, modify it in place with other complex rules, do whatever you can think of.

Do you mean that I simply dump all the state into a JSON like structure and then process the data with something like nu (nushell), and then filter through that?

You would have to write code like:

    fn main() {
        let a = <COMPLEX_TYPE>;
        event!(Level::INFO, logging = ?a);
        a.destruct_solution();
        event!(Level::INFO, logging = ?a);
        a.construct_solution();
        event!(Level::INFO, logging = ?a);
        a.accept_criteria.simulated_annealing();
        event!(Level::INFO, logging = ?a);
        a.is_better() {
             a.make_atomic_pointer_swap();
        }
}

And then I would have look at deltas between the different log lines, instead of calling next in a debugger?

Also, each algorithm max out a thread completely, so what I have shown above will produce GBs of logs per minute

17

I'm not sure I follow. Timing was never on the table in my mind. But even projects like loom and coz show how powerful deterministic randomization is. As with fuzz testing, loom does catch all concurrency bugs [1]. That is its purpose.

No, I don't think that is what I meant. I'll adapt the other code you posted:

fn breakpoint(algorithm: &Algorithm) -> Option<Result<(), Error>> {
    // Do useful debugger scripting here directly,
    // rather than relying on an external debugger
}

fn assert_complex_invariant(&self) -> Option<&Self> {
    // CODE 
}

fn main() -> Result<(), Error> {
    let mut algorithm = Algorithm::new();
    
    algorithm
        .assert_complex_invariant()
        .and_then(breakpoint)
        .transpose()?;

I already addressed the "recompile issue" as mostly a non-issue. When your condition raises the breakpoint, the job of the "script" is to record everything you are interested in, then either continue running the algorithm or terminate. You're "scripting debugger actions" here instead of using the commands like next in an external debugger.

If your goal with debugging is "look at deltas between the different log lines", then sure, do that. I suspect you have needs to inspect, modify, and rollback your state in ways that a general-purpose debugger is not really suitable for. Maybe you are really only familiar with JSON, I don't know. But there are dozens of ways to serialize state, some of which are better than others for analysis. Then again, Rust can do that analysis without ever writing JSON.

"Just write the code to do all of that" is a bit reductionist, but sometimes it's the best way to proceed.

  1. With a large grain of salt that concurrency in the system under test is made deterministic. ↩ī¸Ž

18

My mind has been boggling at what you are building here. If I understand, in general handwaving terms, what is going on:

  1. It comprises all manner of algorithms or even parts of algorithms.
  2. It somehow selects those algorithmic parts at random and/or in response to input data and composes them into the final algorithm.
  3. All this is happening on multiple threads/tasks which will further stir the pot.
  4. The aim is not to have an synchronisations like mutex/channel.

Basically what I'm hearing sounds like a soup pot being filled with random ingredients and stirred. The final test being a tasting by the chef.

I think I have to agree that writing tests for that may be, shall we say, tricky. And that pecking at it with a debugger might help illuminate what goes on.

Philosophically though I have a question:

If one does not create reproducible tests for this thing and ensures its correct operation by spending a few days with the debugger observing it's behaviour, then how does anyone other than yourself know that it works at all? Is it just a case of "Trust me, I wrote it, I debugged it, it works."

My experience is that all the time one spends poking around with a debugger could be better spent creating tests for the things one wants to look at. Unlike debugging sessions those tests are permanent. You don't have to do all that debugger work again when you change something, the tests show you what you broke.

Surely there must be some way to get your project into a more manageable, predictable, testable, shape.

19

Hi, currently no, but its a next big deal for me.

Really don't understand what you mean here; maybe watchpoints are what you need?

1 Like
20

I usually get by fine with dbg! and unit tests. When those aren't enough, Rust Rovers debug integration is really good.

1 Like