Cargo test with a 80MB stack size?

Is there a simple way to run a cargo test with a 80MB stack size?

I can't seem to change it via ulimit -s, and the reason for the 80MB stack size is a recursive descent parser on some deeply nested structure.

ulimit only affects the stack size of the main thread, which grows automatically. Tests run in parallel in separate threads, which have fixed stacks. You could set RUST_MIN_STACK to affect all threads, or have your deeply-nested test launch its own thread with a larger stack.

See std::thread - Rust

3 Likes

If you are using tens of megabytes of stack frames to recursively parse a deeply nested structure, then tat might be a hint that recursive descent isn't a good parsing strategy to use. After all, if stack overflows are a legitimate risk to your code, then increasing the stack size during testing is just hiding the problem.

I'm not sure how feasible it is (i.e. you might not have an extra week to rework code or it may not be under your control), but you might be better off using a stack of state machines to move your stack frames to the heap. Alternatively, you could switch to a different parsing strategy (e.g. LR or whatever), but that would require a complete rewrite.

2 Likes

you could use

let builder = thread::Builder::new().stack_size(80 * 1024 * 1024);
let handler = builder.spawn(|| {
    // thread code
}).unwrap();
handler.join().unwrap();

to execute your test.

3 Likes

In theory, I 100% agree with you. In practice, this is a parser I did not write on a data file I did not create ... and I just need to do a one time task of converting it into a sane format.

I think my use of cargo test threw the question off, implying it was a unit test -- when in actuality, I write small programs with #[test] just to get the free "green arrow" in IntelliJ to click on.

1 Like

Instead of statically setting the stack size to 80MB you could consider using stacker to dynamically grow the stack as required. With stacker you place calls to maybe_grow in any recursive functions that may overflow the stack and stacker will grow the stack whenever there is insufficient room remaining on the current stack. This way you don't need to pre-allocate a large stack in advance.

Edit: Just saw the previous reply. This is only really relevant if you can modify the parser.

2 Likes

This is an interesting technique. If I understand it correctly, it only requires (1) finding the recursive calls and (2) inserting a single line.

In contrast, the previous solution required rewriting a recursive algorithm as an iterative one by simulating the stack with a Vec.