Why using the question mark operator generates a stack overflow?

[glevco]

I was getting fatal runtime error: stack overflow in some code I was working on and I was able to create this minimum reproducible example:

fn f() -> Result<[u8; 800_000], ()> {
    todo!()
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test1() -> Result<(), ()> {
        f();
        Ok(())
    }

    #[test]
    fn test2() -> Result<(), ()> {
        f()?;
        Ok(())
    }
}

Running cargo test test1 fails with not yet implemented (as expected), but running cargo test test2 fails with fatal runtime error: stack overflow. Why is that?

Note: the value of 800_000 is probably different depending on your platform. By increasing it I was able to get stack overflow in both tests, of course, but there's some middle ground that generates the described behavior.

Follow-up question: In my original problem I was designing a struct with multiple fields, including arrays of other structs. After encountering stack overflows during tests, I moved some of those arrays to Boxes, which fixed the issue. So, how can I determine some reasonable max size for structs in the stack? I was "lucky" to get these errors during tests, but it would be possible that I only saw them in production.

Thanks in advance!

[OptimisticPeach]

If you take a look at these functions in the playground and ask it to generate MIR, then what you'll see is that test1 allocates a variable whose size is on the magnitude of [u8; 800_000] on the stack once, whereas the machinery to do the ? operator in test2 requires allocating three such variables whose size is comparable in magnitude to [u8; 800_000].

I'll take a wild guess and say that test1 also fails if you bump 800_000 up to 2_400_000.

[richardscollin]

You can see your max stack size on linux (and maybe all unixes?) with ulimit -a.

I would avoid stack allocated structs which have a size above ~100 bytes.

There's a clippy lint which should warn about this problem if it's a paramater large_types_passed_by_value. They use a size of 256 bytes as the default threshold.

[scottmcm]

@glevco

fn f() -> Result<[u8; 800_000], ()> {

The other posts said things in this direction already, but to be explicit about it:

You almost never want really long arrays like that. If you need a buffer more than, say, thousands of bytes, just use a Vec<u8>.

[drmason13]

@OptimisticPeach

the ? operator in test2 requires allocating three such variables

Do you know where the additional stack allocations come from?

My first guess would be match somehow while pattern matching Ok and Err variants but am curious for more details.

[gretchenfrage]

@scottmcm

You almost never want really long arrays like that. If you need a buffer more than, say, thousands of bytes, just use a Vec<u8>.

Or, if for some reason you really want it to be compile time fixed length, Box<[u8; 800_000]>--although that has some pitfalls, I think you might need to initialize it like

let a: Box<[u8; 800_000]> = vec![0u8; 800_000].try_into().unwrap();

To avoid the array being temporarily put on the stack when constructing it.

But probably just use Vec<u8>.

[richardscollin]

@drmason13

Do you know where the additional stack allocations come from?

I suspect it has to do with how ? is implemented internally with FromResidual and the Try trait. I would guess that it gets desugared into the values that you can see from the MIR:

let mut _1: std::ops::ControlFlow<std::result::Result<std::convert::Infallible, ()>, [u8; 800000]>;
let mut _2: std::result::Result<[u8; 800000], ()>;
let mut _3: isize;
let _4: [u8; 800000];