Code generation for local stack variables in release vs. debug mode

As part of my OS in Rust, I have the following system call entry point:

#[no_mangle]
#[naked]
#[inline(never)]
unsafe extern "C" fn syscall_handler() {

    // switch to the kernel stack dedicated for syscall handling, and save the user task's details
    asm!("swapgs; \
          mov gs:[0x8],  rsp; \
          mov gs:[0x10], rcx; \
          mov gs:[0x18], r11; \
          mov rsp, gs:[0x0];"
          : : : "memory" : "intel", "volatile");


    let (rax, rdi, rsi, rdx, r10, r8, r9): (u64, u64, u64, u64, u64, u64, u64); 
    asm!("" : "={rax}"(rax), "={rdi}"(rdi), "={rsi}"(rsi), "={rdx}"(rdx), "={r10}"(r10), "={r8}"(r8), "={r9}"(r9)  : : "memory" : "intel", "volatile");
    // do stuff with rax, rdi, rsi... 

This works fine in debug mode, and in release mode (with debug info enabled) because it generates assembly code that stores the local stack variables like rdi, rsi, etc at negative offsets from the base pointer rbp.
For example, here's the generated code:

<syscall_handler>:
swapgs 
mov    %rsp,%gs:0x8
mov    %rcx,%gs:0x10
mov    %r11,%gs:0x18
mov    %gs:0x0,%rsp
mov    %rax,-0x1f0(%rbp)
mov    %rdi,-0x1e8(%rbp)
mov    %rsi,-0x1e0(%rbp)
mov    %rdx,-0x1d8(%rbp)
mov    %r10,-0x1d0(%rbp)
mov    %r8,-0x1c8(%rbp)
mov    %r9,-0x1c0(%rbp)
movb   $0x4,-0x1b1(%rbp)

That code works fine, because my syscall handler runs with a stack pointer that points to the top of the current kernel stack (as usual), meaning that it's okay to use negative offsets from the stack pointer / base pointer (base pointer rbp is set before this based on the stack pointer value).

When I build in release mode without debug info, it generates code that uses positive offsets from the stack pointer itself (rsp, not the base pointer) as locations for the local stack variables. This is really weird and causes a problem because the memory above the current stack pointer rsp is out of bounds.

Here's the code generated in pure release mode without debug info:

<syscall_handler>:
swapgs 
mov    %rsp,%gs:0x8
mov    %rcx,%gs:0x10
mov    %r11,%gs:0x18
mov    %gs:0x0,%rsp
mov    %rax,0x1c0(%rsp)
mov    %rdi,0x1c8(%rsp)
 mov    %rsi,0x1d0(%rsp)
mov    %rdx,0x1d8(%rsp)
mov    %r10,0x1e0(%rsp)
mov    %r8,0x1e8(%rsp)
mov    %r9,0x1f0(%rsp)

Why is this code being generated, code that uses a positive offset from the stack pointer? That strikes me as very strange.

Is there any way to avoid that or change the code generation somehow?

StackOverflow has the answer: assembly - Code generation for local stack variables in release vs. debug mode - Stack Overflow

Thanks Steve. After talking with Peter on SO, I still am curious why Rust (or I suppose it's actually LLVM's backend) decides to use positive offsets from the stack pointer. I'm no compiler expert, but I haven't been able to find any resources discussing this online. Is this behavior controllable, or is it a consistent decision that I'll have to work around?

The System V x86_64 ABI defines a 128 byte red zone: Red zone (computing) - Wikipedia

You can decide whether your OS should have such a red zone. For instance, rust.ko (for Linux) sets "disable-redzone": true in the target file.
https://github.com/tsgates/rust.ko/blob/master/x86_64-unknown-none-gnu.json

This appears to be "overshooting" the 128 byte redzone. The debug vs release might be a case of rbp being an allocatable register in release, but reserved (for base pointer, frame walking) in debug. But as the SO answer says, it's possible the compiler is confused between this being a naked fn yet asking the compiler to manage locals.

1 Like

Thanks all. The disable-redzone option was already set, so I think it must boil down to my lack of understanding how local stack variables are allocated in the presence of no frame pointers, especially in naked functions. Much appreciated.