Undefined behaviour and in-process debugging or root allocators?

Up front I want to say that this is a question of curiosity and learning, not something I'm trying to actually do currently.

Rust (famously) doesn't allow accessing uninitialised memory or accessing memory outside of allocations without causing UB. That does to cause issues in certain situations though, and I'm wondering what the resolution to these issues are:

  • Scenario 1: I'm implementing something like AdressSanitizer in Rust: some sort of loadable in-process debugger. It clearly needs to be able to access and print (to the user) various memory in dubious state. How do we go about this without causing UB? Miri gets around this by being on the outside of the interpreter, as does valgrind. Ptrace again will at most cause UB in some other process (not myself). So it is the in-process case I'm interested in here.
  • Scenario 2: I'm implementing the "root allocator" in an operating system or no-std or similar. I can't just fall back to asking the OS for mmap or brk and then portion that out to the my callers. It isn't allocators all the way down after all! How does this root allocator avoid UB due to accessing memory outside allocations? How do we resolve this?
1 Like

You could take the same approach C and C++ devs take: realize the optimizer will tend to fail to make inferences from the UB in this particular case. If things break, use a compiler option to disable the problematic optimizer pass.

There are no practical difference.

The code that you are interested in is in separate library and from compiler's POV the whole memory except for internal data structures in that same library belongs to huge array filled with initialized memory (we just don't know what's there).

The opposite code, added to the actual program only ever touches shadow arrays where everything is from of UB.

In effect your program includes two parts and each one is UB-free. Even if they live in the same process in the end the only part that may see both in not UB-free state is linker.

And linker doesn't care about UB.

In the same way. Mark memory outside of your program as one gigantic array and treat it as if it's [MaybeUninit<u8>]. From compiler's POV there would be no UB and linker doesn't care.

1 Like

Here's some related conversation about "external memory" from the embedded POV. (The rules around provenance are still evolving.)

1 Like