I have large (many GB) result Vec that is largely used in two phases:
From 128 worker threads on multiple numa nodes seldomly write to non-overlapping indexes
Post process the data by sorting and processing consequtive parts of the array on 128 worker threads across multiple numa nodes.
In particular the consequtive parts in 1. and 2. are note the same, so the poat-processing step needs a different division than part 1.
Now, the probem is that when I forst allocate this large result Vec it will be allocated in the physical memory on 1 numa node (where the allocating thread is active). This is fine for phase 1 with seldom writes since it is not memory bound, but phase 2 (sorting) is very much memory bound and the memory beeing allocated on a single numa node means I can only utilize a fraction of the memory bandwidth.
Allocate many hundreads of 2Mb large pages in consecutive process adress space on interleaved numa nodes.
Store the pointer to the first allocation and the length in a struct with a Drop implemenation to make sure that VirtualFreeEx can be correctly called on drop.
Create a not unsafe public method get_usafe(&mut self) that uses from_raw_parts_mut to return a mut slice to the entire unerlying allocation, now interleaved across multiple numa nodes.
Barring bugs, does this sound like a sound approach and am I correct in thinking that this should allow me to create a sound such struct that is safe to use from normal code?
as long the whole allocation resides in a single consecutive chunk of virtual memory address, it should be ok. just pay attention to uninitialized values, for example, if the allocated page cannot be guarantee to contain known valid value, you can only create slice of type &mut [MaybeUninit<T>], or you must use raw pointers to initialize the memory before creating a slice.
this is the your Vec-like container that owns the allocated the memory. you may also store the ptr, len, cap triple just like Vec, e.g. if MaybeUninit is used internally but not exposed to the public interface.
if you want this operation to be NOT unsafe, then you must have unsafe somewhere else, e.g., the constructor of the custom Vec-like container can be a good place as a separating layer between safe and unsafe code.
in this case, you can just implement DerefMut<Target=[T]> so the custom container can be indexed just like a slice, it should be more convenient than explicit calls to the get_mut() method.
if the container was unchecked at construction, then you must do the necessary checks in the get_mut() method internally, since constructing a slice form a raw pointer using slice::from_raw_parts_mut() is an unsafe function, but you want your function get_mut(&mut self) to be NOT unsafe externally.
however, soundness is not that easy. for the safety condition required by slice::from_raw_parts_mut(), the validity of the pointer is easy to check, but the non-aliasing or exclusivity property is not so obvious, since it cannot be checked locally, you must ensure all other APIs cannot create (unintentional) aliases.
Good point that the memory must be valid, hower, as i understand it all bit patters are valid for f32 values, so provided that I do not make it generic over the type but stricutly for f32 that should be fine? (alternatively require T: Default and initialize all memory)
You write that soundness os not that easy which i fail to understand, since the only pointer in existance is the one kept in the struct and the "get_slice" method I proposed takes &mut self that should ensure that there only ever is one reference, right?
Alternatively as you write, if I create the slice during initialization and store that slice in the struct that will be the only slice?
So I fail to see how someone else could create a pointer to the same underlying data (barring them using unsafe operations - but then all bets are off anyway?)
All initialized bit patterns are valid f32 values, but the Rust abstract machine also allows a special uninitialized value for every byte that doesn't represent any specific bit pattern. Pretty much anyÂą read of an uninitialized byte is UB, which allows an allocator implementation freedom to return pointers that are backed by memory that will either trap or return spurious values if they're read from before being written to.
Âą The exact rules are still in a bit of flux, as far as I understand things.
Interesting, and I will then just stick to actaully initializing the memory to be safe, but it does not make fully sense to me:
I get the memory from a system call (rust abstract machine cannot know what is returned - so it cannot assumed that its all uninitialized) - what if e.g. the OS gurenteed that all returned memory would be zero-initialized - would it then still be UB to use it directly in Rust?
The compiler always assumes that memory you read has been properly initialized, which is why reading uninitialized memory is UB: If that assumption is faulty, there's no backstop to ensure the program behaves sensibly. If you get your memory from a syscall (or other FFI) that guarantees it's been zeroed, then it has in fact been initialized, the compiler's assumptions are true, and there is no UB.
Evaluating whether your code is sound requires answering the harder question of whether you've properly isolated the safe code from any possibility of UB. For example, the zeroing might be an extra guarantee of the system kernel you happen to be running on beyond what's guaranteed by the relevant spec. In that case, running the same program on a different machine without the extra guarantee could manifest UB if you don't verify it in some way.
Thank you for your patience- although I admittedly still do not follow fully - in the case of f32 my point is that it does not matter if the OS has zeroed the meory or not - since all possible bit patterns in the memory returned from the OS (regardless of what the OS does) are valid instances of f32 and thus &[f32]s?
I.e. what I fail to understand is how the rust-abstract-machine's concept of the "extra uninitialized bit pattern" you referred to could be a problem here, buy using unsafe to create a slice from my pointer I explicitly tell the compiler that the memory it points to is initialized to some valid [f32;X] value - which I can gurentee that it is regardless of what the OS does since all bit parterns are valid f32s.
Don't get me wrong - Im not argueeing for that I should not initialize the memory - I am just trying to understand Rust's model and why my thinking is flawed.
I think it might be easier if I actualyl write the code - so I will do that and update with a link once done.
One example would be if you get given memory that won't be mapped to a physical page by the kernel's virtual memory system until it's written to. Then you could be in a situation where everything reads as a zero initially, but writing into one of the f32s causes a page fault that assigns actual non-zeroed RAM to be assigned for a few hundred f32s on either side, changing their values arbitrarily.
That sort of changing memory under Rust's feet has to be dealt with quite carefully: It's certainly UB if you have an &f32 or &mut f32 reference to any of the values that changed unexpectedly, and probably for some raw pointer operations as well. &mut MaybeUninit<f32> should be fine, though, because it alerts Rust that this sort of thing might be happening.
I obviously am out on very deep water here, but how could the value read as zero initially?
Since Rust does not known what content is in the (FFI) memory returned by the OS (in partiuclar - the compiler has no reason to suspect aht the memory is zeroed initially) then the rust compile must issue a memory read instruction for any operation wanting to know the content of the allocated memory?
And when that read instruction is issued the page fault will happen and the actual value of the memory will be returned?
In the particular situation being discussed, the idea is that the OS initially maps a singleton page which is read-only, probably zeroed, and used for many things, then swaps it out for a read-write page on the first write fault.
Another situation is that a memory cell could be in a physically ambiguous state if it has never been written to, such that what value is read varies from electrical noise.
These are merely two concrete examples of the kind of phenomena that can lead to misbehavior when reading uninitialized memory. Whether or not either of them applies to your OS and hardware, it is still UB to do so. But all you need to do is allocate zeroed memory, and the problem is addressed. I recommend just doing that and putting further thought into other aspects of your unsafe code.
While I fully agree that it’s UB to read uninitialized memory, the initialization doesn’t have to be done by Rust— If an FFI call gives you access to memory that behaves like it’s initialized, i.e. maintains a consistent value between multiple reads and reads the last-written value after a write, that should be sufficient.
If the OS promises that the memory is zeroed, then you're fine. And TBH most tend to at least have a flag to ask for it, if it's not just the default (to avoid cross-process memory content leaks).
In your specific case, MEM_COMMIT guarantees it's zero-initialized once you later read it, so you don't need to worry about uninit values in the memory as long as you pass that flag to VirtualAlloc. (Assuming you don't go and explicitly write uninitialized values to the memory yourself.)
@RalfJung has a really good blog post explaining why "all bit patterns are valid" doesn't mean it's okay to read uninitialised memory. I'd highly recommend giving it a read.
Those links were very instructive. So based on all the input I got here I have now implented a (windows only) version of an InterleavedArray which I then believe is safe & sound to be used from safe code. Link: Rust Playground
Running the code on a 2-Numa node computer generates (showing that half the blocks are allocated on node 0 and half on node 1):
1
Ok({1: 128, 0: 128})
If anyone wants to have a look at the code it's apprechiated - I think I have no accounted for the points raised such that it should be safe.