Is it possible to allocate `Arc<[MaybeUninit<u8>]>` with bigger alignment?

It seems possible to alloc Box<[MaybeUninit<u8>]> with the desired alignment using alloc::alloc::alloc + Box::from_raw. But for Arc you need to allocate ArcInner which is private and thus inaccessible.

You can first allocate the Box and then convert it to Arc, but this will always realloc which is not cool.

So the question is: Is it possible to allocate Arc<[MaybeUninit<u8>]> with bigger (not u8's) alignment using only one alloc? And if no, what kind of API should we add to std to allow this?

#[repr(C, align(...))]
struct AlignedBytes([u8; LEN]);
// or simply
struct YourStruct { ... }

// and then:
Arc::<AlignedBytes>::new_uninit()
// or
Arc::<YourStruct>::new_uninit()
2 Likes

No, in fact even making a Box in that manner is UB as dealloc must be given the exact alignment it was given when allocating the memory.

However as @Yandros shows, it is possible to create a Box/Arc with MaybeUninit<AlignedBytes> as item type. But it is unsound to transmute that to a box or arc with MaybeUninit<[u8]> as value, since that type has a different alignment.

5 Likes

No that's not possible by definition. Rust's allocator API requires that when you deallocate, you have to pass same size/alignment value to gave when allocate. Otherwise it's UB, means you wrote a wrong program.

When the Arc<[MaybeUninit<u8>]> 's refcount got zero, its backing storage will be deallocated with the alignment of the ArcInner<[MaybeUninit<u8>]>. You can't change its default deallocation layout.

Why don't you just have Arc<[MyCustomAlignmentType]> and .align_to() it into &[u8]? Keep it in mind that in Rust, Sized types can only have its size as a multiple of its alignment.

@Yandrs the thing is — I'm trying to do this in a generic way (i.e.: allocate those bytes with align max(align_of::<A>(), align_of::<B>())), and generics are not allowed in array lengths.

Oh. I've entirely missed this, thanks!

I've probably should have said it in the first place but anyway... I am trying to make a slice-like structure that would store 2 types in a contiguous chunk of memory like this:

struct BiTySlice<L, R> {
    _alignl: [L; 0],
    _alignr: [R; 0],
    // Some len-related fields are omitted (?)
    raw: [u8],
}

So the goal is to allocate this actually :sweat_smile: I know there would be a lot of troubles aligning both L and R in the raw slice and I'll probably won't implement it safe in the first try, but still I want to experiment with this.

The issue is, will you be able to implement it without UB? Use of unsafe is not a problem, at least for experts, but if the code is UB then you have transitioned into the land of random compiler output and you don't have a usable solution.

By "safe" I've meant "without UB" :slightly_smiling_face:

I have some experience with writing and debugging unsafe code, so hopefully sooner or later I might write it properly.

Using Box::from_raw like this is not allowed by the API. If you check the documentation for Box::from_raw it says the following.

For this to be safe, the memory must have been allocated in accordance with the memory layout used by Box .

What does allocation in accordance with the memory layout mean?

It is valid to convert both ways between a Box and a raw pointer allocated with the Global allocator, given that the Layout used with the allocator is correct for the type.

Alignment is part of a layout, so if it gets changed the layouts are incompatible and UB occurs. This may be unnoticed with standard allocator (malloc) as standard allocator ignores alignment arguments, but it will be noticed by tools like Miri. For instance, consider the following program.

use std::alloc::{alloc, Layout};

fn main() {
    unsafe {
        let ptr = alloc(Layout::from_size_align(1, 4).unwrap());
        *ptr = 42;
        Box::from_raw(ptr);
    }
}

This happens to work because by default Rust uses malloc and free functions from C standard library that don't have an alignment parameter. However, alternate allocators can take alignment parameter and a mismatch here would cause UB. For instance, when running this program in Miri the following error occurs.

error: Undefined Behavior: incorrect layout on deallocation: allocation has size 1 and alignment 4, but gave size 1 and alignment 1
  --> src/main.rs:7:27
   |
7  |         Box::from_raw(ptr);
   |                           ^

So what could be done about that? Well, we can wrap our unsized array into an alignment structure. To do so, we need to allocate memory using raw APIs, create a reference to unsized [T] by using slice::from_raw_parts_mut, cast the reference to our wrapped type and then create a box. When a box gets deallocated the alignments will match.

use std::alloc::{alloc, handle_alloc_error, Layout};
use std::mem::{self, MaybeUninit};
use std::ops::{Deref, DerefMut};
use std::ptr::NonNull;
use std::slice;

#[repr(C, align(4))]
struct Align4<T: ?Sized>(T);

impl<T: ?Sized> Deref for Align4<T> {
    type Target = T;
    fn deref(&self) -> &T {
        &self.0
    }
}

impl<T: ?Sized> DerefMut for Align4<T> {
    fn deref_mut(&mut self) -> &mut T {
        &mut self.0
    }
}

fn allocate_align4_slice<T>(len: usize) -> Box<Align4<[MaybeUninit<T>]>> {
    let bytes = mem::size_of::<T>().checked_mul(len).expect("overflow");
    unsafe {
        let allocation = if bytes == 0 {
            NonNull::<Align4<T>>::dangling().as_ptr() as *mut u8
        } else {
            let layout = Layout::from_size_align(bytes, mem::align_of::<Align4<T>>()).unwrap();
            let allocation = alloc(layout);
            if allocation.is_null() {
                handle_alloc_error(layout);
            }
            allocation
        };
        let slice = slice::from_raw_parts_mut(allocation as *mut MaybeUninit<T>, len);
        Box::from_raw(slice as *mut [MaybeUninit<T>] as *mut Align4<[MaybeUninit<T>]>)
    }
}

fn main() {
    let mut boxed = allocate_align4_slice(42);
    unsafe { *boxed[0].as_mut_ptr() = 88 }
}

But that helps with Box only, what about Arc? Well, as far I can tell there is no way to avoid an additional allocation. If an allocation is fine, Arc::from(boxed) should work fine.

If you don't have an alignment requirement, it's possible to use collect() to create Arc<[MaybeUninit<u8>]>. This is done by writing iter::repeat(|| MaybeUninit::uninit()).take(len).collect::<Arc<[MaybeUninit<u8>]>>().

However, this won't help when the data needs to be aligned in a particular way as there is no API to allocate memory with a provided Layout in a way that Arc::from_raw expects it to be allocated (RFC candidate maybe?). The API that would allow this could look like fn alloc_raw(layout: Layout) -> *mut u8.

1 Like

@xfix thanks for the complete explanation!

I thought of writing and RFC but wanted to double-check that it's not already possible. Though, I've thought of API like this:

impl<T: ?Sized> Arc<T>
    unsafe fn alloc_for_layout(layout: Layout) -> Arc<MaybeUninit<T>> { ... }
}

Which clearly won't work because there is no len/vtable parameter for !Sized types. Thanks again for the idea of -> *mut u8, this could work. I'll try to write (pre)-RFC soon. :slightly_smiling_face:

Though I'm curious why using *mut u8 and not simply *mut ()?

1 Like

This matches std::alloc::alloc. The exact type doesn't exactly matter however.

1 Like