How can I reinterpret [u8] to [u64]?

help
21

HI, how about:

#[inline]
fn unaligned_u8_to_u64(array: &[u8]) -> &[u64] {
    let len = array.len() >> 3;
    let ptr = array.as_ptr() as *const u64;
    unsafe {
        std::slice::from_raw_parts(ptr, len)
    }
}

#[cfg(test)]
mod tests {
    #[test]
    fn foobar() {
        let s = "abcdABCDefghEFGHijklIJKL1234567";
        for v in super::unaligned_u8_to_u64(s.as_bytes()) {
            // NOTE: you have to handle endianness by yourself
            println!("{:#018x}", v);
        }
    }

The endian in the for loop is unspecified, I don't know if it's a UB.

22

if you know that stuff is aligned properly or your platform allows not to care about alignment:

fn cast(input: &[u8]) -> &[u64] {
    let ptr = input.as_ptr();
    let input_len = input.len() / core::mem::size_of::<u64>();
    assert!(input.len() % core::mem::size_of::<u64>() == 0, "input.len() cannot be divided by 8");
    
    unsafe {
        core::slice::from_raw_parts(ptr as *const u64, input_len)
    }
}

Otherwise read u64 from ptr using ptr::read_unaligned

Something along these lines:

fn virgin_cast(input: &[u8], output: &mut [u64]) {
    let ptr = input.as_ptr();
    let output_len = input.len() / core::mem::size_of::<u64>();
    assert!(input.len() % core::mem::size_of::<u64>() == 0, "input.len() cannot be divided by 8");
    assert!(output.len() >= output_len, "output.len() needs at '{}'", output_len);
    
    for idx in 0..output_len {
        unsafe {
            output[idx] = core::ptr::read_unaligned(ptr.add(idx * core::mem::size_of::<u64>()) as *const u64);
        }
    }
}

P.s obviously replace asserts with proper result code if you don't trust input size

23

So, the code in How can I reinterpret [u8] to [u64]? - #21 by foobar is error-prone if the the input &[u8] is not properly aligned?

24

It depends on where input comes from.
If &[u8] is really sequence of u8 then probably it is not aligned (it is not an issue on modern HW)
But if you get &[u8] by casting it from let's say &[u64] then it would be aligned and you can just reinterpret &[u8] as &[u64]

TL;DR it is theoretically UB if you cannot guarantee that input &[u8] is properly aligned (which some interpret as error prone, but it depends on target architecture)

25

Very thoughtful, but I doubt about the performance.

26

That's considered instant UB if the reference is unaligned, regardless of the platform it's running on. As @alice mentioned, this is a property of the compiler, not the platform. Also, not accounting for endianness is perfectly legal (in fact, from_ne_bytes is implemented as a transmute); it's just not usually done due to being error-prone. I'd suggest using one of the read_unaligned() or UnalignedU64 solutions above.

27

Do a benchmark. You shouldn't do any unsafe shenanigans unless the benchmark shows a significant improvement in performance, and you actually need that improvement in your use case.

28

Rust requires that all references are properly aligned at all times, so the loop is UB.

29

You can probably also spell this as

#[derive(Copy, Clone)]
#[repr(packed)]
struct UnalignedU64 {
    value: u64,
}
30

I wonder if that's even legal in Go? https://go.dev/ref/spec#Address_operators is unclear whether it does an aligned or unaligned read, though https://go.dev/ref/spec#Size_and_alignment_guarantees does say that uint64 has align 8.

Maybe it just ignores the problem and only runs on architectures where the ordinary load/store instructions don't care?

31

When unaligned_references finally becomes an error, repr(packed) fields will become somewhat magical: they can be read and written as place expressions, but not referenced. I'd prefer the former spelling for that reason.

32

Wouldn’t you actually need something like #[repr(C, packed)] in order to be certain about what the layout will be?

33

I don't know what guarantees it gives, but using #[repr(packed)] for this seems incredibly sketchy to me.

34

Well, per the FCP in Clarify guarantees provided by repr(packed) by joshlf ¡ Pull Request #1163 ¡ rust-lang/reference ¡ GitHub, it guarantees no inter-field padding.

Which, I suppose, doesn't technically say anything about before-the-fields or after-the-fields padding, but personally I'd consider that a spec bug. I'll happily move to fcp merge any PR someone wants to send to clarify that fact.

I think the array version is best for LittleEndianU64 or BigEndianU64, but for Copy types in native endian, I think Packed<u64> is an elegant way -- though the array way is certainly ok too.

35

Ah, I see; that change hasn’t even made it to doc.rust-lang.org/nightly yet :sweat_smile:

I suppose you’re saying that repr(C) is going to be redundant for single-field packed structs, (at least as struct size and alignment as well as field offsets and paddings are concerned).

I see a potential conflict or possibly unintended consequence when interpreting this property together with the sentence

For packed , if the specified alignment is greater than the type's alignment without the packed modifier, then the alignment and layout is unaffected.

If I have a default representation struct that, on its own, turns out to have alingment N, then repr(packed(N)) on the same struct would (at least if you interpret the reference this way) be both:

  • not affecting layout
  • be guaranteed to have minimum required padding between fields for each field’s alignment

so indirectly, these two properties give (AFAICT new) guarantees about the ordinary default representation layout.

Edit: I also don’t fully understand what this “Inter-field padding is guaranteed to be the minimum required in order to satisfy each field's (possibly altered) alignment” is saying: either it’s talking about the total field padding i.e. that the sum of all paddings is as small as possible (but without changing field order for this minimization), or it’s saying that for each field the padding before that field is as small as necessary to satisfy the field’s (altered) alignment.

The second interpretation would pretty-much give the compiler the option to choose a field order and then guarantee a repr(C, packed(N))-style layout for that chosen field order, while the first one would allow the compiler e.g. to lay out #[repr(packed(2)) struct S(u32, u8, u32) as [first u32] [1 byte padding] [the u8] [second u32] instead of [first u32] [the u8] [1 byte padding] [second u32], putting additional padding before the u8 which will be saved after the field keeping the total padding the same.

36

Seems clearer for single field types to use #[align(1)].

37

Remember that align can only increase alignment, not lower it:

#[repr(align(1))]
struct Foo(u64);
dbg!(std::mem::align_of::<Foo>()); // Still 8

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=cd1f7128fc9979ef59d17b669e62ae7f

38

Ah, read that section where it says both align and packed specifies alignment, missed the next where it specifies that align can only increase and packed can only decrease. Seems a bit lame, though I guess decreasing alignment is unsafe.

39

I think the earlier suggestion in this thread of using core::slice::align_to is in the right direction. It automates the usual drudgework of dealing with leading/trailing misalignment that you otherwise need to painstakingly handle when writing code like that by hand. Here's an example I wrote up:

If you want to avoid unsafe code, you can use bytemuck::pod_align_to. Unfortunately the only version of align_to in the standard library is this unsafe version, so if you want to use it directly you must take responsibility to ensure your use case is transmute safe.

40 
#[derive(Copy, Clone)]
#[repr(transparent)]
struct UnalignedU64 {
    value: [u8; 8],
}

Rust requires that all references are properly aligned at all times, so the loop is UB.

Hi, it makes me wonder why your -> &[UnalignedU64] can be worked as expected without any UB?