I want to work with a bytearray as a list of native-endian u128. I have a function:
fn foo(data: &mut [u8]){
for b in data{
*b = some_u8_funct();
}
}
Inside that function I want to iterate every byte of data with 'steps' of 16 bytes, modifying them. I want to do it without conversion or any additional cost. (I can do invariant check that data.len() % 16 == 0).
In C I can just cast char* to long long long* (or what C have currently for 128 bits), but it sounds wrong on many level in Rust. What is the best/idiomatic way to do this?
Basically, I want something like that:
fn foo(data: &mut [u8]){
if data.len() % 16 !=0 {
painc!("Bad aligment");
}
for my_u128 in data.some_magic_function(){
*my_u128 = some_u128_funct();
}
}
I've tried to use for b in data as &mut [u128]{}, but Rust said it's not a primitive type for 'as' conversion.
This may or may not be possible since the alignment of the data reference must be large enough. Note that alignment is different from the length.
In the cases where the alignment is large enough for the cast to be valid, the bytemuck crate can do it with bytemuck::cast_slice. However this will fail if the address of the first byte in the array is not divisible by 8.
If you do have alignment issues that make bytemuck not possible to use. The next best thing is to use byteorder. This is would have a bit of overhead if the data isn’t aligned right but I think it’s unavoidable in that case.
fn foo(data: &mut [u8]) {
let u128_slice: &mut [u128] = bytemuck::cast_slice_mut(data);
for b in u128_slice{
*b = 0x112233445566778899AABBCCDDEEFF;
}
}
fn main(){
let mut data:Vec<u8> = vec![0;32];
foo(& mut data);
for b in data {
println!("{}", b);
}
}
I don't believe that the Vec's allocation is guaranteed to be properly aligned here. You can see the error by passing a different slice from the vector:
fn foo(data: &mut [u8]) {
let u128_slice: &mut [u128] = bytemuck::cast_slice_mut(data);
for b in u128_slice{
*b = 0x112233445566778899AABBCCDDEEFF;
}
}
fn main(){
let mut data:Vec<u8> = vec![0;35];
foo(& mut data[1..]);
for b in data {
println!("{}", b);
}
}
Compiling playground v0.0.1 (/playground)
Finished dev [unoptimized + debuginfo] target(s) in 1.15s
Running `target/debug/playground`
thread 'main' panicked at 'cast_slice_mut>TargetAlignmentGreaterAndInputNotAligned', /playground/.cargo/registry/src/github.com-1ecc6299db9ec823/bytemuck-1.5.1/src/lib.rs:106:3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
It's actually looks like a platform-independent overkill. As far as I understand, Intel allows unaligned read/writes (but hurts your performance for doing that).
I sometimes wonder if a memory-page aligned variant of Vec would be useful. It might be able to help with efficient manipulation of large datasets. I've never run into issues along those lines, though, so I haven't looked into whether it would actually help.
If you know which platform(s) and allocator(s) your code will be running on, you can rely on properties of the allocator. For example, on platforms where the default allocator uses malloc, heap allocations are guaranteed to be suitably aligned for any built-in C type.
(However, if you are writing a library for use by third parties, or a binary that uses the system allocator and may run on arbitrary platforms, then you can't assume such things.)
pub fn foo(data: &mut [u8]) {
for b in data.chunks_exact_mut(std::mem::size_of::<u128>()) {
b.copy_from_slice(&0x112233445566778899AABBCCDDEEFFu128.to_ne_bytes())
}
}
It doesn't look like it generates more assembly (I switched the panic to std::process::exit to reduce noise in the generated assembly of the bytemuck version since that shouldn't affect the happy path) Compiler Explorer