Why is transmute_copy giving garbage?

1

I'm reading binary data from some file, and I can see that the data read is correct when I print it, but when I transmute it, it becomes garbage. What am I doing wrong?

fn main() {
    let v: Vec<u8> = vec![61,0,0,0,0,0,0,0];
    let data: &[u8] = v.as_slice();
    let x = unsafe { std::mem::transmute_copy::<&[u8], i64>(&data) } as i64;
    println!("{:x?}", data);
    println!("{:x?}", x);
    println!("{}", x);
    assert_eq!(x, 61);
}

The output is of this program is:

[3d, 0, 0, 0]
5578bd6f7a40
93977062636096

Here's the playground link.

2

You are transmuting a 4-byte vector and whatever follows in memory into an 8-byte integer. You are also neglecting to take into account whether your machine memory ordering is BigEndian at some granularity, or LittleEndian.

2 Likes
3

I'm sorry, that's my mistake in this example. In the real code it's 8 bytes. Also don't worry about endianness, it's correct. It's little endian, or am I mistaken?

I'll fix the code up there.

4

You are transmuting the pointer not the value. And since you are modifying the length, I don't think a transmute understands it.
This code has an alignment issue.

2 Likes
5

You are transmuting a reference to an integer, not the contents of the slice. You should probably use pointer casts instead of transmute_copy

7

Thanks a lot! This works fine :slight_smile:

8

You should be aware of alignment or lack thereof. Skipping Endian is skipping cross platform support.

let x = unsafe { std::ptr::read_unaligned(data.as_ptr() as *const i64).to_le() };
2 Likes
9

Thanks for the answer. Apart from endianness, are you saying that the answer from @leudz is wrong or can produce invalid results? I took his answer and I'm currently using it.

Why would rust make any assumptions about alignment if I'm providing a raw array of bytes?

10

As long as it is treated as an array of bytes, you're fine. It is when you start reading multi-byted data from it through pointer-casting that this data must be suitably aligned in RAM.

IIRC, the reason is that there is hardware in the wild where unaligned memory accesses must go through either special instructions or a special codegen path that are much slower than those for aligned reads. By requiring aligned pointers, Rust makes sure that binaries can use faster aligned access instructions on those architectures.

1 Like
11

Specifically in the code, there is a from_raw_parts call that creates a &[i64] from a potentially misaligned *const i64 pointer (created by casting the *const u8 pointer).

1 Like
12

@jonh is right alignment is an issue, I'm sorry.
For the endianness though isn't from_le the right method?

13

You're right. There are instructions that only work for aligned data. I worked with emulators before and had to write these. But usually the solution is to check if, e.g., data & 3 != 0 for 4 bytes alignment... not to assume that everything is aligned. That'd be a very expensive assumption. Rust checks boundaries of arrays and vectors for every access by default, which is a little performance hit. By comparison, it's definitely not a performance hit to do that operation before deciding whether an array is aligned.

14

Both your solutions gave correct results, but your solution is now in the code. I'm not sure I should prefer one over the other or not. Maybe out of paranoia I should just use his solution. Thanks to you both :slight_smile:

15

Never mind they do the same thing (why are there two of them?) so I'd use @jonh's it's simpler. This isn't my night =)

16

I would favor from_le_bytes without any use of unsafe. (Now I see it.) Then only change if profiling flags it as bad.
@leudz Both do have a code smell by having the same type. As you say from_le is likely the more appropriate but associate functions typically get disfavored when member function exits.

1 Like
17

Btw, I just discovered that the line you wrote doesn't work for f32 and f64. I'm getting the error:

error[E0599]: no method named `to_le` found for type `f32` in the current scope

Changing f32 to i64 fixes the problem. So it's a type problem, definitely. Do you know how to do this for them?

18

to_le doesn't exist on f32 or f64 (there is from_le_bytes on nightly).
What you can do is use the method above to get a u32/u64 and then use from_bits/from_bits.

19

Well, one optimizing compiler's nightmare is to lose microbenchmarks wrt hand-written assembly just because the source language semantics do not allow a well-known optimization to be carried out, and a branch for checking if a pointer is aligned could get expensive if it ended up at the bottom of the wrong loop...

20

Thanks again @leudz. That looks like the most reasonable solution.

21

I'm not that worried about that kind of optimization... there's I/O in that program, which will be waaaaaaaay slower than any micro-optimizations I do in my code.