Println! never executed after executing unsafe fn?

I've been trying to port an algorithm to Rust which requires some unsafe code, but even when I fix all the warnings, I get no output despite the presence of println!. This doesn't just happen when printing vars that were used in the unsafe code, but even when I try printing a literal like '5' or "hello".

unsafe fn crypt(mut key: u32, data: &[u8]) -> u32
{
    let l = data.len() >> 2;
    let p = data.as_ptr() as *mut u8;
    let data32 = p as *mut u32;
    for i in 0..l
    {
        key = (key << 1) + 0x4878;
        *data32.offset(i as isize) ^= key.to_le();
    }
    
    key
}

fn main() {
    let b=  "hello world lol haha".as_bytes();
    let mut k: u32;
    unsafe {
        k = *(b.as_ptr() as *mut u32);
    }
    println!("{}", k); // works fine
    unsafe {
        k = crypt(k, b); // <- all println!s after this do not execute
    }
    println!("{}", k);
    println!("{}", 5); // even a literal that has nothing to do with crypt() will not print
}

You are seeing a segmentation fault. Running the code in gdb gives more information:

Program received signal SIGSEGV, Segmentation fault.
0x000055555555ba31 in laphicet::crypt (key=1819043176, data=...)
    at src/main.rs:9
9	        *data32.offset(i as isize) ^= key.to_le();

The data behind string literals is not considered mutable. Taking a &'static str literal and casting it to *mut u32 still doesn't let you write to it.

Maybe copy the data to the heap using "hello world lol haha".to_owned() before writing to it.

1 Like

To add to that, crypt is triggering undefined behavior by mutating raw data behind an immutable/shared reference data. The method signature should have been

unsafe fn crypt(mut key: u32, data: &mut [u8]) -> u32

in which case the compiler would have caught the original mistake pointed out by @dtolnay.

1 Like

Good point. I was wondering why the segfault, but did not cross my mind that indeed string literals are not mutable. In fact from here going in C99 N1256 draft 6.7.8/32 "Initialization", Example 8:

The declaration
char s[] = "abc", t[3] = "abc";
defines  ‘‘plain’’ char array  objects `s` and `t` whose  elements  are  initialized  with  character  string  literals.
This declaration is identical to 
char s[] = { 'a', 'b', 'c', '\0' },
t[] = { 'a', 'b', 'c' };
The contents of the arrays are modifiable.
On the other hand, the declaration
char *p = "abc";
defines `p` with  type  ‘‘pointer  to char’’  and  initializes  it  to  point  to  an  object  with  type  ‘‘array  of char’’ with length 4 whose elements are initialized with a character string literal.
If an attempt is made to use `p` to modify the contents of the array, the behavior is undefined.
2 Likes

Good catch! I totally overlooked that. With that and a few tweaks, code works as expected. Thanks! I was using the website sandbox compiler which apparently doesn't print out segfaults.

FWIW here is how I would write the same code. This is 100% safe Rust, significantly more concise, in my opinion more readable, and with exactly the same performance.

extern crate pod;
use pod::Pod;

fn crypt(mut key: u32, data: &mut [u32]) -> u32 {
    for i in data {
        key = (key << 1) + 0x4878;
        *i ^= key.to_le();
    }
    key
}

fn main() {
    let mut s = b"hello world lol haha".to_owned();
    // Assume mutiple of 4 bytes, panic otherwise.
    let b = Pod::map_slice_mut(&mut s).unwrap();
    // Assume at least 4 bytes long, panic otherwise.
    let mut k = b[0];
    println!("{}", k);
    k = crypt(k, b);
    println!("{}", k);
}
4 Likes

That map_slice_mut fn seems vague. Doc says it maps one type to another, but where is it specified it turns a 1 byte array into a 4 byte array?

The magic of type inference. map_slice_mut returns Option<&mut [T]>, and .unwrap() turns it to &mut [T]. But what is T? Well, b is later passed to crypt, which takes an &mut [u32], so the compiler concludes T is u32.

1 Like