Tips and tricks for writing unsafe or FFI code?

Michael-F-Bryan · May 9, 2020, 8:16am

I was thinking of putting together a couple articles on more advanced Rust topics like how to write unsafe or how to do FFI correctly.

What useful tips or tricks have you come across in this area?

One of my favourites is where you can "split" a closure into a pointer to its data and an extern "C" function which can be passed across the FFI boundary to invoke the closure at a later point. I added a generic version of this to my ffi_helpers crate a while back.

(playground)

Yandros · May 9, 2020, 3:04pm

Yep, that's a very important trick to have up one's sleeve

The &'_ mut (impl FnMut(...) -> _) version is nice although a bit dangerous / limited lifetime and exclusivity wise.

That's why I'd suggest to also have a version for Arc<impl 'static + Send + Sync + Fn(...) -> _> , that splits it into three elements (add a free function):

env: *mut c_void,
call: unsafe extern "C" fn (env: *mut c_void, ...) -> _,
free: unsafe extern "C" fn (env: *mut c_void),

you could also even add a 4th pointer: a retain function pointer.

which allows concurrent (Fn), parallel (Sync), arbitrarily long-lived (Arc<impl 'static>) calls of the closure.

if concurrent and parallel is not required, you can also have the Box<impl 'static + Send + FnMut> version.

Michael-F-Bryan · May 9, 2020, 3:24pm

I feel like once you start getting more complex and need to think about destructors, you may as well treat the closure like an proper object and use some sort of vtable (aka poor man's classes - also a nice tool to have).

struct Callback {
  user_data: *mut c_void,
  call: unsafe extern "C" fn(..., user_data: *mut c_void) -> ...,
  free: unsafe extern "C" fn(user_data: *mut c_void),
  clone: unsafe extern "C" fn(user_data: *mut c_void) -> Callback,
  ...
}

That way you can define a bunch of factory functions which will box the closure using the appropriate smart pointer (Box, Rc, Arc, etc.).

Have you ever seen APIs where this level of flexibility would be useful?

matklad · May 9, 2020, 4:20pm

The knowledge that pointer casts are quite versatile and can subsume transmute in a lot of cases (for example, for lifetime laundering).

If you do need transmute, fully specify both types via turbofish.

The „don’t call user supplied closure in unsafe block“ rule of thumb (this causes problems like reentrancy bug in once_cell or double free in naive take_mut).

Using NonNull at rest for proper variance.

“Safe equivalence” principle: if you can implement slow, but safe version of the function, it’s interface is not broken.

*const T and *mut T are actually almost the same, what matters is pointer provenance.

Yandros · May 9, 2020, 7:02pm

I'd imagine something like an on_accept(cb) API, for some listener that may be running for the lifetime of the program in some background thread(s). Granted, I don't think C has the habit of requiring both a "'static bound" and some form of destructor / freeing function at the same time: it is true that in C the callbacks are very often borrowed, and since there is usually some kind of master instance (e.g., a listener "object"), these borrowed callbacks are "just" required to outlive such instance.

parasyte · May 18, 2020, 2:15am

I was working with nom to write an assembler, and found that some of the parsers return a tuple of my output type &str. This made the code either very obnoxious to work with, or required cloning each string slice into an owned String to concatenate the tuple of string slices.

So I wrote this abomination to transform a pair of string slices into a single string slice:

#![deny(clippy::all)]
#![deny(clippy::pedantic)]

/// Merge two string slices into one.
///
/// # Panics
///
/// This function will panic if the `start` and `end` string slices are not in contiguous memory.
pub fn merge_str<'a>(start: &'a str, end: &'a str) -> &'a str {
    // Safety:
    // We are guaranteeing that the string slices are in contiguous memory, and that the resulting
    // string slice will contain valid UTF-8.
    unsafe {
        // Ensure string slices are in contiguous memory
        if start.as_ptr().add(start.len()) != end.as_ptr() {
            panic!("String slices must be in contiguous memory");
        }

        // Convert the two string slices into a single byte slice
        let s = std::slice::from_raw_parts(start.as_ptr(), start.len() + end.len());

        // Convert the byte slice into a string slice
        std::str::from_utf8_unchecked(s)
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::panic::catch_unwind;

    #[test]
    fn test_merge_str() {
        let s = "Hello, world!";
        let t = "Hello, world.";

        assert_eq!(merge_str(&s[..4], &s[4..]), s);
        assert_eq!(merge_str(&s[..0], &s[..0]), &s[..0]);
        assert_eq!(merge_str(&s[..0], &s[..1]), &s[..1]);
        assert_eq!(merge_str(&s[..1], &s[1..1]), &s[..1]);

        // This is a weird edge case which should fail but does not, due to Rust's memory layout.
        // assert!(catch_unwind(|| merge_str(&s, &t)).is_err());
        assert!(catch_unwind(|| merge_str(&t, &s)).is_err());

        assert!(catch_unwind(|| merge_str(&s, &s)).is_err());
        assert!(catch_unwind(|| merge_str(&s[..4], &s[5..])).is_err());
        assert!(catch_unwind(|| merge_str(&s[..5], &s[4..])).is_err());
        assert!(catch_unwind(|| merge_str(&s[..4], &t[4..])).is_err());
    }
}

This function maintains zero-copy parsing semantics, and allows writing parsers like this:

#[derive(Debug, PartialEq)]
pub enum Inst<'a> {
    GlobalLabel(&'a str),
}

fn is_word_start(input: char) -> bool {
    input.is_ascii_alphabetic() || input == '_'
}

fn is_word(input: char) -> bool {
    input.is_ascii_alphanumeric() || input == '_'
}

/// Recognize global labels.
fn global_label(input: &str) -> IResult<&str, Inst> {
    terminated(
        pair(take_while_m_n(1, 1, is_word_start), take_while(is_word)),
        tag(":"),
    )(input)
    .map(|(rest, (start, end))| (rest, Inst::GlobalLabel(merge_str(start, end))))
}

parasyte · May 18, 2020, 9:40am

And I just figured out that this hack is not even necessary because nom::combinator::recognize does exactly the same thing!

/// Recognize global labels.
fn global_label(input: &str) -> IResult<&str, Inst> {
    terminated(
        recognize(pair(
            take_while_m_n(1, 1, is_word_start),
            take_while(is_word),
        )),
        tag(":"),
    )(input)
    .map(|(rest, label)| (rest, Inst::GlobalLabel(label)))
}

But it was still an interesting exercise, and might be useful to someone else for other reasons...

Yandros · May 19, 2020, 10:43am

parasyte:

/// Merge two string slices into one.
///
/// # Panics
///
/// This function will panic if the `start` and `end` string slices are not in contiguous memory.
pub fn merge_str<'a>(start: &'a str, end: &'a str) -> &'a str {
    // Safety:
    // We are guaranteeing that the string slices are in contiguous memory, and that the resulting
    // string slice will contain valid UTF-8.
    unsafe {
        // Ensure string slices are in contiguous memory
        if start.as_ptr().add(start.len()) != end.as_ptr() {
            panic!("String slices must be in contiguous memory");
        }

        // Convert the two string slices into a single byte slice
        let s = std::slice::from_raw_parts(start.as_ptr(), start.len() + end.len());

        // Convert the byte slice into a string slice
        std::str::from_utf8_unchecked(s)
    }
}

FWIW, the above code is unsound: counter-example (Run with MIRI to spot UB)

Indeed, slice::from_raw_parts requires that the slice spans across a single allocation, which in your case your function cannot check for that. You'd need to take a "witness of single-allocation-ness" (the original string being fed to nom), which would have to contain both start and end, or make the function unsafe and add a # Safety clause against this case.

You can find more info in this thread: Pre-RFC: Add join_seq method to slices and strs - libs - Rust Internals, which you have reminded me of: I will submit the documentation enhancement PR soon

parasyte · May 19, 2020, 6:56pm

That's exactly what I needed! It fixes the commented "weird edge case" in the test.

And thank you for the pre-RFC link. I was looking for that but didn't know how to word it.

system · August 17, 2020, 6:56pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Is there an established technique for FFI callbacks? help	4	1212	January 24, 2021
Using Unsafe for Fun and Profit tutorials	13	2447	January 12, 2023
FFI w/ callbacks: am I doing something wrong? compiler bug? help	11	929	January 12, 2023
Bounding lifetimes of raw pointers from FFI help	2	1049	August 7, 2022
Callbacks and Function Pointers across the FFI boundary [Feedback Wanted]	6	1594	January 12, 2023

Tips and tricks for writing unsafe or FFI code?

Related topics