Nullable Pointers in FFI


#1

TLDR: Option<*{const,mut} type> should take advantage of null-ptr optimization and use the memory layout of *{const,mut} type with the local NULL value as the None variant and all others as the Some variant.

Two of Rust’s big draws, for me, are its excellent memory safety features both in the compiler mechanisms and in the language syntax, and its excellent FFI capabilities. I am hoping to use both of those as selling points for adoption of Rust at work.

We write mostly in C/C++, and are well acquainted with using C-API interfaces to move across library and language boundaries, so Rust could conceivably drop in among C without causing a splash.

One of the things I am hoping to use as a demonstration of Rust’s utility is moving null-pointer checks out of run-time, numeric logic:

unsigned char* ptr;
if (ptr == NULL) { return -1; }

and into compile-time, abstract logic:

let ptr: Option<*const u8>;
match ptr {
    Some(p) => do_thing(p: *const u8),
    None => return -1;
}

The Rust book proudly proclaims that in Rust internals, Option-wrapped pointers are in fact just pointers that use NULL value as their representation of None. However, this very clearly does not hold true at the FFI boundary.

The only threads here I can find close to this topic are below. I’ve also read a GitHub issue about Optional references, with which I somewhat agree, as references mean things in Rust that raw pointers do not.

Would it be possible to optimize the Optional raw-pointer case for FFI to match the Optional pointer case within Rust so that FFI-pointer validity checks can be made part of the type system? We know Rust cannot semantically guarantee anything about the meaning behind pointers arriving from FFI, and so I agree that mapping straight to references is not a great idea and should be done separately:

let ptr: Option<*const u8>;
if let Some(p) = ptr {
    let r: &u8 = unsafe { &*p };
}

but being able to specify in the type signatures of functions that they accept potentially-null pointers, and enforce validation of those pointers at compile time, would, I believe, be tremendously useful for mixing in Rust libraries with C callers.

If anyone has a good reason we shouldn’t do this, I’d be very interested to hear it. I recognize that Rust is unable to guarantee validity of FFI pointers as references (aimed at safe chunks of memory), but I do not see why Rust can’t provide zero-cost abstraction of non-NULL validity (aimed at memory that will maybe, but not definitely, segfault) as part of its type system.


#2

I don’t think this should happen by default, at least, as Some(ptr::null()) is a perfectly reasonable value, in a way that doesn’t apply for Option<&T>.

A lot of the internal types use NonZero pointers, but that’s not stable, nor is it #[repr(C)] for FFI (but neither is Option).

If you’re consuming these FFI pointers, perhaps you just want the pointer as_ref() -> Option<&T>?


#3

The repr of Some(ptr::null()) requires a word of discriminant as well as the null-pointer; if Option<*ptr> were implemented with the null-ptr optimization, this would be layout- and semantics- equivalent to Some(None): Option<Option<*ptr>>: the interior Option<*ptr> is just a pointer whose None variant is ptr::null(), and the exterior Some(ptr::null()) is now identical to Some(Option::<*ptr>::None).

I’ll give ptr::as_ref() a whirl though, thanks!


#4

I understand the difference it makes for discriminants. I was trying to say that None and Some(ptr::null()) both have valid and distinct meanings. If I had a function returning Option<*const T>, then None means “I have no pointer to return,” and Some(ptr::null()) means “here is my pointer, it’s NULL.” It’s like the difference between “unknown” and “known to be NULL”.


#5

IMO, that should definitely be typed as Option<Option<*ptr>>; “I don’t even know” is None (unreachable payload), “I know what it is and it’s NULL” is Some(None) (reachable, non-dereferenceable payload), “I know what it is and it’s non-NULL” is Some(Some(addr)) (reachable, dereferenceable payload).

While match raw_ptr.as_ref() { Some(r) => { valid(); }, None => { invalid(); }, } is an excellent use of the type system to enforce null-checking, I’m still of the opinion that being able to declare potential nullability in the function signature via Option<*ptr> would be a useful pattern.

With as_ref():

pub extern fn recv_maybe<T>(ptr: *const T) {
  match ptr.as_ref() {
    Some(r) => { // r is &T
      foo(r);
    },
    None => { handle_null(); },
  }
}
pub extern fn recv_firm<T>(ptr: *const T) {
  foo(unsafe { &*ptr });
}

Since the function signature doesn’t explicitly declare nullability, this reduces the ability to distinguish between functions that expect they may receive null, and functions that expect to only ever be given valid pointers. Semantically, a function that refuses to trust its input and a function that would like to but can’t, should look different; as is, this is boilerplate that must go on every FFI function receiving a foreign pointer, even if the incoming pointer is one that the Rust code had previously given out and is known in the abstract design to be valid.

This also means that browsing the signatures alone is insufficient to determine which functions can expect null, and which do not expect that.

With Option<*ptr>:

pub extern fn recv_maybe<T>(ptr: Option<*const T>) {
  match ptr {
    Some(r) => { // r is coerced to &T
      foo(r);
    },
    None => { handle_null(); },
  }
}
pub extern fn recv_firm<T>(ptr: *const T) {
  foo(unsafe { &*ptr });
}

The recv_maybe function is used when the caller might give it anything; the recv_firm function establishes as part of its contract that incoming pointers must be non-null and valid, and that the caller’s breaking of this contract is a severe logic error. recv_maybe expects that null might happen and has a plan to deal with it; recv_firm has no reason to expect null and has no game plan for handling such a case other than panic.

This makes explicit as part of the signature (like the ?: decoration in TypeScript and, I think, Swift) that some functions are capable of receiving and handling NULL and that some are not, and reduces the amount of boilerplate code that is necessary to handle null checking on functions that should never need it. Furthermore, I’m of the opinion based solely on 1 point of anecdata that seeing the Option in the signature serves to remind readers of the need to check for validity, whereas the lack of Option implies that, since Rust doesn’t have null as a valid member of type sets, the pointer can be reasonably assumed to be not null. Deref is still, obviously, unsafe, but doesn’t require the full checking machinery of Option.


In my personal example, I have a Rust library which can receive maybe-null pointers from FFI callers, and react accordingly. This function then hands back pointers into Rust memory which are known good; sibling functions in the library receive that pointer back from FFI later, and must still go through the motions of checking validity when, as long as the caller is not broken, the pointer must be good still – we can’t prove it to the compiler, but we still know.

In my experience working with libraries that pass control back and forth over boundaries both FFI and not, this occurs reasonably often (libgit2 does it with repository pointers, IIRC) and duplicating the null-checks on incoming pointers is needlessly repetitive.


From a safety perspective, I fully recognize that blindly doing unsafe { &*ptr } or ptr.as_ref().unwrap() is absolutely a bad idea, since it bypasses null checks and might as well be C’s bare *ptr.

Code which wants to be perpetually paranoid (a state I endorse and strive to maintain) should definitely null-check everything, no matter how it elects to do this. My main point here is that, at present, FFI signatures have no means of displaying the difference between taking *T and *T | null, and so we’re still stuck without a typesafe way of denoting what foreign pointers are to be treated with utmost caution and what are, while still dangerous, acceptable to treat somewhat more casually (with the understanding that failure will, of course, bring catastrophe).


I’m certainly not proposing we freeze our ABI or memory model, or leak implementation details of Rust types across FFI. Pointers (and I guess Booleans) are the only type that have their null case as an internal variant rather than an external discriminant, and have this case enforced by hardware. Since FFI boundaries require awareness of the memory model and common representation, I’m certainly amenable to the argument that special-casing Option like this can lead to problems with people writing FFI boundaries that attempt to take other types Optionally and running into problems because C can’t represent that, or muddying the waters about what is/isn’t #[repr(C)].

I still feel that this is a worthwhile use of Option and acceptable instance of fixed representation that shouldn’t restrict how Option behaves anywhere else, but eh.

Hopefully we’ll hear from other folks, especially those who’ve done far more FFI work than I have.


#6

It’s a matter of opinion, but doubling up on Option seems much worse to me.

I don’t have a full response to your points, but FWIW the other two (unstable) abstractions used in the standard library for non-null pointers are Shared and Unique, both building on NonZero. They impart a sense of ownership for dropck which you probably don’t need, but otherwise it seems to roughly fit.

Aside: GCC has the nonnull and returns_nonnull attributes for this scenario.


#7

As nice as Shared/Unique certainly seem, I need a stable ABI that compiles on the stable track. Plain raw pointers and null checking transliterated from C it is, I guess.

Re: 2xOption; my interpretation (which could well be wrong) of Option is that it elevates the absence-form of a type from being one possible concrete value of a type, to a different abstract type. Thus, 0 is not a valid value of an &T; in order to have a potentially null reference it must be denoted as Option::<&T>::None even though its concrete representation is just { value: 0, }, not { tag: 0, value: ???, }.

The whole problem with null-pointers is that they typecheck as pointers but act in extremely different ways; Rust references solve this by not letting null-pointers typecheck as &T. Null-pointer references require ‘Option<&T>’ which has None (null) a distinct type from Some (non-null), even though they optimize to single-pointer memory. That null pointers do typecheck as *const T is predictable and sensible, from the perspective of “what does this look like in memory”, but regrettable from a more abstract viewpoint because it’s the same problem we’ve been having in C.


I suppose this can all be mitigated by declaring FFI functions that should never see null-ptr as receiving &T rather than *const T, but to me that feels supremely bad; &T should never cross FFI because it’s symbolic of a compiler invariant that FFI cannot keep even if it does behave itself.


#8

You appear to be proposing a breaking change, and as such there’s no chance of it being implemented. Rust has been stable for 2 years, and the bar for a breaking change to the language is monumental; especially for one that will silently corrupt unsafe code.

There are many problems with Rust’s raw pointer types, and they’re all being tackled through the Shared and Unique types.


#9

Are you aware of @llogiq’s optional crate that allows you to do this?


#10

@Gankro I was afraid of that. I’ll watch those types; forget I said anything here then I guess.

@marcianx I was not; I’ll give that a look, thanks.