Not matching ABI between C++ and Rust

[hystericfleece]

I generated bindings for some C++ member functions with bindgen:

class Wrapper {
public:
    Wrapper getReversed() const
    {
        ...
    }
private:
   float data[5];
}

To get:

#[repr(C)]
pub struct Wrapper {
   pub data: [f32; 5]
}

extern "C" {
    pub fn Wrapper_getReversed(this: *const Wrapper) -> Wrapper;
}

However, when calling this function, I am getting unexpected (and wrong) results!
I dug a little bit deeper to try to understand why this might be happening, and I saw that Rust passes the result address out: Wrapper in rcx, and input address this: Wrapper in rdx. However, on the C++ side the registers are swapped, which kind of makes sense with rcx containing this per Windows calling conventions:

The calling convention for C++ is similar. The this pointer is passed as an implicit first parameter. The next three parameters are passed in remaining registers, while the rest are passed on the stack.

Which results in UB: out: Wrapper after the call will contain garbage, while this: Wrapper will contain the function call result (note that it is not mut and all).
So this is confusing.. And the only conclusion I came to is that the C++ is compiled with vs2019, while Rust uses vs2022 libs internally.
I also found this discussion, from which it seems like MSVC compiler is not used at all by Rust, so I am getting kind of stuck now.

Rustup on Windows: How to select MSVC version

On my machine I have two versions of visual studio installed: VS2019 and VS2022. They differ in the compilers and also in the libraries they link to. Now I want to compile the wasmer (Webassembly Virtual machine) c-api into a library that can be consumed by a C++ project. The thing is that I need to specifically control which version of the VS library versions the resulting wasmer.lib is being linked against. So I was looking how the toolchain of rustup can be modified. I also tried to modify th…

Would be glad for any help!

Here is an example: GitHub - hystericfleece/rust-cpp-abi

On my machine it starts to print garbage after getReversed is executed.

[derspiny]

C++ compilers do not promise a C-compatible ABI, in general, and conversely, Rust does not promise to make calls compatible with a C++ ABI. So far as I'm aware, the only portable way to interface with C++ programs is to go through a C-compatible layer - functions in the C++ program exported as extern "C", called using a C (not C++) calling convention.

That generally means that objects themselves cannot be passed through the resulting FFI boundary, but rather have to be coerced into some kind of C-compatible value (frequently a void * or a typedef for same, whose expected "real" type is recovered with a cast when needed).

The compilers involved can't stop you from calling a C++ function with the wrong ABI, which is what you're seeing here, but it invalidates your program from that point onwards. Getting garbage back at least means you know something is wrong.

This is not something you can fix, other than by not calling C++ directly from other languages. You can't even reliably call C++ from other C++ compilers on the same platform, in most cases; you might luck out if you happen to be on a platform (like Windows) that has a strong recommendation for how C++ code should be compiled, but most platforms leave it entirely to the implementation.

You might be able to define an FFI-friendly interface for your C++ types as similar to this:

extern "C" {
  void *new_wrapper();
  void destroy_wrapper(void *wrapper);
  void *wrapper_getReversed(void *wrapper);
}

using top-level functions with C calling and naming conventions, and opaque pointers. to keep the C++-specific parts of your program away from non-C++ interfaces. It's tedious and error-prone, but it can work. bindgen can also generate Rust bindings for a limited subset of C++, so it's worth trying that, but passing objects around as values, outside of the C++ code, is unlikely to work well.

[hystericfleece]

So far as I'm aware, the only portable way to interface with C++ programs is to go through a C-compatible layer - functions in the C++ program exported as extern "C", called using a C (not C++) calling convention

Bindgen advertises itself as being at least somewhat C++ compatible. And considering that my struct is simple enough: it doesn't have destructors and doesn't have vtable, its definition should be a simple [f32; 5] (and this is actually true!).

The problem only arises when I try to call member functions, and I believe if it was immediately obvious that their ABI will not match what is going to be generated, then bindgen would just skip generating it.

You might be able to define an FFI-friendly interface for your C++ types as similar to this:

I use a similar approach for things which have v-tables and non-trivial data inside of them, but I really was hoping to simplify things for simple cases.

[khimru]

@derspiny

but most platforms leave it entirely to the implementation

Well, I guess if you count exotic, long dead, platforms then it's even true but macOS/iOS/PadOS and Android with GNU/Linux are covered by Itanium C++ ABI and Windows is more-or-less standartized, too, which, for practical purposes, covers almost everything most people care about.

@hystericfleece

I really was hoping to simplify things for simple cases.

Unfortunately “simple cases” don't include Windows. That's extremely popular, yet also extremely weird and, most importantly, underdocumented, platform thus I'm not surprised that Rust doesn't cover it.

It took years for clang developers to make clang be somewhat compatible to MSVC, perhaps few years down the road Rust would work, too.

Worth filing an issue for it? Would bring some answers, worth at least trying.

[farnz]

Going in a different direction, would cxx.rs simplify your binding work? It's a different direction to bindgen, and has a different set of tradeoffs, but it's designed to make binding Rust and C++ together relatively ease. Chrome developers have extended it with autocxx which generates the bindings automatically for you (with a different set of tradeoffs again to hand-written bindings).

[hystericfleece]

I thought the problem was here on this line:

hystericfleece/rust-cpp-abi/blob/main/src/wrapped.cpp#L4

Constructors make structs non-POD (just as destructors and vtables would), which causes getReversed() to use a different calling convention, which Rust (and bindgen too) does not support.

Tried writing an example without constructors. The struct should be POD now, right?
Sadly it still fails: Try remove constructor and private · hystericfleece/rust-cpp-abi@2be51b4 · GitHub

[hystericfleece]

I wanted to use cxx in general, but then decided not to: for simple cases it felt like it should have been possible to generate bindings without additional layers, while everything else, I needed to write bindings for, quite extensively used std::optional and std::string_view , so I had to generate my own C++ layer and at that point bindgen made more sense.

[CAD97]

bindgen mostly claims to support this C++ API — Wrapper is a standard-layout type and methods are listed as a supported C++ feature. However, bindgen also notes “many C++ specific aspects of calling conventions” as being an unsupported feature.

Because bindgen explicitly claims to not handle any knowledge about C++ calling conventions, this isn't a bug in bindgen per say, but it does feel like bindgen should give a warning when there are straightforward calling convention issues like this one^[1].

I have a vague recollection of the “fix” being that C++ member functions (methods) on Windows use the extern "thiscall" calling convention and not extern "win64" (which is what extern "C" is aliased to on x86_64 Windows). Unfortunately there's no ABI name alias that works independently of target for C++ methods.

---

1. A much more subtle calling convention mismatch issue that the bindgen docs call out is when ABI mandates user defined types to be passed in registers versus by pointer based on whether a type has any non-defaulted special member functions (ctor/dtor). The Rust bindings can't replicate that context so the calling convention always treats types as trivial for the purpose of calls and passes them using the base C ABI instead. This isn't something bindgen can fix — you just need to ensure your FFI signatures don't pass nontrivial C++ types by value. ↩︎

[CAD97]

@hystericfleece

I wanted to use cxx in general, but […] I had to generate my own C++ layer and at that point bindgen made more sense

If you're generating (or writing) FFI binding adapters, using cxx can still be a good tool, since even if you don't use any of the provided runtime support for bridging Rust std or C++ STL types, it still handles generating the shims for bridging methods and other mangled functions in a way more resilient than just bindgen. And if the C++ header is already designed to be FFI friendly, autocxx allows you to bypass needing to specify the API again on the Rust side and just use the C++ header as the source of truth.

The "extra" bindings layers are unfortunately just necessary. If they're fully trivial, the compiler will alias the forwarding layer at the linker level instead of generating duplicate function bodies. If they aren't, then the shim is actually necessary to bridge between the FFI calling convention and the ones used in either language.

There is no silver bullet, unfortunately. But at least doing the legwork to bridge functionality between reasonably usable APIs on either side of the FFI bridge is a linear amount of work to the API surface.

[farnz]

IME, the cxx.rs maintainers are very willing to accept contributions for C++ std type bindings if there's only one good way to do it.

In this case, you'd be looking at adding support for std::string_view, and support for std::optional, both of which have open issues with some guidance on how to do it (and on how to do it downstream of cxx.rs if you'd prefer).

I would strongly recommend cxx.rs over bindgen for C++ bindings for the reason @CAD97 has already given; basically, it handles some of the boilerplate shims between C++ and Rust for you in generated code.