Incorrect FFI argument positions with specific return type

I'm attempting to call into a C wrapper function which wraps a call to OpenCV.

I've observed a case where a function returning the type cv::Vec3b results in the first argument seemingly being treated as the return value and the other arguments getting shifted left one position.

cv::Vec3b's size as well as my Rust struct are both 3 bytes in size, so I'm not suspecting any issues on that front. It's possible I'm doing something else incorrect, but I've been unable to track it down.

See runnable code (required OpenCV 2.4) and more info here:

When you look at the assembly of the C++ code (I'm using release mode, so the code is cleaner):

objdump -dC target/release/build/rust-ffi-return-bug-9c1a9f48b8dd0024/out/native/test.o

you'll see that call_me2, which returns Bytes<char, 3> squeezes the whole returned struct in the %rax register, but the call_me3, which uses opencv's Vec3b returns the struct in a location described by a hidden pointer passed as the first argument (that explains the argument shifting).

The behaviour of call_me3 is really surprising for me, because small structs are passed in registers in C (on x86_64). The only explanation I can think of is this (from, p.27):

If a C++ object has either a non-trivial copy constructor or a non-trivial destructor, it is passed by invisible reference (the object is replaced in the parameter list by a pointer that has class INTEGER)

It's really suspicious why the Vec/Matx types in opencv would need a non trivial constructors though. Edit: It seems that is in fact the case! You can check it by printing std::is_trivially_copyable<cv::Vec3b>::value (it's 0). So now I wonder, how the Rust bindings to opencv have been working at all? Unfortunately, I couldn't make them compile, so I haven't tested them. Anyway, Rust can only use #[repr(C)], which assume register-representation so I have no idea how to work with such structs as Vec3b.

Also, two nits about the code:

  • libc = 0.1 is old and sometimes incorrect, use 0.2.
  • char signedness depends on architecture, so you shouldn't use i8 or u8 to represent it. Either use libc::c_char, or if you want to use u8 in Rust, use uint8_t on C++ side (or even unsigned char, although that's theoretically incorrect).

Thank you!

That excerpt from the C++ specs shines a lot of light on the subject.

Most of the OpenCV wrapper works by returning pointers to C++ objects. In this case I was hoping to return Vec types directly as they're all pretty small.

A little more research on the C++ specs also indicates that returning C++ classes from extern C functions is undefined and may violate the C ABI which is what we are seeing.

From StackOverflow quoting the C++11 draft (7.5.9):

Linkage from C++ to objects defined in other languages and to objects
defined in C++ from other languages is implementation-defined and
language-dependent. Only where the object layout strategies of two
language implementations are similar enough can such linkage be