Need help understanding a crash with a C library


#1

When I compile with the MSVC version of Rust, whenever I open a file with the vorbis library I get a crash. When I compile with the MinGW version, everything works fine.

Here’s what happens:

The exact message given by Visual Studio’s debugger is: Exception thrown at 0x000000013FCD6A66 in basic.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF. You can find a screenshot with the stack trace and the local variables here.

The error doesn’t even make sense. The only memory that is read here (f and callbacks) is the stack.
Because it only happens with MSVC, I suspect a problem during the compilation, but I don’t know enough about this topic to have an idea what to search for.


#2

More investigations. It turns out that the values in the callbacks struct are wrong. This is probably the cause of the problem.

If I add the following line in the Rust code:

unsafe { println!("{:?}", std::mem::transmute::<_, &(usize, usize, usize, usize)>(&callbacks)); }

…I can see that correct values are printed. Even if I put the println right before calling the C library the correct values are still printed.

However inside ov_open_callbacks the values have become garbage.


#3

If I change the function signature here from:

pub fn ov_open_callbacks(datasource: *mut libc::c_void, vf: *mut OggVorbis_File,
                initial: *const libc::c_char, ibytes: libc::c_long, callbacks: ov_callbacks)
                -> libc::c_int;

To:

pub fn ov_open_callbacks(datasource: *mut libc::c_void, vf: *mut OggVorbis_File,
        initial: *const libc::c_char, ibytes: libc::c_long, callbacks: *const ov_callbacks)
        -> libc::c_int;

(*const ov_callbacks instead of ov_callbacks)

Then it works!

However if I do that, then it doesn’t work with the MinGW version anymore :frowning:

Is that an issue with rustc?


#4

I opened https://github.com/rust-lang/rust/issues/28676


#5

Urgh, yeah, sounds like a difference in ABI.

I know that retep998 found an ABI problem with LLVM where it generates calls which are incompatible with MSVC for stdcall member functions.