Was there a recent change to extern "C" functions in nightly Rust?

I recently ran into an issue where I was mis-declaring some ffi functions as extern "C" when they should have been extern "stdcall" which resulted in stack corruption after calling those functions. This is the behavior I would expect when calling a function using the wrong calling convention, but what's confusing me is that this code (erroneously) would run without issue before I updated to the 1.11 nightly build. Was there some change made recently to codegen around extern functions that would have broken my already code?

Did you perhaps switch from 64-bit Windows to 32-bit Windows? Because 64-bit Windows normally uses the C calling convention whereas 32-bit Windows uses stdcall.

Nope. Same machine, same OS and architecture. The only relevant thing that changed is that I updated my rust install as far as I can tell.

Could changing the architecture of the Rust binary have done it? I started using rustup to install Rust, and if it defaulted to installing a 32 bit compiler would that impact the generated executable? I would have been using the 64 bit compiler previously.

Definitely. I don't know if it did that though. You can find out your current version with rustc --version -v

Another possibility: did you used to run the mingw toolchain and are now running the msvc toolchain?

I've been using the MSVC toolchain exclusively for a while now. I did accidentally install the mingw toolchain when I first switched to rustup but the issue still occurred when I switched back to MSVC. I'll have to check the installed version when I get home tonight.

I just checked on my machine and I do indeed have the 32 bit version of the compiler installed. When I get a chance to do so I'll install the 64 bit compiler and see if my old code runs again, but it looks like that was the change that caused the problem. Thanks for the help!

In general, when writing Windows software in Rust, you will want to use the extern "system" calling convention so that either stdcall or C is selected automatically depending on your environment.

AFAIK 64 bit Windows uses yet another calling convention, neither stdcall nor cdecl (I don't know if those are even applicable).

It does appear that "Microsoft x64" is a unique thing:

The "C" calling convention is not a specific calling convention (e.g. cdecl), but rather the calling convention that is commonly used by C progams on your platforms. See also FFI and LLVM: llvm::CallingConv Namespace Reference

Surprisingly, there does not seem to be a way in rust to force-use the 64-bit SystemV calling convention on Windows.

Windows 64-bit has a single calling convention so you can just use extern "C" there and everything will work fine.

Windows 32-bit has multiple calling conventions, cdecl stdcall and fastcall (plus some other less common ones that nobody cares about). "cdecl" is the same as "C". "stdcall" is the same as "system". Most of Windows API uses "stdcall", and various libraries will use either "cdecl" or "stdcall", and you'd have to check carefully to know for sure which one they used.

1 Like