Suppress name mangling for extern “stdcall”

Not really a tutorial, but an answer to a frustrating riddle: How would you call a stdcall function without a mangled name from a Windows DLL? The standard answer is you don't, because anyone who uses stdcall without mangling names is clearly DoingItWrong™. But wait, there is a way!

For example, let's say MyAwesomeLibrary.dll exports myAwesomeFunction. How would we tell our Rust program about it? We might naïvely try:

#[link(name = "MyAwesomeLibrary")]
extern "stdcall" {
	pub fn myAwesomeFunction(...) -> ...;
}

Unfortunately, that results in link-time error undefined reference to `myAwesomeFunction@N'. That's because stdcall function names are expected to have an @N appended, where N is the decimal number of bytes in the argument list. Our next attempt might use the link_name attribute to force a particular name:

#[link(name = "MyAwesomeLibrary")]
extern "stdcall" {
	#[link_name = "myAwesomeFunction"]
	pub fn myAwesomeFunction(...) -> ...;
}

Again, the same undefined reference to `myAwesomeFunction@N' error appears. Fine, if stdcall doesn't work, then let's just make it extern "C" to skip the name mangling§:

#[link(name = "MyAwesomeLibrary")]
extern "C" {
	pub fn myAwesomeFunction(...) -> ...;
}

Okay, that did something! ... but all is not well. The program runs, and maybe even calls the function successfully, but eventually encounters (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION). This happens because, although the function is referenced by the correct name, the stdcall convention specifies that the callee cleans up the stack instead of the caller. Well, in our program, both caller and callee try to clean up the stack, resulting in chaos.

Various attempts have been made to generate an import library[3] that aliases a mangled symbol@N to a bare symbol in the DLL. But that's a lot of work for something that the toolchain should do for us. Indeed, there is evidence on the GitHub issue tracker[4] that one could provide hints to the underlying LLVM linker. So here's the final attempt that actually works:

#[link(name = "MyAwesomeLibrary")]
extern "stdcall" {
	#[link_name = "\x01_myAwesomeFunction"]
	pub fn myAwesomeFunction(...) -> ...;
}

§You'd notice that, with name mangling turned off by way of \x01, one must manually adhere to the C convention of prefixing with _.

There's also libloading, of course, but one could potentially avoid abusing libloading by using this linker hint.

[1]: hxxps://users.rust-lang.org/t/17141
[2]: hxxps://users.rust-lang.org/t/80508
[3]: hxxps://qualapps.blogspot.com/2007/08/how-to-create-32-bit-import-libraries.html
[4]: hxxps://github.com/rust-lang/rust/issues/17806

I perhaps don't follow, but why not simply #[no_mangle] extern "stdcall"?

1 Like

How do you make #[no_mangle] work like that, though? This was one of the things I tried, and it did not let me import a non-mangled symbol. What I think it does is prevent Rust from exporting a mangled symbol. It would be great if this attribute would cover imports also.