Inclusion of `-lkernel32` and others when compiling `#![no_std]` for i686-pc-windows-gnu

When cross-compiling a #![no_std] crate for i686-pc-windows-gnu it is linked with the following flags the following flags: "-lgcc_eh" "-l:libpthread.a" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-lmingwex" "-luser32" "-lkernel32".

The crate defines its own _entry and main (stmain) functions, and both -nostdlib and -nostartfiles are being passed to the linker. The crate can be found here: GitHub - Irate-Walrus/stardust-rs: An i686 & x86_64 position independent implant template for Rust 🦀

Following is the error output:

error: linking with `i686-w64-mingw32-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/user/go/bin:/home/user/.cargo/bin:/home/user/.local/scripts:/home/user/.local/bin:/home/user/go/bin:/home/user/.cargo/bin:/home/user/.local/scripts:/home/user/.local/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/user/.dotnet/tools" VSLANG="1033" "i686-w64-mingw32-gcc" "-fno-use-linker-plugin" "-Wl,--dynamicbase" "-Wl,--disable-auto-image-base" "-Wl,--large-address-aware" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/i686-pc-windows-gnu/lib/rsbegin.o" "/tmp/rustcExzT5y/symbols.o" "/home/user/Source/stardust-rs/target/i686-pc-windows-gnu/release/deps/stardust-0cbc72f4e9364613.stardust.d04ac4ac0cdcadb1-cgu.0.rcgu.o" "-Wl,-Bstatic" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/i686-pc-windows-gnu/lib/libcompiler_builtins-461ff43f4f8a3a70.rlib" "-Wl,-Bdynamic" "-lgcc_eh" "-l:libpthread.a" "-lmsvcrt" "-lmingwex" "-lmingw32" "-lgcc" "-lmsvcrt" "-lmingwex" "-luser32" "-lkernel32" "-Wl,--nxcompat" "-o" "/home/user/Source/stardust-rs/target/i686-pc-windows-gnu/release/deps/stardust-0cbc72f4e9364613.exe" "-Wl,--gc-sections" "-no-pie" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs" "-Wl,--verbose" "-fpack-struct=8" "-falign-jumps=1" "-w" "-Wl,-s,--no-seh,--enable-stdcall-fixup" "-Wl,--subsystem,console" "-Wl,-T./stardust/scripts/windows.ld" "-nostdlib" "-nostartfiles" "-static" "-fno-ident" "-Wl,--gc-sections,--build-id=none" "-falign-labels=1" "-Wall" "-fno-asynchronous-unwind-tables" "-Wl,-e_start" "-Wl,-Map=/home/user/Source/stardust-rs/target/i686-pc-windows-gnu/release/build/stardust-5660e9920ec42a40/out/stardust.map" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-1gnu/lib/rustlib/i686-pc-windows-gnu/lib/rsend.o"
  = note: /usr/bin/i686-w64-mingw32-ld: /home/user/Source/stardust-rs/target/i686-pc-windows-gnu/release/deps/stardust-0cbc72f4e9364613.stardust.d04ac4ac0cdcadb1-cgu.0.rcgu.o:stardust.d04ac4ac0cdcadb1-cgu.0:(.text.prologue+0xa): undefined reference to `stmain'

I believe the issue is that rustc unconditionally passes these includes for all mingw targets: rust/compiler/rustc_target/src/spec/base/windows_gnu.rs at 5afd5ad29c014de69bea61d028a1ce832ed75a75 · rust-lang/rust · GitHub

How do I resolve this? Is the only solution to patch rustc itself? What further issues would this cause?

Your time and help is much appreciated :slight_smile:

edit: Oh, you are seeing the linking errors on 32-bit builds. Below is 64-bit. I'll try the i686 target in a minute.


I was hoping I would be able to reproduce this issue and help find a solution, but:

"It Works For Me :tm:"

$ cargo make --env TARGET=x86_64-pc-windows-gnu build

... snip ...

[cargo-make] INFO - Execute Command: "cargo" "build" "--package=runner" "--target" "x86_64-pc-windows-gnu"
   Compiling winapi-x86_64-pc-windows-gnu v0.4.0                                                          
   Compiling libc v0.2.164                                                                                
   Compiling winapi v0.3.9                                                                                
   Compiling runner v0.1.0 (/home/jay/other-projects/stardust-rs/runner)                                  
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.80s                                     
[cargo-make] INFO - Build Done in 6.65 seconds.                                                           

$ file target/x86_64-pc-windows-gnu/release/stardust.exe

target/x86_64-pc-windows-gnu/release/stardust.exe: PE32+ executable (console) x86-64 (stripped to external PDB), for MS Windows

This is in Ubuntu 22.04 on WSL with the current rustc nightly. All I installed was:

$ rustup target add x86_64-pc-windows-gnu
$ cargo install --locked cargo-make

And I have the mingw toolchain installed for other projects. (No idea when I installed or what apt packages, but I have mingw-w64 version 9.0.0-1ppa3, FWIW.)

$ rustc --version

rustc 1.84.0-nightly (b19329a37 2024-11-21)
1 Like

Ok, I did a little digging. The issue is unrelated to the linker flags. Those are always passed even for the 64-bit target.

The linker needs leading underscores for some symbols. This patch will make it compile for i686-pc-windows-gnu:

diff --git a/stardust/scripts/windows.ld b/stardust/scripts/windows.ld                      
index 595ac24..d8ee602 100644                                                               
--- a/stardust/scripts/windows.ld                                                           
+++ b/stardust/scripts/windows.ld                                                           
@@ -43,7 +43,7 @@ SECTIONS                                                                  
        */                                                                                  
         . = ALIGN(0x1000);                                                                 
                                                                                            
-        _data_offset = .;                                                                  
+        __data_offset = .;                                                                 
         *(.global*)                                                                        
         *(.data*)                                                                          
         *(.bss*)                                                                           
diff --git a/stardust/src/stcore/arch/i686/i686.asm b/stardust/src/stcore/arch/i686/i686.asm
index 8264e12..34df2a6 100644                                                               
--- a/stardust/src/stcore/arch/i686/i686.asm                                                
+++ b/stardust/src/stcore/arch/i686/i686.asm                                                
@@ -24,7 +24,7 @@                                                                           
         mov    esi, esp          // store current esp in esi                               
         and    esp, 0xFFFFFFF0   // align esp to 16 bytes                                  
         sub    esp, 0x10         // allocate stack space for alignment                     
-        call   stmain            // call the main function                                 
+        call   _stmain           // call the main function                                 
         mov    esp, esi          // restore esp                                            
         pop    esi               // restore esi                                            
         ret                                                                                

You may need to separate the linker script for 32-bit and 64-bit Windows builds. Because this patch as-is breaks x86_64 (__data_offset will not be found). This is a requirement for the __cdecl calling convention. Or, maybe find a way to change the calling convention, if that is compatible with what you are trying to do...

You're a wizard! Thank you! Successfully compiles, although still crashes so time time fire up xdbg XD. Annoyingly this naming convention only holds for x32 so you're right on the requirement for i686-specific linker script. What I'm trying to do? Mainly be stubborn as a donkey and prove to a colleague that it can be done while learning a bunch in the process.

1 Like

And the changes which allow compilation are up! Marking this as solved :slight_smile:

For completeness, llvm has a flag to opt-out: --no-leading-underscore. (This does change the calling convention, making it incompatible with other libraries.)

I think it can be passed through the environment (haven't tried this myself):

RUSTFLAGS='-Cllvm-args=--no-leading-underscore'
1 Like

It appears I may have been too hasty here, I'm seeing a bunch of libraries being included in the map which I've never seen linked in previous targets. This is leading to significantly larger binary.

This includes a bunch of addition unwinding code in the .text section and references to strlen, malloc etc. from libmsvcrt. I've been trying to avoid these by using compiler-builtins. An import table is also being generated.

At the moment the program is crashing as it is generating a .rdata$.refptr.__data_offset which holds at hard coded memory address leading to an exception. Unfortuantly the initial question still stands.

For that, I would recommend trying out some link-time optimization settings. And also play around with some of the other things mentioned in GitHub - johnthagen/min-sized-rust: 🦀 How to minimize Rust binary size 📦

Hmm, it's using identical instructions to x86_64 but I tried the link optimization stuff. Amusingly the -Zlocation-detail=none results in a SEGV during compilation. Comparing the output linker maps I think this is the main difference:

Archive member included to satisfy reference by file (symbol)

/usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o)
                              /home/irate-walrus/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/i686-pc-windows-gnu/lib/rsbegin.o (__register_frame_info)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defs01135.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (abort)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defs01177.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (free)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defs01228.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (malloc)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defs01234.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (memcpy)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defs01279.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (strlen)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defh.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defs01135.o) (_head_lib32_libmsvcrt_def_a)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_deft.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmsvcrt.a(libmsvcrt_defh.o) (_lib32_libmsvcrt_def_a_iname)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libmingw32.a(lib32_libmingw32_a-tlsmcrt.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (_CRT_MT)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32s01416.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (_imp__Sleep@4)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32s00996.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (_imp__LeaveCriticalSection@4)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32s00898.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (_imp__InitializeCriticalSection@4)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32s00321.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/libgcc_eh.a(unwind-dw2-fde.o) (_imp__EnterCriticalSection@4)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32h.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32s01416.o) (_head_lib32_libkernel32_a)
/usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32t.o)
                              /usr/lib/gcc/i686-w64-mingw32/13-win32/../../../../i686-w64-mingw32/lib/../lib/libkernel32.a(libkernel32h.o) (_lib32_libkernel32_a_iname)

Why there is reference to these, dunno. The tell is also really obvious when you check what is being linked in the .text section between the two.

You may have to use -Z build-std to disable unwinding within the standard library. (min-sized-rust also covers this.) That is probably why you are seeing those references.

Another awesome tool to help out with this kind of analysis is cargo-show-asm.

No luck with -Z build-std, still included. The crate doesn't use the standard library anyhow so I don't know how much that flag would affect things. cargo-show-asm looks cool, I've just been using radare2 as that's what I've become comfortable with.

Note that the windows-gnu tries to emulate a unix-ish target in some ways so it includes a lot of stuff to make that work. You might have more luck with windows-gnullvm which is closer to a native Windows target.

1 Like

Attempting to cross-compile for i686-pc-windows-gnullvm results in the error:

error: linker `i686-w64-mingw32-clang` not found

I have clang and mingw-w64 installed so I'm wondering whether this is a expected dependency on windows rather than linux.

Hm, if your distro doesn't have it then you can download llvm-mingw from Releases · mstorsjo/llvm-mingw · GitHub

core also has unwinding, which you are are more likely to be using. See: Unstable Features - The Cargo Book

Using -Z build-std will implicitly compile the stable crates core , std , alloc , and proc_macro .

I don't know if it will help, but if you are using any of those libraries, you might want to build with -Z build-std=core,alloc,panic_abort -Z build-std-features=panic_immediate_abort.

Including these arguments still results in libgcc_eh.a and others being linked. It also gives an error regarding compiler_builtins being included twice but I worked around this by removing the dependency.

So I've given this a go, and while it compiles, it fails during linking:

  = note: lld: error: unknown argument: -T./stardust/scripts/windows.i686.ld
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

I assume this is because i686-w64-mingw32-ld doesn't have a -T argument to specify a linker script (although I don't know any better).

My current hypothesis is that panic=abort is being ignored leading to rsbegin.o (__register_frame_info) being linked. This then has a dependency on libgcc_eh.a(unwind-dw2-fde.o) (for unwinding).

libgcc_eh.a(unwind-dw2-fde.o) then has dependencies on libmsvcrt.a for malloc, free, abort, and so on and so forth.

Even when building with the following flags, this is still the case.

Thank you @parasyte and @chrisd for your time. While I do not have a solution, I do believe I understand the source of the issue here.

I would still like to resolve this, where further should I look? Potentially raise an issue with the compiler team?