Kernel modules made from Rust

#1

Hi there!

I doing a little experiment writing FreeBSD kernel module with Rust. However, I’m stuck with a linking issue…
The project is inspired by https://github.com/tsgates/rust.ko so the approach is similar.

Method:
I build object files from rlibc, core and my simple rust hello world (which is called from the kernel module’s c-code).
I have custom target json file, similar to rust.ko.

Problem:
When loading the module into the kernel I get error message:
__floatundisf undefined.
(kernel modules are linked with the command: ld -d -warn-common -r -d -o ${module}.ko ${obj_files})

However, if I link the same objects with a simple test executable using clang, it works.
EDIT: Not really true, adding -nostdlib to clang and it reports same error as ld.

Linking the same executable with ld and I get a bunch of undefines like
__floatxxx __mulxxxx __udivxxxx
and so on

I have soft-float enabled and tried no-compile-rt both true and false in the json…

I’d be grateful for any hints how to proceed…

EDIT:
As suspected, when adding -nostdlib flag, ld and clang behaves the same, reporting many undefines.
Post updated to reflect.

Using Rust in a Linux kernel module
#2

Exact error message

core.0.o: In function `_$LT$f32$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$::from_int::h179883275ee91593':
core.cgu-0.rs:(.text._ZN59_$LT$f32$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$8from_int17h179883275ee91593E+0x19): undefined reference to `__floatundisf'
core.cgu-0.rs:(.text._ZN59_$LT$f32$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$8from_int17h179883275ee91593E+0x5b): undefined reference to `__eqsf2'
core.cgu-0.rs:(.text._ZN59_$LT$f32$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$8from_int17h179883275ee91593E+0x80): undefined reference to `__floatundisf'
core.0.o: In function `_$LT$f64$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$::from_int::h239c41b5536e77ff':
core.cgu-0.rs:(.text._ZN59_$LT$f64$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$8from_int17h239c41b5536e77ffE+0x19): undefined reference to `__floatundidf'
core.cgu-0.rs:(.text._ZN59_$LT$f64$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$8from_int17h239c41b5536e77ffE+0x61): undefined reference to `__eqdf2'
core.cgu-0.rs:(.text._ZN59_$LT$f64$u20$as$u20$core..num..dec2flt..rawfp..RawFloat$GT$8from_int17h239c41b5536e77ffE+0x86): undefined reference to `__floatundidf'
core.0.o: In function `core::num::_$LT$impl$u20$i128$GT$::overflowing_mul::hc378d57b7ee18460':
core.cgu-0.rs:(.text._ZN4core3num22_$LT$impl$u20$i128$GT$15overflowing_mul17hc378d57b7ee18460E+0x48): undefined reference to `__muloti4'
core.0.o: In function `core::num::_$LT$impl$u20$u128$GT$::overflowing_mul::h483e3f93b6a5fc03':
core.cgu-0.rs:(.text._ZN4core3num22_$LT$impl$u20$u128$GT$15overflowing_mul17h483e3f93b6a5fc03E+0xaf): undefined reference to `__udivti3'
core.0.o: In function `_$LT$f32$u20$as$u20$core..ops..Mul$GT$::mul::ha65f193a4ec3d58b':
core.cgu-0.rs:(.text._ZN38_$LT$f32$u20$as$u20$core..ops..Mul$GT$3mul17ha65f193a4ec3d58bE+0x12): undefined reference to `__mulsf3'
core.0.o: In function `_$LT$f64$u20$as$u20$core..ops..Mul$GT$::mul::h243b6af4b9bfddb0':
core.cgu-0.rs:(.text._ZN38_$LT$f64$u20$as$u20$core..ops..Mul$GT$3mul17h243b6af4b9bfddb0E+0x1b): undefined reference to `__muldf3'
core.0.o: In function `_$LT$u128$u20$as$u20$core..ops..Div$GT$::div::hfd3c131e8c692064':
core.cgu-0.rs:(.text._ZN39_$LT$u128$u20$as$u20$core..ops..Div$GT$3div17hfd3c131e8c692064E+0x48): undefined reference to `__udivti3'
core.0.o: In function `_$LT$f32$u20$as$u20$core..ops..Div$GT$::div::h9a13f8f66da5f35b':
core.cgu-0.rs:(.text._ZN38_$LT$f32$u20$as$u20$core..ops..Div$GT$3div17h9a13f8f66da5f35bE+0x12): undefined reference to `__divsf3'
core.0.o: In function `_$LT$f64$u20$as$u20$core..ops..Div$GT$::div::he844e8be03334b0d':
core.cgu-0.rs:(.text._ZN38_$LT$f64$u20$as$u20$core..ops..Div$GT$3div17he844e8be03334b0dE+0x1b): undefined reference to `__divdf3'
core.0.o: In function `_$LT$u128$u20$as$u20$core..ops..Rem$GT$::rem::hbe179fae4587c6a8':
core.cgu-0.rs:(.text._ZN39_$LT$u128$u20$as$u20$core..ops..Rem$GT$3rem17hbe179fae4587c6a8E+0x48): undefined reference to `__umodti3'
core.0.o: In function `core::fmt::num::_$LT$impl$u20$core..fmt..Display$u20$for$u20$i128$GT$::fmt::h4d2a89fe835906cb':
core.cgu-0.rs:(.text._ZN4core3fmt3num53_$LT$impl$u20$core..fmt..Display$u20$for$u20$i128$GT$3fmt17h4d2a89fe835906cbE+0x1fd): undefined reference to `__umodti3'
core.cgu-0.rs:(.text._ZN4core3fmt3num53_$LT$impl$u20$core..fmt..Display$u20$for$u20$i128$GT$3fmt17h4d2a89fe835906cbE+0x225): undefined reference to `__udivti3'
core.0.o: In function `core::fmt::num::_$LT$impl$u20$core..fmt..Display$u20$for$u20$u128$GT$::fmt::hde81aeb0f4416fa9':
core.cgu-0.rs:(.text._ZN4core3fmt3num53_$LT$impl$u20$core..fmt..Display$u20$for$u20$u128$GT$3fmt17hde81aeb0f4416fa9E+0x1ec): undefined reference to `__umodti3'
core.cgu-0.rs:(.text._ZN4core3fmt3num53_$LT$impl$u20$core..fmt..Display$u20$for$u20$u128$GT$3fmt17hde81aeb0f4416fa9E+0x214): undefined reference to `__udivti3'
#3

When asked to link, clang just executes ld with various flags added. You can see those flags with -v (and manually replicate them), or ask it to pass additional arguments directly to the linker with -Wl,foo. Either way should suffice to let you produce a relocatable object file while still adding the libraries you need. Alternately, you can probably just ask rustc to add the arguments and have it do the linking itself (by invoking gcc/clang, which in turn invokes ld) as usual, though you’ll have to fool it as to output type.

(You may also want to add --gc-sections to reduce binary size, though I’m not sure if that’s sufficient.)

#4

You’re not allowed to do floating point operations inside the kernel. The missing functions are provided by libgcc.a which the kernel specifically excludes.

#5

Yes, this is why I’m trying to build a libcore that does not use floats.
Others have succeeded before so it should be possible…

My target.json

{
	"llvm-target": "x86_64-unknown-none",
	"target-endian": "little",
	"target-pointer-width": "64",
	"os": "none",
	"arch": "x86_64",
	"data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128",
	"pre-link-args": [ "-m64" ],
	"cpu": "x86-64",
	"features": "+soft-float,-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-3dnow,-3dnowa,-avx,-avx2",
	"disable-redzone":         true,
	"custom-unwind-resume":    true,
	"eliminate-frame-pointer": true,
	"linker-is-gnu":           true,
	"no-compiler-rt":          true,
	"archive-format":          "gnu",
	"code-model":              "kernel",
	"relocation-model":        "static"
}

Built with

rustc --target x86_64-unknown-none -Z no-landing-pads --out-dir . ~/dev/rust-src/src/libcore/lib.rs
#6

After adding this I got my kernel module to load and execute the Rust code:

#[no_mangle]
pub extern "C" fn __floatundisf() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __eqsf2() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __floatundidf() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __eqdf2() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __muloti4() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __udivti3() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __mulsf3() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __muldf3() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __divsf3() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __divdf3() {
    loop {}
}
#[no_mangle]
pub extern "C" fn __umodti3() {
    loop {}
}

Is there no current way of disabling float completely?

#7

I haven’t read back through it, but there was a thread about float-free core on the internals forum last year: https://internals.rust-lang.org/t/pre-rfc-float-free-libcore/3403/

What do the pure-Rust kernels like Redox do?

#8

So! A quick summary here:

You can use +soft-float in your target spec to not have the instructions generated.

There’s no good way, currently, to just absolutely remove floating point from libcore; back before the +soft-float trick, the osdev community maintained patches to libcore that added a cfg flag; but that tied everyone to specific nightlies, which was a huge pain. Now, with xargo + +soft-float, things sort of just work.

This RFC:

Would allow us to have a lint inside of libcore that would warn if you tried to use floating support inside the kernel.

I’m a bit confused; I would agree with @comex that you want to add --gc-sections here to get rid of these, but it’s not clear to me why it’s not doing it; rustc usually passes that already, I thought. Hrm.

Also, -Z no-landing-pads isn’t really a thing anymore, you want to set up panic=abort instead. You can do that like this: https://github.com/intermezzOS/kernel/blob/master/Cargo.toml#L19-L23

Oh and, as mentioned above breifly, @Yohanesu75, you probably want to use xargo here: https://crates.io/crates/xargo It will handle setting up libcore for you transparently.

Hope this helps!

#9

add --gc-sections

FreeBSD kernel modules are built with ld -r which is incompatible with --gc-sections…

EDIT: Is there any soft-float static library to link in for the missing functions? I read something about compiler-rt but not sure what that is used for…

#10

Thanks for the info. I read about this and that in different forums but wasn’t sure about the current state of things. Maybe I’ll give xargo a go.

#11

I am not totally sure. compiler-rt is for intrinsics, basically. Given that I don’t have compiler-rt turned off in my OS’s target spec, maybe that’s what it uses, though I also don’t see --gc-sections anymore; when I converted from makefiles to cargo.toml, I guess that went away. I also thought xargo couldn’t cross-compile compiler-RT. Hrm.

#12

Yeah… These __mulxx etc functions are annoying. I guess they are soft float replacements and if I link to libgcc and friends all is ok, however, the kernel won’t be happy if I static link gcc in the kernel module.

Wonder if these can be generated by llvm?.. I guess maybe not, since they are missing even with soft-float flag.

Looks like stubs or patched libcore. I think that this problem does not exist on Linux because of how the kernel modules are linked.

EDIT: As you can see, I’m fumbling in the dark :slight_smile:

#13

xargo seems nice. Build works perfect without me having to keep the core source code around manually.
So, to summarize the whole thing, this is what I have.

My custom target config

{
	"llvm-target": "x86_64-unknown-freebsd",
	"target-endian": "little",
	"target-pointer-width": "64",
	"os": "none",
	"arch": "x86_64",
	"data-layout": "e-m:e-i64:64-f80:128-n8:16:32:64-S128",
	"pre-link-args": [ "-m64" ],
	"cpu": "x86-64",
	"features": "+soft-float,-mmx,-sse,-sse2,-sse3,-ssse3,-sse4.1,-sse4.2,-3dnow,-3dnowa,-avx,-avx2",
	"disable-redzone":         true,
	"custom-unwind-resume":    true,
	"eliminate-frame-pointer": true,
	"linker-is-gnu":           true,
	"no-compiler-rt":          true,
	"archive-format":          "gnu",
	"code-model":              "kernel",
	"relocation-model":        "static"
}

Building stock core with xargo build --target mytarget works fine but soft float depends on some functions defined in libgcc (and possible some more files).

Since we can not link the kernel module with --gc-sections on FreeBSD, we have to find some other way to fix this.

Declaring stubs for missing functions like this is enough to get hello world up and running (patching core would be another solution).

#[no_mangle]
pub extern "C" fn __floatundisf() {}
#[no_mangle]
pub extern "C" fn __eqsf2() {}
#[no_mangle]
pub extern "C" fn __floatundidf() {}
#[no_mangle]
pub extern "C" fn __eqdf2() {}
#[no_mangle]
pub extern "C" fn __mulsf3() {}
#[no_mangle]
pub extern "C" fn __muldf3() {}
#[no_mangle]
pub extern "C" fn __divsf3() {}
#[no_mangle]
pub extern "C" fn __divdf3() {}

However, concerning is that I also need to declare stubs for these integer functions… Can one assume that these will not be called as long as one don’t use floats?

#[no_mangle]
pub extern "C" fn __udivti3() {}
#[no_mangle]
pub extern "C" fn __umodti3() {}
#[no_mangle]
pub extern "C" fn __muloti4() {}
#14

One can’t. These will definitely be called if you perform certain 128-bit integer operations.

I’m sure it would technically be possible to come up with a linking mode that combines -r and --gc-sections, but I don’t know if anyone has implemented this. In fact, I suspect something like this is done for C modules, otherwise there would be a lot of unused code in FreeBSD kernel memory.

#15

One can’t. These will definitely be called if you perform certain 128-bit integer operations.

Thought so… Guess they are not related to soft-float but maybe missing because of the no-compiler-rt flag. What I don’t really understand is why all these bare-metal / kernel target configs disable compiler-rt. Why not enable it and get llvm’s built in functions?

#16

There’s a pure-rust impl of intrinsics at compiler-builtins. These are expected to eventually become the implementation of intrinsics for rust programs, once fully ported.

In the meantime you could extract the necessary intrinsics directly into your project or something.

#17

I don’t think that no-compiler-rt even does anything (anymore?), a quick search of the Rust source didn’t find anything. All the undefined references are from the compiler_builtins, have you tried adding 'extern compiler_builtins ’ to the root of your crate. I think that should work provided you use Xargo.

#18

Thanks! I guess that would solve it. Some are there, however, most of my missing functions have not been ported from C to compiler_builtins yet.

#19

Yes, but I mean the current, compiler-rt based, compiler_builtins crate that is part of std crates I think. I assume Xargo will build that but I could be wrong.

#20

Xargo will happily build compiler_builtins if you tell it to in Xargo.toml:

[dependencies]
compiler_builtins = {}

:slight_smile: