and there is link time optimization between the two?
the inline / inline(always) annotations in rust should be respected by the C code.
i did get this to work (incl the inline annotations) using rustc --crate-type=staticlib -Clinker-plugin-lto -Copt-level=2
and something like clang -c -flto=thin the_c_part.c
but the issue was, the resulting "object files" are actually llvm bitcode, and the user of the library would have to compile with clang -fuse-ld=lld -flto=thin. i just want a plain .lib file.
so the question is: how to combine rust & C into a static librarywithLTO?
this last part was supposed to mean: as the rust & C are "linked" into a static library, they're supposed to be optimized together. now, this is neither LTO nor "linking", which caused a bunch of confusion.
Note that static libraries are not linked (yet). They're just a bunch of loose .o files. You can't have link-time optimization before linking, because LTO's main purpose is to optimize code in the final context it's used in. Without that context you can only speculate, and that context-free speculation is regular optimization.
The best you can do is to ensure their .o files include all of the data needed to perform LTO on them when they're linked.
Tell us when you'll solve that puzzle. My estimation are: between 5 or 10 years of work and maybe PhD or dozen of them.
it's like an attempt to turn minced meat back into whole piece. Plan .lib file doesn't contain anything besides ready-to-use object code.
So you would either need to add lots of information into hidden section which your special liker would use, or, less reliably, may try to reconstruct something closer to bitcode given just a stream of instructions.
The fact that they are just an object files is not a big deal.
The fact that all information needed to do LTO (which is, by it's very nature requires knowledge beyond function A calls B and there are nothing besides that fact) is already lost in object file is a problem.
Thus only two solution: stash that information somewhere (in some extra sections of object file maybe) or [try to] recreate it.
Both are very complicated problems and are not something I, personally, would like to spend my time on.
Without Don't leak non-exported symbols from staticlibs · Issue #104707 · rust-lang/rust · GitHub LTO on staticlibs won't be as effective even if you somehow enable it. LTO depends on as little as possible symbols being exported, but currently all symbols in staticlibs that are not local to a single object file are exported from the entire staticlib, preventing LTO from removing them or propagating constant arguments.
ok guys, sorry.
looks like i worded that question poorly.
the goal is to produce a static library with cross language optimization.
just like the C compiler can do cross translation unit optimization if you do clang -O2 file1.c file2.c -o my_lib.a
i basically want clang -O2 rust_part.rs c_part.c -o my_lib.a
which is possible (and does not require a phd) if you do this:
compile rust & c independently to llvm bit code (--emit=llvm-bc & -emit-llvm)
use llvm-link to produce a combined bit code file.
clang -c -O2 combined.bc -o my_lib.a
the issue is however: rustc's--emit=llvm-bc does not include the dependencies.
the regular --crate-type=staticlib produces a lib file with the dependencies included.
and it's neither trivial, nor stable to go fetch the compiled dependency modules yourself and pass them to clang.
especially, because the stdlib isn't included in the target folder.
and that's why i want to know how to do this the "official" way.
i guess another solution might be to:
compile the C library to llvm bitcode.
tell the rust compiler to "link" to that bitcode file. similar to how you can tell it to "link" to a static library. only that it would have to be passed to llvm.
Yes. It's really hard to understand what do you ask about.
If you would have actually read the article that you have found you would have seen the following tidbit: the Rust compiler's documentation now offers a compatibility table for the various versions of Rust and Clang.
LLVM couldn't do LTO with just random binary files. It specifically needs bytecode (although it may be disguised as .o file to fool the build system) and bincode must be produced by the exact same version of LLVM for all object files.
Don't even dream about the ability to mix versions. It's the exact same issue with single-language LTO, only there it's easier to guarantee you wouldn't get mix of different compilers.
If you don't do that then there are no LTO (with the existing tools).
You have already found the article which explains more-or-less all the steps. If you tried them and they failed then the next question would be to show us what failed.
But your stated goal: the ability to use all that in a form where the user can use whatever compiler they want is not just unachievable currently, it's very explicit NON-GOAL for all that machinery.
That's where years of development and PhD come from (if that's your true goal).
If you don't, really, want that then maybe a better explanation of what you have tried and what worked and what failed would move us closer to some kind of actually achievable goal.
But your stated goal: the ability to use all that in a form where the user can use whatever compiler they want is not just unachievable currently, it's very explicit NON-GOAL for all that machinery.
that's a misunderstanding: i want to ship the library as a static library (as opposed to: as llvm bitcode) so consumers of the library can use any compiler.
the question is pretty simple: how to write a library in rust & c, with cross language optimization, shipped as a static library?
the article describes how to do the first two parts: library & cross language optimization.
1: but the result is llvm bitcode.
so the user of the library would have to use clang. this part:
# Link everything
clang -flto -O2 main.o -L . -lxyz
2: unless, i turn those bitcode files into a static library using the llvm tools.
and that's the furthest i've gotten.
3: but: those llvm bitcode files don't include dependencies of the rust part.
in particular, linking against the resulting static lib will result in unresolved stdlib symbols & require lots of manual linking against system libraries (on windows).
i'll share my current test setup tomorrow. then it'll be clearer where the issue lies.
LTO is a compiler specific operation. LTO-enabled objects generated from LLVM toolchain contains LLVM bitcode as metadata. For ones from GCC contains GIR as well.
This is impossible. LTO is a compiler-specific massive hack that changes meaning of what is a library and what is linking. It requires code in the "static" not-actually-a-library file to contain code that hasn't been fully compiled yet, and is in compiler's internal intermediate representation. It's almost like including source code in the library instead of machine code. You can't LTO a compiled machine code.
I'll reiterate that static libraries are not really reusable libraries for distribution. They're not like dynamic libraries, but static. They are just a bunch of .o files in an .ar file. It's not self-contained. It doesn't include dependency information. It's just an unfinished compilation state zipped up in a single file to look less messy on disk. It still requires all of the flags that you'd use if you just called cc *.o on a bunch of .o files, because that's what it is.
But partial linking exists which means you can turn bunch of .o files and bunch of .bc files into one coherent .o file.
Probably even with Rust. And probably even with LTO.
Thus technically what topicstarter wants may exists, he just needs not static library but partially linked object file.
But I, for one, wouldn't touch such a thing with ten-feet pole given the choice of picking anything else (dynamic library, something with source, remote service which I can put in container, etc) thus I'm not sure I even want to help building such monstrocity.
// compile rust with (outputs `lib.lib`)
rustc --crate-type=staticlib -Clinker-plugin-lto -Copt-level=2 ../src/lib.rs
// compile C with
clang -c -g -O2 -flto=thin -emit-llvm -o the_c.bc ../the_c.c
// extract the `lib.lib` file output by rustc using
llvm-ar x --output lib lib.lib
// separate the "object files" into "real object files" (`lib-o`)
// and "llvm bitcode files" (`lib-bc`).
// combine `the_c.bc` with the rust library's bitcode files into `biglib.bc` using
llvm-link lib-bc/* the_c.bc -o biglib.bc
// compile the combined bitcode to get `biglib.o`
clang -c -O2 biglib.bc
// put all "real object files" back into an archive using
llvm-ar cr biglib.lib lib-o/* biglib.o
// users can now compile with msvc for example:
cl main.c biglib.lib *native-libs* // where the list of native libs can be obtained using `rustc --print native-static-libs`.
and that's why i wanted to know how to tell the rust compiler to do this.
all i really want is to: add a C file to the rust compilation, so they're optimized together.
now, rustc isn't a C compiler, so the closest thing would be to add an llvm bitcode file to rustc's input files. if that's not possible, i'm afraid that nice recipe above is how do to this.
why would anyone build such a monstrosity?
it's just a static library. not some scary monster.
this is incredibly common practice, if you want to, for example, ship a program as a single .exe file instead of a huge folder with a bunch of dlls. like a small, clean "portable version" of a program.
looks like that "LTO" part caused a bunch of confusion. i updated the question.
perhaps now, you can think of a simpler solution than my "recipe" above?
and as a bit of feedback for the "post contributors": maybe assume that what i'm trying to do is reasonable and try to understand what i actually want to do, instead of telling me, what i'm trying to do is impossible or requires years of research. especially, when i've stated multiple times, that i've almost got it working.
i'd like to see more constructive responses on this forum. trying to understand instead of preaching & teaching.
something similar happened last time i posted here. people told me how what they thought i was trying to do is stupid or impossible.
now granted, part of that is on me. i only post here, when i have some rather nasty, rust specific issues, that i'm confused about. but that's what forums are for, getting help, when you need help. sadly people here don't seem to put much effort into clearing up confusion.
an example: @kornel instead of saying "This is impossible. LTO is a compiler-specific massive hack",
you could ask "are you sure you want to ship the library with LTO? LTO is compiler specific, so your library will only work with specific compilers."
and then i would have responded: "oh no, that's a misunderstanding. i don't mean LTO in the sense that my library is supposed to be optimized at link time, when the user builds their program. i want the rust and c parts to be optimized together before creating the static library. so the user can use it with any compiler that supports binary object files."
no problem!
i'll be more careful with my choice of technical terms next time
for now, i'm marking my weird recipe as the solution.
still hoping that there's some way to feed llvm bitcode files directly to rustc, that'd be sweet.
