Cargo build script with ffi module

help
1

I created a crate that aims to port a C++ library into Rust.
The C++ repo is added as a git submodule.
The build script builds the library with cmake then generates the necessary bindings with bindgen.

When I tried to publish the crate, I got an error status 413 Payload Too Large. This is due to the size of the git submodule of the library that is in the root folder and is larger that 100MB.

What is the best practice in this case ? Should I make the build script clone the repository into the target folder ?
Should I assume that the user has the library installed on his machine ? In that case, there might be some issues : bindgen uses files in the source code of the library that don't end up in the install folder. Also, how can I guarantee that the right version is used ?

The best solution for me right now is to make the build script clone the repo into the target folder, but I feel that there isn't much support for git operations in cargo build.rs...

Thanks in advance

2

That's huge. Maybe you can avoid bundling everything in the submodule to get below the size threshold?

There is git2 or gix you could use to clone the repo during the build. Or invoke git via Command.

3

I tried excluding the submodule directory from the Cargo.toml. This drastically reduced the size of the package! thanks.
But now the issue is that cmake uses env::current_dir() to find the submodule. Which is different in both cargo build and cargo publish commands.
I will try to use gix instead.

4

I didn't mean to exclude the whole submodule, but files and subdirectories from it that aren't essential to building you C++ library. Maybe there are some big asset files or examples or something that bloat up your distribution tarball but aren't necessary for building the library.

5

You're right there were a lot of bloat (mostly asset files). I added them in "exclude" in the Cargo.toml, but still, I ended up having 10MB of source and build files.
I tried excluding the whole folder and cloning it with git2 but that was very unpractical. And the cargo publish command was complaining that files were added to the source folder while publishing the crate. So I ended up reverting the git2 part, excluded the whole folder and published with the --no-verify option.

Thank you

6

If you excluded the C source the package needs, then you've published a package that won't build.

7

Yes, but it will build if you clone the package from git with all its submodules recursively.

8

As far as I understand, crates.io will serve the .crate archive generated during publish, with no attachment to git.

Users would instead have to clone your git repo manually, which negates the benefit of having it published to crates.io

2 Likes
9

What I ended up doing is that I created a "sys" crate in a subdirectory as a dependency that generates the necessary bindings and compiles the library.
This way, only the main crate gets compiled and pushed.

This also improved the build time because the cargo didn't need to rerun the build script every time I compile the crate.

This was inspired by the nuklear-rust crate.