Build script dependency doesn't work properly

I have a package with build script that generates .rs file with include_bytes!("/path") in it. /path itself is generated by build script of another package specified both in dependencies and build-dependencies.

While this always works fine in the monorepo where both packages live and are specified by relative paths, it doesn't seem to work when used in another package via git dependency, but only in CI, locally works fine too.

I'm quite surprised how is this even possible, I would have expected listing it in build-dependencies would be sufficient to ensure build order and availability of that file.

Here is one crate: subspace/Cargo.toml at c4e76bff5c724aff123993d60049166e059977e2 · subspace/subspace · GitHub
Here is another that above depends on and generates that missing file: subspace/Cargo.toml at c4e76bff5c724aff123993d60049166e059977e2 · subspace/subspace · GitHub
And this is CI run where things didn't work properly: publish · subspace/subspace-desktop@ad99093 · GitHub

According to CI logs it seems like the build script should have finished and file should have been generated already.

Are you writing the file to the source directory? That isn't allowed. And in fact attempting to publish a crate which modifies the source directory during building will fail. You have to write to the directory pointed to by the OUT_DIR env var. This directory is different for each crate. I couldn't quite puzzle together which crate generates the file and where it does so and which reads it and where it does so.

No, it is written under target directory, that behavior we largely inherit from one of the dependencies.
In that case it is target/x86_64-unknown-linux-gnu/release/wbuild/CRATE_NAME (because we use --target x86_64-unknown-linux-gnu).

How are you trying to read it from the other crate?

One build script generates a WASM blob (this is why it creates another crate subdirectory under target and produces .wasm files there). Then in parent package I want to include it as binary into a variable. Since parent crate is also compiled into WASM, file needs to be available at compile time. So I generate an .rs file in build script there and include it directly: subspace/lib.rs at c4e76bff5c724aff123993d60049166e059977e2 · subspace/subspace · GitHub

OUT_DIR is not shared between crates. One thing you could try is the links feature: Build Scripts - The Cargo Book I believe for that you did add links = "some_unique_name" to the [package] section of the crate producing the file. Then put println!("cargo:WASM_FILE={}", path_to_wasm_file); in it's build script and finally use env!("DEP_PACKAGE_NAME_WASM_FILE") to get the path of the wasm file from the parent package. (where PACKAGE_NAME is replaced by the name of the package that produced the wasm file)

1 Like

Code doesn't expect OUT_DIR to be shared between crates, if you look at the error carefully, you'll see that generated files are not stored under OUT_DIR of another crate, they are stored under target/x86_64-unknown-linux-gnu/release/wbuild/CRATE_NAME instead.

An idea with DEP_PACKAGE_NAME_WASM_FILE is interesting and indeed less hacky than my current solution, but the problem is that dependency doesn't seem to be built at correct time. The path to the generated WASM file looks correct and it should have been there.

At subspace/lib.rs at c4e76bff5c724aff123993d60049166e059977e2 · subspace/subspace · GitHub you are attempting to read a wasm file compiled by the dependency, right? OUT_DIR here refers to the current crate.

That file is generated by build script of the current crate in OUT_DIR, it is correct.

Then I have no clue what is going on.

Yeah, the only explanation I have is that build script of dependency specified in build-dependencies didn't run, but that doesn't make any sense. It is harder to debug due to CI environment and locally everything works fine all the time :confused:

I didn't figure out why it didn't work the way I had it set up originally, but with links and cargo:WASM_FILE it does work properly. Thank you for pointing it out, I had no idea it exists :heart: