Generate concatinated path for include_bytes at compile time

help
1

We have a library crate which is a simple little web framework. It contains some static assets like javascript files, style sheets, images, etc. A few different applications use this framework (including its assets).

By default the (Rocket) applications set up a file serving mount which points to the web framework crate's static directory, but we also support a feature called builtin-content which is used for deployment builds. This feature makes the application build the static files into the binary:

  const LOGO_SVG: &'static [u8] =
    include_bytes!("../../libs/htstatic/static/img/logo.min.svg");

"How can the application reference files in the library crate?", you ask. That's a good question -- it works because all the projects are checked into the same repository, so the application and the static directory and files are deterministically relative to each other.

However, having the library and the application in the same repository has become a substantial burden. We want to separate them so the framework library is in its own crate and each application gets its own crate.

We still want to keep the static files which belong to the framework in the framework's library crate, which means we have to be able to reference its directory from the applications. This can be done by adding a function to the library which gets the library crate's "static assets" path:

pub fn get_static_base_dir() -> PathBuf {
  Path::new(env!("CARGO_MANIFEST_DIR")).join("static")
}

Then the applications set up a static file serving mount which points to framework::get_static_base_dir(). This works fine.

However, it obviously does not work for the builtin-content feature, because:

  • It returns PathBuf
  • include_bytes!() is (understandably) picky about path input type.

I tried making get_static_base_dir() const:

pub const fn get_static_base_dir() -> str {
  concat!(env!("CARGO_MANIFEST_DIR"), "/static")
}

.. and then making the include_bytes!() in the application use concat!():

  let path = concat!(framework::get_static_base_dir(), "img/logo.min.svg");
  const LOGO_SVG: &'static [u8] = include_bytes!(path);

.. but this is clearly not a solution.

Is there a way to do this? I know that all the data is available at compile time, so I know it's possible in theory, but I feel like I'm missing a few important puzzle pieces to stitch it together.

tl;dr

I want to call include_bytes() in an application crate to include a file which lives in a library crate, and I want that library crate to have a function which returns its static assets path at compile time, and the application crate concatenates the path returned from the library crate and a string literal to form the input path to include_bytes().

2

This fundamentally can never work, because macro expansion happens before parsing, and thus before local variables are known about, let alone before name resolution can actually happen. The same applies to a function call, even a const function call -- at macro time, the only names that can be resolved are macro names.

This also doesn't really make sense, because const fns can also be called at runtime, but at runtime this path is essentially meaningless, so you probably didn't want this at all.

If there's a way to do this, it'd need to be something that's entirely at macro-time.

So I'm not sure if this works -- I forget expansion order details, or if you can force-pre-evaluate things like env! -- but it might be possible as something like this:

const LOGO_SVG: &'static [u8] = include_bytes!(concat!(framework::get_static_base_dir!(), "img/logo.min.svg"));

Where it's

  1. Using a macro from the crate instead of a fn, and
  2. Not using local variables.

But stepping back a bit, why do you want this? Why doesn't the library crate expose the file if it's the one who owns it? You're smuggling implicit dependencies here, which is a recipe to make your build system hate you.

If the library provides these files, why not have a static/const for each image in the library? Then the application can just not use the ones it doesn't want.

1 Like
3

Fair point, and I should have been clear about that: The framework contains static content which belongs to different products for different customers (logos and such), and we can't have customer binaries contain assets intended for other customers.

We thought about adding cargo features to switch between the various options, but felt that over time it would become too cluttered with feature flags.

Another solution we looked into was having build scripts copy all the assets into an a directory under $OUT_DIR and then include_dir that directory. However, this had other annoyances; it requires a clean between builds if a shared target directory is used and the developers need to remember that the js/css files now exist in two places, and it's not the ones in the source tree that are actually used.

To make a long story short, we realized that the lowest friction solution we could come up with would be if we could store assets which are unique to the application with the application, and assets which are reused in multiple applications in the framework crate, and the application would "handpick" each asset for inclusion from the framework crate.

With that said, I'm very much open to other suggestions.

4

Is it possible to construct the path in a build.rs script? If so, you could do something like this:

// in build.rs:
fn get_static_base_dir() -> PathBuf {
    Path::new(env!("CARGO_MANIFEST_DIR")).join("static")
}

let logo_svg = get_static_base_dir().join("img").join("logo.min.svg");
println!("cargo:rustc-env=MY_LOGO_SVG_PATH={}", logo_svg.to_str().unwrap());

// in application:
const LOGO_SVG: &'static [u8] = include_bytes!(env!("MY_LOGO_SVG_PATH"));
5

Does your crew use make?
https://www.gnu.org/software/make/

You could maybe evoke your 'cargo' commands with a Makefile and have make generate the environment variable that you put in include_bytes!()

6

I'm trying to avoid using build wrappers -- after decades of having to create more and more complex build systems for C/C++ projects (and all the fiddling to keep them working on all our supported platforms), one of the things I promised myself when I started using Rust was that plain cargo should be able to build our Rust projects.

That being said, I'm starting to realize that I may need to rethink that policy.

1 Like
7

I think this is a point where it's important to look very carefully at exactly what you ship.

If you put a const in your library, then it'll definitely exist in the rust metadata file. But I suspect you're not actually shipping that. And just because there's a const, that doesn't necessarily mean you get anything in the .a/.so for that library.

So I'd suggest that you try the following:

  • Make consts for all the assets in the library, which you never use in the library.
  • Make statics initialized from those consts in the customer-specific application.
  • Look at the binary files that you'd actually ship and see whether the unwanted assets actually show up in them.
4 Likes
8

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.