Memory consume of include_bytes!

If I use include_bytes! to bundle a data file into a Rust executable, will the data consume physical memory? Or does it use some kind of virtual memory?

It'll probably be included in the data segment of the compiled executable, use objdump on your compiled artefact to check. (edit: yes it will use memory, "virtual" memory is an OS abstraction)

If it ends up in the read-only .rodata section, there's a benefit that any number of running processes will share the same physical memory pages for that data, just mapped to their own virtual memory addresses.

2 Likes

So it is not a good option from memory consumption pointview if I have a relatively large data file (say, around 1G).

If it's immutable and the exact same data is used every time then I'd say it's probably fine to include it in the binary, though it will make a binary with a rather extreme requirement w.r.t. available RAM.

Of course not including it will likely mean that each application instance will need to pull it from somewhere, meaning that you'll use N * the size of the data for N instances.

I'm surprised there is no mention of const vs. static whatsoever in this thread;

I think it is important to mention that the only memory that ends up in the static (read-only) memory of a binary is that which is bound to a static, or that which is potentially reachable at runtime (and then is either implicitly static-promoted or the const values get bundled in a hardcoded fashion within the code section)

  • (depending on how smart the compiler is, it may sometimes incorrectly assume the whole array of bytes is reachable at runtime, but in other cases not).
//! This file should only bundle the numbers `0` and `1` within
//! the binary:
const NUMBERS: [u64; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];

pub
fn get_0 ()
  -> u64
{
    NUMBERS[0] /* bundled within machine code */
}

pub
fn get_addr_of_1st ()
  -> &'static u64
{
    &NUMBERS[1] /* bundled within static storage */
}

is equivalent to:

pub
fn get_0 ()
  -> u64
{
    0 /* 0 in code */
}

static ONE: u64 = 1; /* 1 in static data */

pub
fn get_addr_of_1st ()
  -> &'static u64
{
    &ONE
}

/* `2, 3, …, 9` are discarded */

However, both the following programs may bundle the whole array in static memory:

static NUMBERS: [u64; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];

and

const NUMBERS: [u64; 10] = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];

pub
fn get_addr_of_ith (i: usize)
  -> &'static u64
{
    &NUMBERS[i] // runtime index may range over all the elements of `NUMBERS`, thus the whole array needs to be in memory.
}

The conclusion to all this is that if you use:

const BYTES: [u8; include_bytes!("…").len()] = *include_bytes!("…");

and

  • neither static BYTES: [u8; …] = include_bytes!("…");

  • nor const BYTES: &[u8] = include_bytes!("…");

then you will make it easier for the compiler not to bundle in the final binary the unused / unreachable parts of that included file.

3 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.