How to provide storage for borrowed structures?

So I have some code that looks like this (this is a stand-in for me using a library, so I have to use a lifetime parameter for my struct):

#[derive(Debug)]
struct HelpMe<'a> {
    data: &'a [u8],
}

#[derive(Debug)]
struct HelpMeHelper<'a> {
    helper: &'a HelpMe<'a>,
}

fn help_me_new<'a>(data: &'a [u8]) -> HelpMeHelper<'a> {
    let helper = HelpMe { data: &data };
    return HelpMeHelper { helper: &helper };
}

fn help_me() {
    let data: Vec<u8> = Vec::new();
    let helper = help_me_new(&data);
}

Compiling it gives me this error:

error[E0515]: cannot return value referencing local variable `helper`
   --> src/main.rs:237:12
    |
237 |     return HelpMeHelper { helper: &helper };
    |            ^^^^^^^^^^^^^^^^^^^^^^^-------^^
    |            |                      |
    |            |                      `helper` is borrowed here
    |            returns a value referencing data owned by the current function

Fair enough, I need to provide storage for my data. But how? My immediate attempt is to provide a mutable option:

#[derive(Debug)]
struct HelpMe<'a> {
    data: &'a [u8],
}

#[derive(Debug)]
struct HelpMeHelper<'a> {
    helper: &'a HelpMe<'a>,
}

fn help_me_new<'a>(data: &'a [u8], opt: &'a mut Option<HelpMe<'a>>) -> HelpMeHelper<'a> {
    let new = Some(HelpMe { data: &data });
    *opt = new;
    return HelpMeHelper { helper: &opt.as_ref().unwrap() };
}

fn print_help_me<'a>(help: &'a HelpMe<'a>) {
    println!("option is {:?}", help);
}

fn help_me() {
    let data: Vec<u8> = Vec::new();
    let mut option = None;
    let helper = help_me_new(&data, &mut option);
    print_help_me(option.as_ref().unwrap());
}

This gives the error:

error[E0502]: cannot borrow `option` as immutable because it is also borrowed as mutable                                     
   --> src/main.rs:249:19
    |
248 |     let helper = help_me_new(&data, &mut option);
    |                                     ----------- mutable borrow occurs here
249 |     print_help_me(option.as_ref().unwrap());
    |                   ^^^^^^
    |                   |
    |                   immutable borrow occurs here
    |                   mutable borrow later used here

I can simplify the problem down to this:

struct HelpMe<'a> {
}

fn help_me_new<'a>(_: &'a mut HelpMe<'a>) {
    todo!();
}

fn help_me_really<'a>(_: &'a HelpMe<'a>) {
    todo!();
}

fn help_me() {
    let mut help = HelpMe {};
    help_me_new(&mut help);
    help_me_really(&help);
}

In my mind this is because there's two lifetimes here: One for mut help and one for the immutable version, but I'm asking for an explicit lifetime. But my intention was to have HelpMe<'a> exist as long as <'a>, not be a specific single lifetime. I know there's subtyping and variance, but either way it's not happening here.


More broadly I'm also not too sure if this is even the right approach? I feel like maybe this is the case for Cell and friends?

Keep in mind that Rust lifetimes generally correspond to the duration of borrows, and not the liveness scope of variables per se. (In case that's a point of confusion.)

Here's a brief explanation of what's going on. A HelpMe<'a> cannot be used outside of 'a, and you're creating an exclusive reference (&mut) also of duration 'a. The inner lifetime is invariant (&'m mut T is invariant in T), so ultimately this means you can't use HelpMe<'a> directly ever again, because it's always exclusively borrowed.

Two lifetimes might fix things, but honestly I don't know that I understand your unminimized use case.

i think it simply doesn't make sense to create both HelpMe and HelpMeHelper in the same function since that makes so you have always a 1 to 1 relationship between istances of the two so it might as well be joined in the same struct
either have two functions so that you have (&Data)->HelpMe. and (&HelpMe)->HelpMeHelper.
it's also possible that you might do with one less layer of indirection and have HelpMeHelper contain HelpMe rather than a ref to it

if you want help_me_new to be maximally useful you could do


fn help_me_new<'helpme, 'data>(
    data: &'data [u8],
    opt: &'helpme mut Option<HelpMe<'data>>,
) -> HelpMeHelper<'storage>

as @quinedot said, &'a mut Foo<'a> is almost always wrong.

the HelpMeHelper must be bound by the lifetime 'helpme, because after 'helpme has expired the HelpMe referred by the HelpMeHelper may be destroyed, for example by calling option = None

To store data somewhere, not on the stack, use Box. Somewhere is also known as heap:

#[derive(Debug)]
struct HelpMeHelper<'a> {
    helper: Box<HelpMe<'a>>,
}

fn help_me_new<'a>(data: &'a [u8]) -> HelpMeHelper<'a> {
    let helper = HelpMe { data: &data };
    return HelpMeHelper { helper: Box::new(helper) };
}
#[derive(Debug)]
struct HelpMeHelper<'a> {
    helper: &'a HelpMe<'a>,
}

fn help_me_new<'a>(data: &'a [u8]) -> HelpMeHelper<'a> {
    let helper = HelpMe { data: &data };
    return HelpMeHelper { helper: &helper };
}

The &HelpMe contained in the return value from help_me_new must remain valid even after the function returns. Since the HelpMe that reference points to lives in a local variable that is destroyed when help_me_new returns, a reference to that value cannot escape the function body in any way. There are no clever tricks or lifetime/mutability changes that will allow a reference to a local variable to escape the function body - this is the compiler preventing you from creating a dangling pointer.

The most obvious practical fix, to my mind, is to have HelpMeHelper own its helper field, rather than borrowing it. For example, this succeeds:

#[derive(Debug)]
struct HelpMe<'a> {
    data: &'a [u8],
}

#[derive(Debug)]
struct HelpMeHelper<'a> {
    helper: HelpMe<'a>,
}

fn help_me_new<'a>(data: &'a [u8]) -> HelpMeHelper<'a> {
    let helper = HelpMe { data: &data };
    return HelpMeHelper { helper };
}

fn help_me() {
    let data: Vec<u8> = Vec::new();
    let helper = help_me_new(&data);
}

This does make HelpMeHelper somewhat redundant, but if you're adding additional fields to it that are not also part of HelpMe, then that's fine. Personally, this is the simplest option.

@mroth's suggestion to have HelpMeHelper indirectly own its HelpMe via a Box is also a good idea if you specifically need that indirection, for example if HelpMe is large and expensive to copy. You might use an Rc or Arc instead if it needs to be allocated on the heap and also shared between multiple owners.

I appreciate the responses. Let me add some more context. I am using the library skrifa to render a font:

    let run = glyph_run.run();
    let font = run.font();
    let font_size = run.font_size();
    let font_data = font.data.data();
    let font_index = font.index;

    let font_ref = skrifa::FontRef::from_index(font_data, font_index).unwrap();
    let font_outlines = font_ref.outline_glyphs();
    let normalized_coords = run
        .normalized_coords()
        .iter()
        .map(|coord| skrifa::instance::NormalizedCoord::from_bits(*coord))
        .collect::<Vec<_>>();
    let location_ref = skrifa::instance::LocationRef::new(&normalized_coords);
    let skrifa_size = skrifa::instance::Size::new(font_size);
    let hinting_options = skrifa::outline::HintingOptions::default();
    let hinting_instance = skrifa::outline::HintingInstance::new(
        &font_outlines,
        skrifa_size,
        location_ref,
        hinting_options,
    )
    .unwrap();
    let glyph_names = skrifa::GlyphNames::new(&font_ref);

I would like to store hinting_instance and font_outlines in a structure to cache this work. These data structures depend on each other and use borrows to refer to each other. The data structures are:

pub struct FontRef<'a> {
    data: FontData<'a>,
    pub table_directory: TableDirectory<'a>,
    /// The index of this font in a TrueType collection
    ttc_index: u32,
    /// Whether this font is a member of a TrueType collection.
    ///
    /// We use a bool rather than an Option to avoid bloating the struct
    /// size.
    in_ttc: bool,
    // Whether the table directory is sorted and thus we can use binary search for
    // finding table records. In principle, fonts are required to have a sorted
    // table directory, but certain fonts don't seem to follow that requirement.
    table_directory_sorted: bool,
}

#[derive(Debug, Clone)]
pub struct OutlineGlyphCollection<'a> {
    kind: OutlineCollectionKind<'a>,
}

It seems I can't put these in a struct because of the lifetime constraint.

HintingInstance contains no borrows, and shouldn't present problem (unless you're dropping/moving/modifying it while trying to keep a reference to it around, etc).

OutlineGlyphCollection<'a> keeps the font data borrowed (it ultimately contains a reference to the font bytes). So if that data is local to a function you're trying to return the collection from, it won't work. But if that data is stored in the caller, it can work, provided you get the lifetimes lined up right.


For example, consider this method you've called:

// note: `FontRef<'a>` is a newtype around `&'a [u8]`
impl<'a> MetadataProvider<'a> for FontRef<'a> {
    fn outline_glyphs<'s>(self: &'s FontRef<'a>) -> OutlineGlyphCollection<'a> {
        // ...
    }

The returned value keeps the [u8] bytes borrowed, but it doesn't keep the FontRef<'a> itself borrowed. That would happen if they returned OutlineGlyphCollection<'s>, but they don't -- they return OutlineGlyphCollection<'a>.

You can drop the FontRef<'a> and still use the returned value, because the OutlineGlyphCollection<'a> isn't borrowing the FontRef<'a>, it's borrowing the bytes from elsewhere (the same thing that the FontRef<'a> is borrowing).

Similar to the method, if you have a function where you pass in a borrow of the bytes &'a [u8], or a FontRef<'a>, you can return a OutlineGlyphCollection<'a> based on those as well.


Example.

#[derive(Debug)]
struct HelpMe<'a> {
    data: FontRef<'a>,
}

#[derive(Debug)]
struct HelpMeHelper<'a> {
    helper: OutlineGlyphCollection<'a>,
}

fn help_me_new<'a>(data: &'a [u8]) -> HelpMeHelper<'a> {
    let helper = HelpMe { data: FontRef::new(data) };
    return HelpMeHelper { helper: helper.data.outline_glyphs() };
}

The HelpMe and thus the FontRef drop at the end of help_me_new, but it still compiles because HelpMeHelper (and the OutlineGlyphCollection) are borrowing the bytes which are stored outside the function somewhere, not helper or helper.data.

Note that helper is OutlineGlphyCollection<'a>, not &'a OutlineGlyphCollection<'a> or such.

So my immediate attempt at that was this:

#[derive(Default)]
struct RenderedFont<'a> {
    font_data: Vec<u8>,
    normalized_coords: Vec<skrifa::instance::NormalizedCoord>,
    outline_glyphs: Option<skrifa::outline::OutlineGlyphCollection<'a>>,
    hinting_instance: Option<skrifa::outline::HintingInstance>,
}

fn render_font<'a, 'b>(run: &'a parley::Run<MyBrush>, rendered: &'b mut RenderedFont<'b>) {
    let font = run.font();
    let font_size = run.font_size();
    rendered.font_data = font.data.data().to_owned();
    let font_index = font.index;
    rendered.normalized_coords = run
        .normalized_coords()
        .iter()
        .map(|coord| skrifa::instance::NormalizedCoord::from_bits(*coord))
        .collect::<Vec<_>>();
    let font_ref = skrifa::FontRef::from_index(&rendered.font_data, font_index).unwrap(); // TOOD: unwrap
    let location_ref = skrifa::instance::LocationRef::new(&rendered.normalized_coords);
    let skrifa_size = skrifa::instance::Size::new(font_size);
    let hinting_options = skrifa::outline::HintingOptions::default();
    rendered.outline_glyphs = Some(font_ref.outline_glyphs());
    let hinting_instance = skrifa::outline::HintingInstance::new(
        rendered.outline_glyphs.as_ref().unwrap(),
        skrifa_size,
        location_ref,
        hinting_options,
    ).unwrap(); // TODO: unwrap
    rendered.hinting_instance = Some(hinting_instance);
}

This is called in this code:

fn render_glyph_run(glyph_run: &parley::GlyphRun<MyBrush>, _glyph_cache: &mut GlyphHashMap) {
    let mut rendered = RenderedFont::default();
    render_font(glyph_run.run(), &mut rendered);

    for glyph in glyph_run.positioned_glyphs() {
        // TODO: wrap this in a glyphcache struct?
        // TODO: separate out in to font caching too?
        let glyph = render_glyph(&rendered, &glyph);

It hits the same error due to RenderedFont being a mutable reference.

I guess I'm suppose to use multiple structs, one for each lifetime?

That's a self-referencial struct, where outline_glyphs is pointing to data owned by font_data. They're not natively supported by Rust in a useful way. You can't move something or get a fresh &mut to something that is borrowing itself (with references), for example.[1]

You could keep them in separate structs, or perhaps try a third-party crate that supports creating self-pointing types (ouroboros and yoke are two such crates).


  1. If you could, you could get a &mut [u8] to the contents of font_data, which is undefined behavior if something else is aliasing those bytes -- which the OutlineGlyphCollection would be. ↩︎

I see. I think the problem I've had in my head is that 'lifetime' here sounds to me as the same as the object's lifetime not a reference's lifetime. Thanks for the explainer. :slight_smile:

I agree that much confusion would be avoided if those 'a things were called something other than "lifetimes".

A few days later and I'm still struggling with this. I can have the caller provide data like this:

    let mut font_data = Vec::<u8>::new();
    let mut outline_glyphs = Option::<skrifa::outline::OutlineGlyphCollection>::None;
    let mut hinting_instance = Option::<skrifa::outline::HintingInstance>::None;
    render_font(glyph_run.run(), &mut font_data, &mut outline_glyphs, &mut hinting_instance);

    for glyph in glyph_run.positioned_glyphs() {
        let glyph = render_glyph(&glyph, &outline_glyphs.as_ref().unwrap(), &hinting_instance.as_ref().unwrap());

It seems like it's genuinely impossible to group or abstract away this data, the caller must always provide it and pass it to every function individually. I can't create an abstraction that holds the font data and cache data together. Is this right? Excluding crates that add self-referential data.

it is indeed true, as @quinedot said, that you cannot hold the owned daa font_data in the same struct as outline_glyphs or hinting_instance, because that would be self-referential.
but you can and maybe should hold the outline_glyphs and hinting_instance in the same struct, and you should certainly not use the weird Option system you have rn, because all it does is pollute your code with superfluous unwraps

you had something nice with render font going earlier but it seems you abandonned it for a worse design, so here it is fixed.

struct RenderedFont<'a> {
    font_data: &'a [u8], // is that useful ? maybe it should just be removed
    normalized_coords: Vec<skrifa::instance::NormalizedCoord>,
    outline_glyphs: skrifa::outline::OutlineGlyphCollection<'a>,
    hinting_instance: skrifa::outline::HintingInstance,
}

fn render_font<'a>(run: &'a parley::Run<MyBrush>) -> RenderedFont<'a> {
    let font = run.font();
    let font_size = run.font_size();
    let font_data = font.data.data();
    let font_index = font.index;
    let normalized_coords = run
        .normalized_coords()
        .iter()
        .map(|coord| skrifa::instance::NormalizedCoord::from_bits(*coord))
        .collect::<Vec<_>>();
    let font_ref = skrifa::FontRef::from_index(font_data, font_index).unwrap(); // TOOD: unwrap
    let location_ref = skrifa::instance::LocationRef::new(&normalized_coords);
    let skrifa_size = skrifa::instance::Size::new(font_size);
    let hinting_options = skrifa::outline::HintingOptions::default();
    let outline_glyphs = font_ref.outline_glyphs();
    let hinting_instance = skrifa::outline::HintingInstance::new(
        &outline_glyphs,
        skrifa_size,
        location_ref,
        hinting_options,
    ).unwrap(); // TODO: unwrap
    RenderedFont {
        font_data,
        normalized_coords,
        outline_glyphs,
        hinting_instance
    }
}

what you need to focus on is there is only one reference dependency going on here : OutlineGlyphCollection borrows from font_data, so they cannot be in the same struct in an owned way. that is all. there is nothing more. and because render_glyph_run takes a GlyphRun by reference, there is no problem.

i have not compiled my example so there may be 1 or 2 tiny mistakes, but looking at the documentation it looks all correct

Not in safe Rust as was already stated.

You would need to use unsafe… and creating sound API around self-referential structures is hard. It took more than a decade to create couple of crates that are currently believed to be sound.

That's why use of third-party crate is recommended.

To be clear, using yoke should be straightforward (if it’s sound for the struct to be self-referential, which sounds like it should be the case). You’d make a struct (say FontDataStuff<'a>) containing only the self references, apply derive(Yokeable) to that, and then your desired type is Yoke<FontDataStuff<'static>, Vec<u8>>.

In my case I am trying to outlive a glyph run, otherwise I have to re-manage a font each time I render new text or encounter new fonts in laid out text. So the font data has to be somewhere else, ideally in whatever abstraction I build that hides font rendering details.

if you don't want to depend on the Run reference
you will need something like this

fn render_font<'a>(run: &parley::Run<MyBrush>, storage : &'a mut Vec<u8>) -> RenderedFont<'a> {
    let font = run.font();
    let font_size = run.font_size();
    storage.clear();
    storage.extend_from_slice(font.data.data());
    let font_data = &*storage;

   /*
 ... same as above
*/
}

you could also use a "global" (not actually global but as global as you need it to be) FrozenVec from the elsa crate that contains all your fonts in one place

use elsa::vec::FrozenVec;

fn render_font<'a>(run: &parley::Run<MyBrush>, storage : &'a FrozenVec<Vec<u8>>) -> RenderedFont<'a> {
    let font = run.font();
    let font_size = run.font_size();
    let font_data = storage.push_get(font.data.data().to_owned());

   /*
 ... same as above
*/
}

Thanks everyone for the help! I really REALLY appreciate it. I did look at yoke last night and figure it was probably what I would want here, but I can't shake the feeling that this is not the right design decision to make.

Caching rendered glyphs is easy and I've already written code to do it, but I assumed that it would be better to cache the font loading code out of caution as the API has no performance notes or explanation of how long it should take. It seems like it's not worth the trouble. My interest in asking is more to learn about the options I have for situations like these. If it really becomes a performance bottleneck then I know what to do.

I know this thread is way past being over but I did want to ask: How do people generally deal with threading data through a Rust program? Because in my case say I have a FontRenderer that takes in a string and returns some useful data, with a mutable cache inside it. If I put this in some god object that means I now just have an exclusive mutable god object that I have to pass around everywhere, which doesn't seem ideal.

It seems like I have to instead write my programs instead in a functional style where I do passes over my data or have some event architecture or something like that to keep the nesting and passing functions through the call stack to the minimum. Is this right?

I started writing this before your last post. The top part you probably understand by now, the bottom part gets around to some of your questions.

Fundamentally, OutlineGlyphCollection<'a> cannot outlive whatever owns the font data (the references it has to that data would dangle). You can move that ownership out of Run<_> -- that's what @Morgane55440's last suggestion does -- but now it cannot outlive the new owner (the Vec<_>). Whether that is better, worse, or indifferent for consumers than borrowing from the Run<_> depends on other details.

The self-referencial supporting crates allow you to move the ownership into the RenderedFont you return. But Rust's native borrowing semantics to date do not support doing things like moving the owner of something borrowed. The crates work by unsafely erasing the borrowing relationships and adding indirection, among other tricks and precautions.[1]

If you don't want to go that route, and don't want to leak memory, and want to be able to pass the OutlineGlyphCollection<'_> up the stack from where you create it... then the function that creates the OutlineGlyphCollection<'a> has to take in something borrowed for 'a -- which means that uses of the OutlineGlyphCollection<'a> keep the borrow alive, which means the owner of the data cannot be destructed.

In which case...

...the only way you can have the consumer not participate in the borrowing is if it begins and ends within the API boundaries.


Note that the font crate you're using basically decided to involve you in the borrowing, at least in the case of these outline glyphs, and now you have the same decision to make about your consumers. One choice is to pass it on to your consumers (like the &'a mut Vec<u8>), but I'm mostly assuming you want to avoid that.

If upstream hadn't put you in this position -- if the outlines were your own type -- there would be more possibilities, like storing offsets instead of references to avoid borrowing (but probably introducing more checks and more possibilities for logic errors like using outlines from one font on a different font).

But in this particular position...

Self-referencial crates have been discussed. Leaking is another approach -- usually undesirable, but may be applicable for "always lives the entire runtime anyway" data.

Just not making unneeded optimizations is indeed another way to handle the issue.

Using callbacks is another approach to keeping the borrows within your API boundary.

impl SomeAbstraction {
    fn method<R, F>(&mut self, some: DataINeed, mut callback: F)
    where // Exact useful bounds depends on the use case
        F: FnMut(FontData<'_>),
    {
        // Set everything up and call the callback instead of returning
    }
}

One usually avoids having borrowing types in long-lived structures. That would mean not caching borrows. (When it happens, you often have a place where the owners are constructed, and everything else happens below that in the stack, with some ubiquitous lifetime being present.)

Mostly types own the data the manage, including any internal caching. Within their own APIs, the data is naturally available in their methods, or is typically passed around to others using references. If you need to, you might have an Arc<Mutex<_>> for shared caching, though now you've introduced the possibility of lock contention. If you have a truly global cache, you may use a LazyLock<Mutex<_>> or such instead, to avoid everyone having Arc<_> as a field or otherwise passing a handle everywhere.


  1. It's a very tough nut to crack soundly. ↩︎