How to write software without self-referential structs?

help
1

I'm running into a conceptual block with Rust that I just can't see my way around. But since folks write significant software with Rust everyday, I feel like I must just be misunderstanding something fundamental.

I'm using a 3rd-party crate (in this case, https://crates.io/crates/glium glium), which defines some resources (framebuffers) which depend on other resource types (i.e. textures) by reference. In a trivial program, this is fine - I create all my textures in main(), and then declare my framebuffers inside {} (to introduce a new lexical scope and keep the borrow-checked happy).

The problem comes when I want to make a non-trivial program, and try to store those textures and framebuffers inside a struct. Since the framebuffer stores a reference to the textures, I can't store them in the same struct (Rust doesn't allow self-referential structures). I can store them in different structs, but I can't nest those two structs in any other type (for the same reason), so I'm left having to allocate all my types manually in the main function. This is a problem if need to allocate any structs dynamically.

Here's the (obviously non-working) example I'm trying to work out a way forward with:

extern crate glium;

use glium::backend::Facade;
use glium::texture::{MipmapsOption, Texture2d, DepthTexture2d, UncompressedFloatFormat, DepthFormat};
use glium::framebuffer::MultiOutputFrameBuffer;

pub struct Renderable<'a> {
    position: Texture2d,
    normal: Texture2d,
    depth: DepthTexture2d,
    framebuffer: MultiOutputFrameBuffer<'a>,
}

impl<'a> Renderable<'a> {
    pub fn new(display : &Facade, width : u32, height : u32) -> Self {
        let position = Texture2d::empty_with_format(display, UncompressedFloatFormat::F32F32F32F32, MipmapsOption::NoMipmap, width, height).unwrap();
        let normal = Texture2d::empty_with_format(display, UncompressedFloatFormat::F32F32F32F32, MipmapsOption::NoMipmap, width, height).unwrap();
        let depth = DepthTexture2d::empty_with_format(display, DepthFormat::F32, MipmapsOption::NoMipmap, width, height).unwrap();

        let outputs = &[("color", &position), ("normal", &normal)];

         Self{
            position,
            normal,
            depth,
            framebuffer: MultiOutputFrameBuffer::with_depth_buffer(display, outputs.iter().cloned(), &depth).unwrap(),
        }
    }
}

Any ideas as to how to encapsulate types that contain references? Any reading material you can point me at?

Much obliged,

  • Tristam
2 Likes
2

Yeah, in general, this is a big restriction of Rust at the moment. Rental crate can sometimes help, but its API is not lightweight.

Here's what the author of glium thinks about the issue: https://gist.github.com/tomaka/da8c374ce407e27d5dac.

4 Likes
3

Ok, I just solved it in a singularly unergonomic way with the owning_ref crate instead.

Any references to efforts to solve this at the language level? Some sort of 'self lifetime, or a concept of an offset reference?

4

Hi, you can not store a reference to something still being moved in stack-memory, like the textures in your new() function.

Easiest would be to declare the Renderable-struct using Box-ed members, moving them to heap-memory.
eg.

pub struct Renderable<'a> {
    position: Box<Texture2d>,
    normal: Box<Texture2d>,
    depth: Box<DepthTexture2d>,
    framebuffer: MultiOutputFrameBuffer<'a>,
}

and

let position: Box<Texture2d> = Box::new(Texture2d::empty_with_format...);

The other way might be to split into two functions, first place the texture in local memory and then create the Renderable-struct referencing them.

let (position, normal, depth) = create_textures(display, width, height); // no longer moved around

let rendarable = Renderable::new(&display, &position, &normal, &depth, width, height);

It might be necessary to declare that the lifetime of the referenced textures lasts at least as long than the newly-created Renderable-object, being 'a.

impl<'a> Renderable<'a> {
    pub fn new(display : &Facade, position: &'a Texture2d, normal: &'a Texture2d, depth: &'a DepthTexture2d, width : u32, height : u32) -> Self
3 Likes
5

This thread on the internals board has a lengthy discussion of some of the issues involved. The recent rust-lang AMA even has a mention of it.

Long story short is it's a tough problem with lots of subtle dangers to guard against. Speaking of which, you didn't say exactly what your solution was, but if you're using the OwningHandle type from the owning-ref crate, note that it is unsound in the general case. See this issue for details. This may or may not affect your particular use case, but it's something to be aware of.

1 Like
6

Yes, I was using OwningHandle. However, the ergonomics of the solution are driving me nuts - it requires multiply nesting private structs, which in turn bleeds into all the trait implementations for the type.

Of course, since we are already in the world of unsafety, it seems one can solve some of the ergonomic issues by embracing it wholeheartedly, and just wiring up the references from pointer dereferences in an unsafe block:

pub struct Renderable<'a> {
    position: Texture2d,
    normal: Texture2d,
    depth: DepthTexture2d,
    framebuffer: Option<MultiOutputFrameBuffer<'a>>,
}

impl<'a> Renderable<'a> {
    pub fn new(display : &Facade, width : u32, height : u32) -> Self {
        let mut r = Renderable {
            position: Texture2d::empty_with_format(display, UncompressedFloatFormat::F32F32F32F32, MipmapsOption::NoMipmap, width, height).unwrap(),
            normal: Texture2d::empty_with_format(display, UncompressedFloatFormat::F32F32F32F32, MipmapsOption::NoMipmap, width, height).unwrap(),
            depth: DepthTexture2d::empty_with_format(display, DepthFormat::F32, MipmapsOption::NoMipmap, width, height).unwrap(),
            framebuffer: None,
        };

        unsafe {
            let outputs = &[("position", &*(&r.position as *const Texture2d)), ("normal", &*(&r.normal as *const Texture2d))];

            r.framebuffer = Some(MultiOutputFrameBuffer::with_depth_buffer(display, outputs.iter().cloned(), &*(&r.depth as *const DepthTexture2d)).unwrap());
        }

        r
    }
}

Of course, I still need an Option wrapper in there to allow lazy initialisation of fields, since there doesn't appear to be a way to obtain the address of an instance in the middle of instance construction.

And I'm probably going to have to Box the Renderable to prevent it from being moved in memory and invalidating the pointers...

7

So if you can guarantee that the data that a field points to will always be valid, you can resort to using raw pointers the same as you would use them in C or C++, with all the unsafety-ness that follows. Until better solutions are created, this the path of least resistance. You'll be responsible for ensuring that the data the pointers point to is valid.


struct Structure {
    data: String,
    raw_view: *const str,
}

impl Structure {
    fn new(data: String) {
        let raw_view = &string as *const str;
        Structure { data, raw_view }
    }

    fn push_str(&mut self, string: &str) {
        self.data.push_str(string);
        self.raw_view = &self.data as *const str;
    }
}
1 Like
8

You should change your struct so that the FBO appears first. Drop order is now defined as being forward, so as it stands now your texture attachments will be dropped before the FBO. Even with that fix, there are many such potential issues when you just decide to go unsafe like this. I strongly encourage taking another look at rental, since it's designed to handle exactly this kind of scenario safely.

EDIT: Forgot to mention, another issue with this version is that the referenced fields will be moved when the struct is moved. This will invalidate any pointers to them held by MultiOutputFrameBuffer, leading to possible segfaults or worse. Rental also statically prevents this type of error.

Using rental, your example might look something like this:

#[macro_use]
extern crate rental;
extern crate glium;

use glium::*;
use glium::texture::*;
use glium::framebuffer::*;
use glium::backend::Facade;

pub struct Attachments {
    pub position: Texture2d,
    pub normal: Texture2d,
    pub depth: DepthTexture2d,
}

rental! {
    pub mod rentals {
        use super::*;

        #[rental]
        pub struct Renderable {
            attach: Box<Attachments>,
            framebuffer: MultiOutputFrameBuffer<'attach>,
        }
    }
}

pub struct Renderable (rentals::Renderable);

impl Renderable {
    pub fn new(display : &Facade, width : u32, height : u32) -> Self {
        Renderable(
            rentals::Renderable::new(
                Box::new(Attachments{
                    position: Texture2d::empty_with_format(display, UncompressedFloatFormat::F32F32F32F32, MipmapsOption::NoMipmap, width, height).unwrap(),
                    normal: Texture2d::empty_with_format(display, UncompressedFloatFormat::F32F32F32F32, MipmapsOption::NoMipmap, width, height).unwrap(),
                    depth: DepthTexture2d::empty_with_format(display, DepthFormat::F32, MipmapsOption::NoMipmap, width, height).unwrap(),
                }),
                |a| {
                    let outputs = &[("position", &a.position), ("normal", &a.normal)];

                    MultiOutputFrameBuffer::with_depth_buffer(display, outputs.iter().cloned(), &a.depth).unwrap()
                }
            )
        )
    }
}

Tested and this compiles fine, with no unsafe required.

3 Likes
9

That works great for the declaration, but how dow one actually use this struct? The suffix can be rented cleanly, but the prefix seems to be private and inaccessible.

Which seems a bit chicken-and-egg. If I didn't need to use the prefix ever again, we wouldn't be need be dealing with references in the first place.

1 Like
10

On rental structs that use shared borrows such as this, you can access all fields with the rent_all suite of methods. For mutable rentals, only the suffix is accessible to ensure soundness, but with shared rentals it's perfectly safe.

2 Likes
11

Gotcha. Yes, that works nicely. I'm still working out the details, but this is avoiding unsafe entirely, and it's not that bad on the ergonomic front.

1 Like
12

Just curiosity: why do you need to update the raw_view pointer after the push? push_str may change the original pointer?

13

Yes, String maintains allocations and re-allocations of the inner str that it owns. When that inner str has to grow, chances are that the pointer will no longer be valid.

2 Likes
14

Now, I'm not sure I fully understand the issue, but for having self-referential structs and such I think the correct way in Rust was to use RefCell, at least I have used that for Graphs, where the nodes can have pointers to other nodes and etc... Of course this highly depends on the specific scenario and how are you going to use those references in the first place.

15

Using refcounting works fine if you control all the types that you're using, but that isn't always the case. As rust's ecosystem grows, it will be more and more common that your types will include fields of types that you don't control, which means the API decisions they make can have far reaching and subtle consequences. Any struct that takes a lifetime parameter is instantly unusable in many scenarios where the crate author may have never intended that to be the case.

Now, we could always just ask upstream to provide refcounted APIs in addition to borrowing, but that seems to undermine one of the core features of rust, if it's necessary to forego borrowing and lifetimes for full generality. Addressing this at the language level will allow people to write APIs in a more natural way without having to worry about locking out certain usecases.

16

Yep. In general, I'd rather not ever produce self-referential structs in my own code, and there are plenty of ways to avoid them.

Unfortunately, taking any dependency which stores references in structures is liable to force you to deal with this situation. Most of the time I'd avoid those dependencies entirely, but Glium is a little too foundational to justify rolling my own replacement :slight_smile:

1 Like