Rust Borrow Checker: Why does reordering assertions affect mutable borrow lifetime from &mut dyn Trait in tests?

I have a struct in rust defined as below:

pub struct ObjectDictionary<'a> {
    entries: BTreeMap<u16, ObjectEntry>,
    storage: Option<&'a mut dyn ObjectDictionaryStorage>,
}

The ObjectDictionary has a storage attribute which I use to store data. ObjectDictionaryStorage is a trait because the storage could be of many different types.

pub trait ObjectDictionaryStorage {
    fn load(&mut self) -> Result<BTreeMap<(u16, u8), ObjectValue>, Error>;
    fn save(&mut self, parameters: &BTreeMap<(u16, u8), ObjectValue>) -> Result<(), Error>;
    fn clear(&mut self) -> Result<(), Error>;
    fn restore_defaults_requested(&self) -> bool;
}

Now, I am trying to run some tests and I created the following mock to emulate storage:

    struct MockStorage {
        saved_data: BTreeMap<(u16, u8), ObjectValue>,
        restore_requested: bool,
        save_called: bool,
        load_called: bool,
        clear_called: bool,
    }
    impl MockStorage {
        fn new() -> Self {
            // Not shown here for brevity.
        }
    }
    impl ObjectDictionaryStorage for MockStorage {
        fn load(&mut self) -> Result<BTreeMap<(u16, u8), ObjectValue>, Error> {
            // Not shown here for brevity.
        }
        fn save(
            &mut self,
            params: &BTreeMap<(u16, u8), ObjectValue>,
        ) -> Result<(), Error> {
            // Not shown here for brevity.
        }
        fn clear(&mut self) -> Result<(), Error> {
            self.clear_called = true;
            self.saved_data.clear();
            Ok(())
        }
        fn restore_defaults_requested(&self) -> bool {
            self.restore_requested
        }
        fn request_restore_defaults(&mut self) -> Result<(), Error> {
            self.restore_requested = true;
            Ok(())
        }
        fn clear_restore_defaults_flag(&mut self) -> Result<(), Error> {
            self.restore_requested = false;
            Ok(())
        }
    }

Now, during my tests, I would like to check if the restore_requested and clear_called flags have changed. I can easily check restore_requested because one of the traits' functions return it. But since none of the trait functions return clear_called, I am not being able to access it because the MockStorage is mutably borrowed to the ObjectDictionary.

See below an example of a test that would fail.

    #[test]
    fn test_init_restores_defaults_if_flagged() {
        let mut storage = MockStorage::new();
        storage
            .saved_data
            .insert((0x6000, 0), ObjectValue::Unsigned32(999));
        storage.restore_requested = true;

        let mut od = ObjectDictionary::new(Some(&mut storage));
        od.insert(
            0x6000,
            ObjectEntry {
                object: Object::Variable(ObjectValue::Unsigned32(0)),
                name: "StorableVar",
                access: AccessType::ReadWriteStore,
            },
        );

        od.init().unwrap();

        assert!(storage.clear_called); // Test fails here.

        assert!(!od
            .storage
            .as_ref()
            .unwrap()
            .restore_defaults_requested()); 

        assert_eq!(od.read_u32(0x6000, 0).unwrap(), 0); // Back to default
    }

The error message reads

cannot use storage.clear_called because it was mutably borrowed; use of borrowed storage.

Weirdly enough, I just realized that if I place assert!(storage.clear_called)'; at the end of the test as the last command, everything builds!

    #[test]
    fn test_init_restores_defaults_if_flagged() {
        let mut storage = MockStorage::new();
        storage
            .saved_data
            .insert((0x6000, 0), ObjectValue::Unsigned32(999));
        storage.restore_requested = true;

        let mut od = ObjectDictionary::new(Some(&mut storage));
        od.insert(
            0x6000,
            ObjectEntry {
                object: Object::Variable(ObjectValue::Unsigned32(0)),
                name: "StorableVar",
                access: AccessType::ReadWriteStore,
            },
        );

        od.init().unwrap();

        assert!(!od
            .storage
            .as_ref()
            .unwrap()
            .restore_defaults_requested()); 

        assert_eq!(od.read_u32(0x6000, 0).unwrap(), 0);

        assert!(storage.clear_called); // Test doesn't fail anymore!
    }

Can someone explain to me what is happening and what am I missing? Probably something about the assert! macro that I am not familiar with I guess...

The full error message will be longer and give more precise details as for what parts of the code are relevant. Feel free to run cargo test in a terminal in case you’re unsure how otherwise to get the full error message in your IDE setup.

2 Likes

The assert! macro isn't doing anything special here, it is just a matter of how you are using od and storage. When you create od, you pass in a &mut reference to storage. That means storage can no longer be used for as long as od is in use. In the example with a failing test, you access storage in an assert! and then have two more assertions referencing od (including one that uses the &mut reference in od.storage), indicating that you aren't actually done with od.

In the working example, storage isn't used from when od is created until after the last use of od, at which point the borrow of storage ends and the use of storage at the end is fine.

1 Like

Hi @steffahn , thanks for reaching out! Here it goes the full error message for your reference:

$ cargo test
   Compiling my-crate v0.1.0 (C:\Users\fabio\github\my-crate\crates\my-crate)
error[E0503]: cannot use `storage.clear_called` because it was mutably borrowed
   --> crates\my-crate\src\od\mod.rs:604:17
    |
592 |           let mut od = ObjectDictionary::new(Some(&mut storage));
    |                                                   ------------ `storage` is borrowed here
...
604 |           assert!(storage.clear_called); // Storage should have been cleared
    |                   ^^^^^^^^^^^^^^^^^^^^ use of borrowed `storage`
605 |
606 |           assert!(!od
    |  __________________-
607 | |             .storage
    | |____________________- borrow later used here

For more information about this error, try `rustc --explain E0503`.
error: could not compile `powerlink-rs` (lib test) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

Hello @jameseb7 , thanks for reaching out and for the insights. In my head I thought that both od and storage had the exact same lifetimes as they were all inside the same block of code and, therefore, od and storage wouldn't be dropped until the block was fully executed. But, from the looks of it, it seems od gets dropped and it frees storage before the end of the block of execution then, which is something new for me.

For my test purposes though, your insight helped me find a solution. I am now doing everything I need to do with od inside an inner block of code and only then do I run my check on the storage. This is working now:

#[test]
    fn test_init_restores_defaults_if_flagged() {
        let mut storage = MockStorage::new();
        storage
            .saved_data
            .insert((0x6000, 0), ObjectValue::Unsigned32(999));
        storage.restore_requested = true;

        {
            let mut od = ObjectDictionary::new(Some(&mut storage));
            od.insert(
                0x6000,
                ObjectEntry {
                    object: Object::Variable(ObjectValue::Unsigned32(0)),
                    name: "StorableVar",
                    access: AccessType::ReadWriteStore,
                },
            );

            od.init().unwrap();
            assert_eq!(od.read_u32(0x6000, 0).unwrap(), 0);
        }

        assert!(storage.clear_called);
        assert!(storage.restore_requested); 

    }

Thanks for replying. Meanwhile, @jameseb7 has already written an explanation, and I hope you can see the connection to the compiler message yourself: it points at the &mut storage borrow where od is created, and at the od.storage usage in the later assert, and notes how the other usage site of storage (through access to storage.clear_called in the earlier assert) is problematic.

It’s the basic property of &mut _ in Rust being an exclusive mutable borrow, meaning that no other access to a variable (like storage) is allowed while it’s mutably borrowed.

Now, figuring out the “duration” of such a borrow for determining what exactly the “while” in my previous sentence means can be a bit more tricky at times, but basically, such a duration generally refers to some section of your program code (not necessarily exactly a full block – it used to be a more simple model a long-ish time ago, but it’s more flexible with so-called “non-lexical” lifetimes nowadays) which is as short as possible while including all possible places where your borrow (i.e. the reference) is used. But references in Rust are first-class citicens and can be stored in data structures themselves (or even transformed e.g. with pointer arithmetic in low-level code) so the way the compiler is supposed to know what actually does or doesn’t constitute a “usage” of a reference is by looking at the lifetime 'a-style marks in type signatures.

Your ObjectDictionary::new function for instance — though you aren’t sharing it here – probably looks something like

impl<'a> ObjectDictionary<'a> {
    fn new(storage: Option<&'a mut dyn ObjectDictionaryStorage>) -> Self {…}
}

or perhaps similar (possibly also featuring lifetime elision), but in the end it’ll be written some way or another that encodes the property that the lifetime of the mutable reference you pass in is the same as the lifetime parameter 'a of the ObjectDictionary<'a>, like a free-standing function

fn new_dict<'a>(storage: Option<&'a mut …>) -> ObjectDictionary<'a> {…}

would. And to the compiler / borrow-checker this means that it’ll know to consider any usage of the resulting ObjectDictionary<'a> the same as any usage of the mutable borrow that was passed into the function.

Hence the claim in the error message that mutably storage.clear_called is borrowed in the

let mut od = ObjectDictionary::new(Some(&mut storage));

line[1], and that same borrow is later used in a line that accesses od (or more precisely od.storage).


  1. in fact the whole of storage is mutably borrowed, but the storage.clear_called is a part of it, and the borrow checker has a little bit of capability of differentiating between different fields of a struct for certain operations, the direct access to storage.clear_called in the earlier assert call would only need access to storage.clear_called and would be fine if only other parts of storage were mutably borrowed at that point ↩︎

2 Likes

Hi again @steffahn ! Thanks again for the clarification and the amazing comment. I spent the last few minutes reading about non-lexical lifetimes and now I understand why the second version of my test was compiling. I wasn't aware that lifetimes can be that "flexible". That's good to know that and I am happy to have learned something new regarding rust today. Thanks a lot once again for the lesson!

Another subtle point to be aware of is destructors (i.e. hand-written Drop implementations). Those count as access to a value, so if a type of yours (or one of its fields the type of which still contains the relevant lifetime annotation) has a custom Drop implementation, then scopes can become relevant once again, as implicit drops will happen at the end of a scope.

You can try it yourself. If you re-write the example that does work with

        assert_eq!(od.read_u32(0x6000, 0).unwrap(), 0);

        assert!(storage.clear_called); // Test doesn't fail anymore!

by adding a custom impl<'a> Drop for ObjectDictionary<'a> { fn drop(&mut self) {} } then the code will start to fail compilation once again. And as you have done above with an added block

        {
            let mut od = ObjectDictionary::new(Some(&mut storage));
            od.insert(…);

            od.init().unwrap();
            assert_eq!(od.read_u32(0x6000, 0).unwrap(), 0);
            // od implicitly dropped here
        }

or with an explicit call to something like drop(od); it would still work though.


Another point – not relevant here – to be aware of is that other generics can also indirectly contain lifetime marks at the type level. E.g. a container like Vec<T> can still hold a borrow (in a way the compiler understands) if you instantiate T accordingly, e.g. Vec<&'a mut i32>; and a generic function like fn make_vec<T>(x: T) -> Vec<T> { vec![x] }, if called with something like T = &'a mut i32 will still [as it should be to avoid any possibility for use-after-free] establish a connection between the borrow you pass in, and the resulting vector. Lifetime generics really are just a special case of general generics. Feel free to also check out this great introductory video demonstrating some examples like this:

2 Likes