Lifetime error with self-referential type (redb::WriteTransaction)

I've been working on using redb::WriteTransaction as a field of a struct and encountered a lifetime error that I don't understand and fail to reproduce by simplification. I thought it would be interesting to discuss the error and understand the underlying cause.

Context:
Definition of redb::WriteTransaction:

pub struct WriteTransaction<'db> {
    db: &'db Database,
    transaction_tracker: Arc<Mutex<TransactionTracker>>,
    mem: &'db TransactionalMemory,
    transaction_id: TransactionId,
    table_tree: RwLock<TableTree<'db>>,
    freed_tree: Mutex<BtreeMut<'db, FreedTableKey, FreedPageList<'static>>>,
    freed_pages: Arc<Mutex<Vec<PageNumber>>>,
    open_tables: Mutex<HashMap<String, &'static panic::Location<'static>>>,
    completed: bool,
    dirty: AtomicBool,
    durability: Durability,
    live_write_transaction: MutexGuard<'db, Option<TransactionId>>,
}

My custom struct:

struct UseTransaction<'db> {
    txn_write: Option<redb::WriteTransaction<'db>>,
}

impl<'db> UseTransaction<'db> {
    fn compute_some_thing(&'db mut self) {
        // ... do action on &'db mut self.
    }
}

#[test]
fn test_lifetime_with_redb_transaction() {
    let use_transaction = UseTransaction { txn_write: None };
    use_transaction.compute_some_thing();
}

The error:

error[E0597]: `use_transaction` does not live long enough
  --> tests/tests.rs:58:5
   |
58 |     use_transaction.compute_some_thing();
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ borrowed value does not live long enough
59 | }
   | -
   | |
   | `use_transaction` dropped here while still borrowed
   | borrow might be used here, when `use_transaction` is dropped and runs the destructor for type `UseTransaction<'_>`

I tried to reproduce the error using a simplified example, but the simplified code does not produce the same error, so I seems miss somethings:

#[derive(Debug)]
struct FakeDB(usize);

impl FakeDB {
    fn create_b(&mut self) -> FakeTransaction {
        FakeTransaction(&mut self.0)
    }
}

#[derive(Debug)]
struct FakeTransaction<'a>(&'a mut usize);

#[derive(Debug)]
struct FakeUseTransaction<'a> {
    b: FakeTransaction<'a>,
}

impl<'a> FakeUseTransaction<'a> {
   fn compute_some_thing(&'a self) {
       // ...
   }
}

#[test]
fn test_lifetime() {
    let mut a = FakeDB(1);
    let mut b = a.create_b();
    let c = FakeUseTransaction { b };
    c.compute_some_thing();
    println!("{:?}", c);
}

It seems that the issue might be related to the way redb::WriteTransaction handles lifetimes. I'm curious to know if anyone has thoughts on the cause of the error in my original code or suggestions on how to resolve it.

That's a &'db mut UserTransaction<'db>, which translates to "exclusively borrow the UseTransaction<'db> for the rest of it's validity ('db)". Once you do that, you can never use the UseTransaction<'db> again -- you can't call methods on it, you can't move it, and non-trivial destructors cannot be ran on it either.

You have a non-trivial destructor apparently, so that cannot compile.

Try fn compute_some_thing(&mut self).

Reproduction.

6 Likes

Thank you @quinedot , I understand now.

But how to deal if the method (e.g.: compute_some_thing(&mut self) "need" defined lifetime from the nested ref struct.

Full example here:

const TABLE: TableDefinition<&'static [u8], &'static [u8]> = TableDefinition::new("test_table");

struct UseTransaction<'db,'txn> {
    txn_write: Option<redb::WriteTransaction<'db>>,
    table: Option<Table<'db, 'txn, &'static [u8], &'static [u8]>>,
}

impl<'db, 'txn> UseTransaction<'db, 'txn> {
    fn compute_some_thing(&mut self) -> Result<(), redb::Error> {
        let txn_write = self.txn_write.as_mut().unwrap();
        let table = txn_write.open_table(TABLE)?;
        self.table = Some(table);
        Ok(())
    }
}

#[test]
fn test_lifetime_with_redb_transaction() {
    let mut use_transaction = UseTransaction {
        txn_write: None,
        table: None,
    };
    use_transaction.compute_some_thing();
}

Error:

error: lifetime may not live long enough
  --> tests/tests.rs:52:9
   |
48 | impl<'db, 'txn> UseTransaction<'db, 'txn> {
   |           ---- lifetime `'txn` defined here
49 |     fn compute_some_thing(&mut self) -> Result<(), redb::Error> {
   |                           - let's call the lifetime of this reference `'1`
...
52 |         self.table = Some(table);
   |         ^^^^^^^^^^ assignment requires that `'1` must outlive `'txn`
   |
   = note: requirement occurs because of the type `Table<'_, '_, &[u8], &[u8]>`, which makes the generic argument `'_` invariant
   = note: the struct `Table<'db, 'txn, K, V>` is invariant over the parameter `'db`
   = help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance

Resume:

How to deal with kind of situation? There is a way to define lifetime when we init the struct.

You are trying to create a self-referential type by storing a value (self.table = …) that ultimately borrows from self.

If you think about it logically, this is simply not possible. If a self-referential type were allowed, then moving it would invalidate the self-reference. No amount of lifetime annotations changes this simple fact.

You'll have to redesign your code so that you don't store table in the same type from which it borrows.


(To be 100% precise, using some pretty nasty lifetime tricks and shared mutability, it is technically possible to instantiate a self-referencing type. However, it's completely useless: the compiler won't let you drop or move it for the reasons explained above, so it's absolutely impossible to do anything useful with it, and it makes your code fail compilation anyway, just at a different point.)

2 Likes

You are trying to create a self-referential type by storing a value (self.table = …) that ultimately borrows from self.

@H2CO3 Thanks, I see.

You'll have to redesign your code so that you don't store table in the same type from which it borrows.

Here I try to redesign the code to put table in a different type.

const TABLE: TableDefinition<&'static [u8], &'static [u8]> = TableDefinition::new("test_table");

struct CacheTable<'db,'txn>(Option<Table<'db, 'txn, &'static [u8], &'static [u8]>>);

struct UseTransaction<'db> {
    txn_write: Option<redb::WriteTransaction<'db>>,
}

impl<'db> UseTransaction<'db> {
    fn compute_some_thing(&'db self, cache_table: &mut CacheTable<'db,'_>) -> Result<(), redb::Error> {
        let txn_write = self.txn_write.as_ref().unwrap();
        let table = txn_write.open_table(TABLE)?;
        cache_table.0 = Some(table);
        Ok(())
    }
}

But I fall in the same error than before:

error: lifetime may not live long enough
  --> tests/tests.rs:56:9
   |
53 |     fn compute_some_thing(&self, cache_table: &mut CacheTable<'db,'_>) -> Result<(), redb::Error> {
   |                           -      ----------- has type `&mut CacheTable<'_, '2>`
   |                           |
   |                           let's call the lifetime of this reference `'1`
...
56 |         cache_table.0 = Some(table);
   |         ^^^^^^^^^^^^^ assignment requires that `'1` must outlive `'2`
   |
   = note: requirement occurs because of the type `Table<'_, '_, &[u8], &[u8]>`, which makes the generic argument `'_` invariant
   = note: the struct `Table<'db, 'txn, K, V>` is invariant over the parameter `'db`
   = help: see <https://doc.rust-lang.org/nomicon/subtyping.html> for more information about variance

I need to add 'db lifetime to fn compute_some_thing(&'db self ... but I would find myself in the situation of the beginning.

Code/Error with 'db
const TABLE: TableDefinition<&'static [u8], &'static [u8]> = TableDefinition::new("test_table");

struct CacheTable<'db,'txn>(Option<Table<'db, 'txn, &'static [u8], &'static [u8]>>);

struct UseTransaction<'db> {
    txn_write: Option<redb::WriteTransaction<'db>>,
}

impl<'db> UseTransaction<'db> {
    fn compute_some_thing(&'db self, cache_table: &mut CacheTable<'db,'_>) -> Result<(), redb::Error> {
        let txn_write = self.txn_write.as_ref().unwrap();
        let table = txn_write.open_table(TABLE)?;
        cache_table.0 = Some(table);
        Ok(())
    }
}

Error:

error[E0597]: `use_transaction` does not live long enough
  --> tests/tests.rs:65:5
   |
65 |     use_transaction.compute_some_thing(&mut cache_table).unwrap();
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ borrowed value does not live long enough
66 | }
   | -
   | |
   | `use_transaction` dropped here while still borrowed
   | borrow might be used here, when `use_transaction` is dropped and runs the destructor for type `UseTransaction<'_>`

My goal is to be able to store Table somewhere, to reuse it as a reference as needed.

The table apparently borrows from the transaction; if that is the case, then the transaction needs to be kept alive as well while you want the table. I can't reproduce the error due to the lack of context and a full specification of dependencies and a self-contained example, but I strongly suspect that this is the problem.

1 Like

The table apparently borrows from the transaction;

@H2CO3 You're right, definition here transaction.open_table(&'txn self, ...).

I can't reproduce the error due to the lack of context and a full specification of dependencies, and a self-contained example,

My bad, but I'm stuck on it. I have tried to reproduce it in a playground by simulate:

Here code self-contained example: Rust Playground.

That compile and run with success :confused:

The two are not equivalent. In this Playground, compute_some_thing() takes &'db self, whereas in the version you claimed to have the error, it takes &self. And if you make this change in the playground, it fails to compile with the exact same error.

@H2CO3 Sure, in this comment I have mention of it:

...
I need to add 'db lifetime to fn compute_some_thing(&'db self ... but I would find myself in the situation of the beginning.

Code/Error with 'db :

const TABLE: TableDefinition<&'static [u8], &'static [u8]> = TableDefinition::new("test_table");

struct CacheTable<'db,'txn>(Option<Table<'db, 'txn, &'static [u8], &'static [u8]>>);

struct UseTransaction<'db> {
    txn_write: Option<redb::WriteTransaction<'db>>,
}

impl<'db> UseTransaction<'db> {
    fn compute_some_thing(&'db self, cache_table: &mut CacheTable<'db,'_>) -> Result<(), redb::Error> {
        let txn_write = self.txn_write.as_ref().unwrap();
        let table = txn_write.open_table(TABLE)?;
        cache_table.0 = Some(table);
        Ok(())
    }
}

Error:

error[E0597]: `use_transaction` does not live long enough
  --> tests/tests.rs:65:5
   |
65 |     use_transaction.compute_some_thing(&mut cache_table).unwrap();
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ borrowed value does not live long enough
66 | }
   | -
   | |
   | `use_transaction` dropped here while still borrowed
   | borrow might be used here, when `use_transaction` is dropped and runs the destructor for type `UseTransaction<'_>`

So in the playground with 'db. The compilation should fail with the above error use_transaction does not live long enough.

That's probably because Table has a Drop impl or drop glue.

1 Like

(I only skimmed most of this thread and then looked at the last playground link, but:)

If you change the declaration order, it compiles.

+    let use_transaction;
     let db_wrapper = DbWrapper(Database());
     let mut cache_table = CacheTable(None);
-    let use_transaction = db_wrapper.begin_write();
+    use_transaction = db_wrapper.begin_write();
     use_transaction.compute_some_thing(&mut cache_table);
2 Likes

@H2CO3 Thanks, my bad. I've already made this mistake... before.

So, if I understand correctly, there's no other way to separate transactions from tables.

    let (use_transaction, mut cache_table) = db_wrapper.begin_write();

Rust Playground :white_check_mark:

I tried to combine CacheTable and UseTransaction within a MetaUseTransaction but encountered the same problem.

struct MetaUseTransaction<'db, 'txn> {
    txn_write: Rc<RefCell<UseTransaction<'db>>>,
    cache_table: Rc<RefCell<CacheTable<'db, 'txn>>>,
}
...

Here Rust Playground :x:

Seeing this it seems that there is no way to be able to access within the same struct CacheTable (redb:Table) and UseTransaction (redb:WriteTransaction) this is a bit sad...


@quinedot Thanks to pointed it. It leads me to believe there could be an alternative approach contrary to my previous statement. The idea is to have a part of the struct with its own lifetime; is there a code design that supports this?

Note that interior mutability (RefCell here) doesn't help with self-referential types; interior mutability "only" allows you to mutate values through shared references. This doesn't change the fact that self-references would be invalidated after a move.

(It's pretty weird to see borrowed data, ie. data with a lifetime parameter, behind Rc, btw. Rc can't make borrowed data owned, if that's what you were thinking.)

@H2CO3 Thanks for pointing it out; you're right. Here is the same proposal without interior mutability Rust Playground :x:

Use #![deny(elided_lifetimes_in_paths)] and consider every lifetime you have to add.

// `self` is the only thing to borrow from, so not much to think about.
impl Database {
    pub fn begin_write(&self) -> WriteTransaction<'_> {
// Only the first parameter is actually a borrow of `self`,
// `'txn` can be unconstrained since you just assign `None`
impl DbWrapper {
    fn begin_write<'txn>(&self) -> MetaUseTransaction<'_, 'txn> {

And before fixing...

impl CacheTable<'_,'_> {
    fn open(&mut self, use_transaction: &UseTransaction) {

That's a lot of lifetimes, let's just give them all names first so we get better errors.

impl<'db, 'txn> CacheTable<'db, 'txn> {
    fn open<'s, 'b, 'u>(
        &'s mut self, 
        use_transaction: &'b UseTransaction<'u>
    ) {

Playground.

Ah yes, this makes sense, we need a &'txn UseTransaction<'db> since we're trying to assign the argument to a field with those lifetimes.

// Fixed
impl<'db, 'txn> CacheTable<'db, 'txn> {
    fn open(&mut self, use_transaction: &'txn UseTransaction<'db>) {

And now a new error pops up.

We're right back where this thread started: There's no safe way to create a self-referential struct such as this without exclusively borrowing yourself for the rest of your validity:

impl<'db, 'txn> MetaUseTransaction<'db, 'txn> {
    // That's a `&'txn mut MetaUseTransaction<'db, 'txn>`
    fn compute_some_thing(&'txn mut self) -> Result<(), ()> {

...and that won't work because you have a non-trivial destructor. (Even when you don't it's pretty much always too inflexible to be what you really want.)


The answer to "can I safely construct self-referential structs when there are non-trivial destructors involved" is no.

3 Likes

@quinedot Thank you for this detailed investigation, I learned a lot, how to debug lifetimes, etc ..

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.