Simplifying API design with lifetimes and I think I coded myself into a corner

help
1

The problem: When compiling my experimental project bbolt-nub I get this lifetime error:

error[E0207]: the lifetime parameter `'tx` is not constrained by the impl trait, self type, or predicates
   --> bbolt-nub/src/io/pages/mod.rs:131:6
    |
131 | impl<'tx, RD> LolCopiedIter for LazySlice<RD::PageData, RD>
    |      ^^^ unconstrained lifetime parameter

How I got here:
I'm recreating my bbolt-rs project to fix several deficiencies with the API design and faaaaar too much unsafe code. I also wanted to experiment with cleaning up the code as what I originally wrote is directly ported Go code. In its current state it is... not fun to work on. A few months back I began work on its replacement. I'm the approaching alpha stage.

Design goals:

  • Cleaner API
  • Support multiple backend for experimentation (Is File faster than MemMap? Where does io_uring fall?)
  • Support pluggable page types and search behaviours, like multithreading
  • Support direct key/value access (MemMap, Memory) via &'tx [u8] and lazy loading (File)
  • Keep backwards compatibility with Go code

In my original port I had to pollute types and traits with multiple lifetimes since I was using RefCell everywhere. It is not pretty. The experimental API design looks approximately like this.

An important design requirement is the ability, at compile time, to provide the user what type of keys/values will be returned from the database. Go compatibility requires the database return &'tx [u8] where 'tx is the lifetime of the Transaction. I also want a lazy loading slice where the API user gets a wrapper that loads bytes in as needed. Unfortunately the lazy loading aspect doesn't allow the user to view those keys/values a &'tx [u8], but I'm fine with that.

The experimental API does what I need by preventing keys/values from escaping the transaction boundary no matter the type. As an unexpected bonus I get the guarantee that all of the lazy loading resources are released once the user commits the transaction for free.

The 'tx lifetime is defined via the RwLockReadGuard (or RwLockWriteGuard) that locks the file access. I want all of the data read from the IO interface to have the 'tx lifetime applied to them. All sub slices of that data (lazy loaded or otherwise) should as well. I created a test IO API for what I think it should look like.

pub trait ReadPage<'tx>: Sized {
  type PageData: TxPage<'tx>;

  fn read_page(&self, disk_page_id: DiskPageId) -> crate::Result<Self::PageData, DiskReadError>;
}

pub struct PageReaderWrap<'tx, T, R> {
  reader: RwLockReadGuard<'tx, R>,
  translator: T,
}

impl<'tx, T, R> PageReaderWrap<'tx, T, R>
where
  R: ReadPage<'tx>,
{
  fn read(&self) -> crate::Result<R::PageData, DiskReadError> {
    self.reader.read_page(DiskPageId(0))
  }
}

#[derive(Debug, Copy, Clone)]
struct DummyReader;

impl<'tx> ReadPage<'tx> for DummyReader {
  type PageData = SharedBuffer;

  fn read_page(
    &self, disk_page_id: DiskPageId,
  ) -> error_stack::Result<Self::PageData, DiskReadError> {
    todo!()
  }
}

pub struct BaseWrapper<R: for<'tx> ReadPage<'tx>> {
  f: RwLock<R>,
}

impl<R: for<'tx> ReadPage<'tx>> BaseWrapper<R> {
  fn fork(&self) -> PageReaderWrap<u64, R> {
    PageReaderWrap {
      reader: self.f.read(),
      translator: 6u64,
    }
  }
}

fn t() {
  let t = BaseWrapper::<DummyReader> {
    f: RwLock::new(DummyReader),
  };
  let m = t.fork();
  assert_eq!(true, m.read().is_ok())
}

The problem:
I'm currently stuck where I need to have an Iterator<Item=u8> for any slice type. &'tx [u8], the Lazy loading slice, the sub slice to the lazy loading slice, etc... The implementing a trait to generically get Iterators<Item=u8> breaks on the lazy loading slice, LazySlice, implementation. The LazySlice is generic over the IO Read trait NonContigReader. The implementation breaks compilation because 'tx isn't bound to anything despite binding 'tx to everything I can think of.

I would be grateful for any suggestions on how to fix this issue! Also a design review would be great if anyone has suggestions there, too! I'm kindof making it up as I go :smiley:

Bonus points:
I'm not a super huge fan of how the LazyPage struct is defined. The current implementation is generic over the data type and the io reader type even though the io reader type defines the data type. I've found that I need to keep the traits bound by 'tx out of the structs or I start getting weird lifetime errors.

All of the pertinent IO page traits are here. The implementations are at

2

I think one solution might be to hold 'tx in every struct with PhantomData. It's doable, but I'd like to minimize how much generic gets used. I liked Trait<'tx> because I could keep both the API and the lifetimes out of the types.

3

You might be able to do something like...

pub trait LolCopiedIter<'a> {
  type CopiedIter: Iterator<Item = u8> + DoubleEndedIterator + ExactSizeIterator;
  fn lol_copied(&'a self) -> Self::CopiedIter;
}

Depending on how its used, you might hit some of the speed bumps described in this article.

1 Like
4

What I ended up doing was creating an explicit RefIntoCopiedIter that looks like

pub trait RefIntoCopiedIter {
  type Iter<'a>: Iterator<Item = u8> + DoubleEndedIterator + ExactSizeIterator + 'a
  where
    Self: 'a;
  fn ref_into_copied_iter<'a>(&'a self) -> Self::Iter<'a>;
}

That way no matter what type I'm working with I always work with a reference of the datatype.