Buffers: my crate for managing memory for containers

Hi everyone!

I made a crate named buffers (link to GitHub) and I'd love to have some feedback in the name itself, the idea, the project organization, and the actual implementation.

The main idea is to separate the memory layout and management into it's own interface (multiple traits) which enable the following with the same container:

  • Inline containers
  • Dynamic allocations (including using custom allocators)
  • Enable the optimizations and specific behaviour in a case-by-case basis (composite types)
  • Make a custom buffer for an specific behaviour or type
  • Using non-contiguous memory (eg. structure of arrays)

A buffer only manages the space itself and not the values, which is the container's responsability. For example: a vector knows that all positions from 0 to len have valid objects. Tracking that on the buffer would require a vector of bools (wasteful but maybe could be useful for testing). This separation of concerns turns out to make RawVec no longer useful for the vector's implementation and IMHO makes the container's code make more sense.

Usage examples:

use buffers::{
  base_buffers::{
    HeapBuffer,
    InlineBuffer,
  },
  composites::{
    ZstoBuffer,
    ExponentialGrowthBuffer,
    SvoBuffer,
  },
  collections::Vector,
};

// Inline vector of 128 u32
let inline_vector: Vector<u32, InlineBuffer<_, 128>> = Vector::new();

// Dynamically allocated vector with optimizations
type ExampleBuffer<T> = ZstoBuffer< // Optimization for types where T is a Zero-Sized Type
  ExponentialGrowthBuffer<          // Make the buffer to grow exponentially
    SvoBuffer<                      // Add Small Vector Optimization
      128,
      HeapBuffer<T>,                // Save the values on the heap (base buffer)
    >
  >
>;
let example_vector: Vector<u32, ExampleBuffer<_>> = Vector::new();

PS: buffers already exists on crates.io but it's empty. I'm waiting to be sure I like the name before asking for it.

I think you'll probably be interested in things like the Storage API, see https://rust-lang.zulipchat.com/#narrow/stream/219381-t-libs/topic/Recruiting.3A.20Storage.20API.20Project.20Group/near/287654002.

People are attempting to find a better abstraction than allocators to use in the standard library containers, for the same kind of "don't rewrite the container to change the storage" goal you mention here.

Thank you for the info! While I found some other disscussion on that direction I didn't realize that they were activelly trying to implement it in the standard library. Maybe the internals forum would be a better fit to discuss this.

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.