Embeddable DB in Rust?

1

  1. This is a followup to "out of the tar pit" rust impl

  2. I am starting to believe that inside every interactive application is an ad-hoc, opaque, poorly-implemented half-working database for storing state. At the two extremees we have: 2a. state is centralized, well defined, in a db-like fashion and 2b. state is distributed, spread out, requiring lots of pointer chasing

  3. Unfortunately, it seems "embedded DB in Rust" doesn't have particularly nice libraries. By "embeddable" I mean running in same process, and not occuring the overhead of IPC. The choices seem to be sqlite3 (probably best so far, but a bit heavyweight, and I'd like access to the raw db in-memory representation of the data), mentat (no longer maintianed), sled (seems still in dev).

  4. Suggestions/recommendations? It seems either: 4a. I am wrong -- and most sufficiently complicated GUI apps don't need an "embeddable DB" for state management or 4b. there should be a nice library to make this happen.

Thanks!

2

If you're looking for a relational model built for GUI applications, then you're probably looking for an Entity-Component System. Well, they're closer to column stores than the typical row stores, but still. They're used pretty widely nowadays, popular enough that traditional OO people have started complaining about it.

SPECS, being a completely generic "data layer" rather than a full engine or GUI framework, is probably what you want.

2 Likes
3

@notriddle "Entity Component system / SPECS" looks very interesting. Thanks! (I'm not sure how I never heard of this until now.)

4

Off the top of my head:

LMDB

RocksDB

LevelDB

UnQLite

And a few pure Rust embedded DBs that I found with a few carefully chosen keywords. These are in various stages of completeness.

4 Likes
5

Probably because nobody's been talking about it except game developers and functional programming nerds.

There are a couple reasons why game developers (re)invented ECS. Reason 1 is that they need easy refactoring, and the relational model gives them that. Reason 2 is that they need good performance, and the column store model allows them to turn out code that has good locality and can easily be vectorized.

1 Like
6

@parasyte : Thanks! Your answer is a comprehensive answer to the question i actually asked.

@notriddle 's ECS / SPECS links answers the question I intended to ask.

7

@notriddle : Are there any "high performance game design patterns" book / resource list you would recommend?

I am not designing a game, however, I'm writing a high performance GUi app in OpenGL/WebGL, so there's probably a thing or two I can learn from the hardcore game dev community.

8

I already linked a bunch of them, but here's the specific, nice, long one:

http://t-machine.org/index.php/2007/09/03/entity-systems-are-the-future-of-mmog-development-part-1/

9

@notriddle : Are these the high level ideas ?

  1. Entity -- basically a u32.

For each ComponentType, we can have 0 or 1 of that ComponentType attached to the Entity.

  1. Component: basically a struct.

For the i-th ComponentType, we have some storage that does:
Entity -> Option<Component_i>

Here, the common storage type as BTreeMap, HashMap, DenseVec (two vec w/ indirection), SparseVec (allocates space for every entityt, whether the entity has given component or not).

Sorta looks like a "table" of a database.

  1. System: each system consists of:
    3a. a set of Components
    3b. some update function

When run, the System runs through all entities -- and if the entity has all the required components, then the update function runs on the entity.

There is some way to specify dependency / order among systems.

====

Is that the core principles of ECS systems ?

10

AFAIUI, yes.

  • World = Database/Table
  • Entity = Row
  • ComponentType = Column
  • Component = Cell
  • System = Query/Application
11

At first, I wasn't sure whether to think of a "ComponentType" as a Table or a Column.

However, given that we can not "decompose" a ComponentType into smaller components, I agree with your view of "ComponentType == sorta == Column".

The only thing "missing" would be an ability to:

easily serialize ECS to disk in a way where we can use something like SQL to query/manipulate the data while it's "dead / on disk"

12

This topic was automatically closed after 2 days. We invite you to open a new topic if you have further questions or comments.