Querying structured data on disk

For the following problem, is there a more light weight solution than "load into sqlite" ?

  1. I have a bunch of structured/relational data. One file per table. Either CSV or some binary packed format.

  2. I need to run READ ONLY queries over this data. No updates/inserts/dels/writes.

I am wondering if there is some library/crate that provides a solution that is more lightweight than sqlite.

Out of curiosity, why don't you want to use sqlite?

I'd say Sqlite is the lightweight solution here. As the creator of Sqlite said, it was explicitly designed for your task of working as a local store/cache so you can do some data processing, replacing the myriad of ad-hoc "pile of files" databases people might otherwise come up with.

3 Likes

I have previously used XML file in that situation. It had to be editable by hand and version nicely, so a file per entry was used (only ~100 entries total).

How about describing the rows as a struct and using serde?

  • csv looks thoroughly-documented.

  • csv-index is a bit less complete, but potentially useful. The xsv tool is built on top of this.

You can (but bear with me) even use SQLite to query the CSV directly.

However, this feature is not in the amalgamation. I don't know about the Rust SQLite crate(s), but I know that the Go SQLite packages use the amalgamation. If this is true for Rust as well (it would probably make sense), then this might become complicated.

1 Like

Is the data smaller than you RAM? Then load it in RAM, chuck it in a HashMap or whatever you need to query it.

In https://lib.rs server I'm using serde to load 3GB large messagepack blobs, and laughing about it.

3 Likes