For the following problem, is there a more light weight solution than "load into sqlite" ?
I have a bunch of structured/relational data. One file per table. Either CSV or some binary packed format.
I need to run READ ONLY queries over this data. No updates/inserts/dels/writes.
I am wondering if there is some library/crate that provides a solution that is more lightweight than sqlite.
Out of curiosity, why don't you want to use sqlite?
I'd say Sqlite is the lightweight solution here. As the creator of Sqlite said, it was explicitly designed for your task of working as a local store/cache so you can do some data processing, replacing the myriad of ad-hoc "pile of files" databases people might otherwise come up with.
I have previously used XML file in that situation. It had to be editable by hand and version nicely, so a file per entry was used (only ~100 entries total).
How about describing the rows as a struct and using serde?
You can (but bear with me) even use SQLite to query the CSV directly.
However, this feature is not in the amalgamation. I don't know about the Rust SQLite crate(s), but I know that the Go SQLite packages use the amalgamation. If this is true for Rust as well (it would probably make sense), then this might become complicated.
Is the data smaller than you RAM? Then load it in RAM, chuck it in a HashMap or whatever you need to query it.
In https://lib.rs server I'm using serde to load 3GB large messagepack blobs, and laughing about it.
This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.