LilLil LogLog - I'm building a distributed binary log

I really wanted to avoid it, but I guess there's no way out of it now.

I really need a simple, scalable (up and down) content-oblivious distributed binary log. I could not find anything that would fit my needs exactly, and I find it a little surprising. So I started to build it myself.

Why do I need it? When building a distributed system it's often easiest to model everything in a event-driven way. Events happen, and get recorded, some other parts of the system follow events and do some actions and/or produce new events.

It is possible to build such a system on top of existing databases and solutions, but it always comes with plenty of overheads and pitfalls. And at the end of the day - an append only binary log is like the simplest, most straightforward data structure there can be, right? All I need is somewhere to store ordered list - how hard can it be? So maybe even I can get it done - first in a single-node version, and then just add Raft clustering.

Anyway - I don't have much yet, other than some design ideas and initial commit: GitHub - rustshop/loglog: Distributed, fault tolerant, strongly consistent, performant (event)log in Rust . If someone is interested or has any feedback, please let me know.

Linky no worky


Thanks. I had no idea it was private. :smiley:

May I suggest to change the LogLog part of the name to something else in order to avoid confusion? LogLog is a very specific probabilistic data structure (similar to a Bloom filter but for multiset cardinality estimation), and it's very different from a consistent distributed storage solution.

when I was writing a blockchain in OCaml, and needed a binary write-ahead log, I went and got the source to leveldb and mimicked its low-level record log. It's a pretty excellent design, with support built-in for recovering from damage and all sorts of other good things.

That's what I'd start with: a single-node log based on that. Then move on to distributed, etc, etc.

1 Like

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.