Both Rust crates & Algorithms / Theory papers welcome.
Imagine we have a cluster computers; where computers are regularly being added and removed (crash) from the cluster.
Suppose we care about two ops:
Set(u64, V) => ()
Get(u64) -> Option
When we have n machines, we want the space of u64 to be split over the n machines.
When a new machine is added, we want it to "steal portion of the keyspace" from existing machines.
When a machine crashes, we want to "redistributed key space (of crashed machine)" to the running machines. During the crash, we lose the V's, so we end up temporarily returning None's until a Set happens.
Question: what algorithms solve this problem? what Rust crates exist for this ?
I have no idea, my knowledge of distributed systems ends about where @ZiCog's does :) But you can almost certainly read and understand many of the academic papers about this problem without reading a whole textbook first, CS papers tend to be pretty friendly that way (much more so than e.g. pure math papers).
Two papers I would start with are Chord, which introduced the use of consistent hashing for distributed key-value storage, and Dynamo, a practical implementation of similar architectural ideas, used for some of Amazon's early distributed systems.