Raft, updates through followers

I want to maintain a state say a Hashmap in a cluster computers; where computers are regularly being added and removed (crash) from the cluster.

I could use Raft, but one issue is that only a single node(leader) is active and all others just follow leader[First Image]. But I want followers to also update the state as they also receive requests from clients[Second Image].

Question: Could this problem be solved with Raft? Else what algorithms solve this problem? Any rust crates exists for this?

Thank you for help :slight_smile:

There are many different approaches you could take here. One which directly satisfies your requirements is to have followers (loosely) track which node is the leader, and have them forward requested transactions to the leader. Responses should then be synthesized by the node which received the request once the corresponding transaction is committed in its Raft log. Don't try to cheat here; I quickly found out working with Raft that "everything should go through the distributed log, and action only be taken once committed", if that makes sense. If you think cheating that rule will somehow work (except for the specific optimizations mentioned in the Raft thesis, like read-only transactions), I can almost guarantee you'll find some corner case which breaks under poor network conditions.

The reason I say "loosely track leadership" is because leadership is a fickle thing in Raft. While leadership is technically strongly defined for a given leadership term, the "current leadership term" can differ between nodes (due to network partitions, message reordering, etc.), there can be multiple nodes who concurrently (in terms of wall clock time) think they are currently leader, or nodes may never find out who was elected leader for a term before moving on to the next one. The way I like to think of it (and indeed this analogy is made in the original Paxos paper) is that leadership is a value only truly known to the Ether, or by the gods, or whatever.

In addition, followers which forward requests to suspected leaders should be ready to re-forward (or tell clients to re-submit) requests to a newly elected leader. This is because, if the suspected leader is not actually a leader any more by the time the request reaches it, it has no choice but to reject the request.

The Signal app's Secure Value Recovery system takes a slightly different approach, which is to offload the forwarding and leader tracking to dedicated frontend nodes, so that replicas don't have to run that complex code and also risk crashing due to large request queues and whatnot.

This is indeed one of the trickier parts of Raft, and often makes me wonder whether the strong leadership "simplification" is actually a simplification at all. Other paxos-derived protocols, like Egalitarian Paxos and SDPaxos allow any node to commit transactions, potentially avoiding these "transaction forwarding" issues, but don't have nice understandable whitepapers and communities around them, so are harder to implement.

P.S. you can find a factored-out version of Signal's Raft implementation here, which is focused on source readability/auditability/simplicity.

2 Likes

This topic was automatically closed 90 days after the last reply. We invite you to open a new topic if you have further questions or comments.