from Hacker News

Building a Distributed Log from Scratch: Data Replication

by bhattisatish on 12/28/17, 1:29 PM with 21 comments

  • by majidazimi on 12/28/17, 4:00 PM

    Regarding Kafka replication: What if the following scenario happens?

    1. Leader replicates to followers

    2. Followers get the messages and send ACK to leader

    3. Leader gets the confirmation from followers, increments high watermark (HW) and replies client that messages is commited.

    4. NOW: Leader fails before it could piggyback to followers that HW has been incremented.

    5. The question is: Since potential leader is not aware of the fact that HW has been incremented, it becomes the new leader and truncates the log to old HW, which means we have lost an entry in the log that has been confirmed by previous leader.

    As a result, client has been confirmed that the message has been successfully written, but it has been lost.

  • by yazaddaruvala on 12/28/17, 7:03 PM

    Are there any distributed logs which operate more as a framework than a service/platform?

    Imagine Kafka but I am able to run my processors within the same JVM. You might even call them servlets for Kafka.

  • by chrisweekly on 12/28/17, 2:33 PM

    Outstanding material. Bookmarked.

  • by jaequery on 12/28/17, 5:23 PM

    id like to hear if blockchain can be used here to simplify all of this, anyone?