from Hacker News

Jepsen: Bufstream 0.1

by aphyr on 11/12/24, 2:12 PM with 85 comments

  • by diggan on 11/12/24, 4:37 PM

    > While investigating issues like KAFKA-17754, we also encountered unseen writes in Kafka. Owing to time constraints we have not investigated this behavior, but unseen writes could be a sign of hanging transactions, stuck consumers, or even data loss. We are curious whether a delayed Produce message could slide into a future transaction, violating transactional guarantees. We also suspect that the Kafka Java Client may reuse a sequence number when a request times out, causing writes to be acknowledged but silently discarded. More Kafka testing is warranted.

    Seems like Jepsen should do another Kafka deep-dive. Last time was in 2013 (https://aphyr.com/posts/293-call-me-maybe-kafka, Kafka version 0.8 beta) and seems like they're on the verge of discovering a lot of issues in Kafka itself. Things like "causing writes to be acknowledged but silently discarded" sounds very scary.

  • by williamdclt on 11/12/24, 3:35 PM

    I'm very surprised by this:

    > [with the default enable.auto.commit=true] Kafka consumers may automatically mark offsets as committed, regardless of whether they have actually been processed by the application. This means that a consumer can poll a series of records, mark them as committed, then crash—effectively causing those records to be lost

    That's never been my understanding of auto-commit, that would be a crazy default wouldn't it?

    The docs say this:

    > when auto-commit is enabled, every time the poll method is called and data is fetched, the consumer is ready to automatically commit the offsets of messages that have been returned by the poll. If the processing of these messages is not completed before the next auto-commit interval, there’s a risk of losing the message’s progress if the consumer crashes or is otherwise restarted. In this case, when the consumer restarts, it will begin consuming from the last committed offset. When this happens, the last committed position can be as old as the auto-commit interval. Any messages that have arrived since the last commit are read again. If you want to reduce the window for duplicates, you can reduce the auto-commit interval

    I don't find it amazingly clear, but overall my understanding from this is that offsets are committed _only_ if the processing finishes. Tuning the auto-commit interval helps with duplicate processing, not with lost messages, as you'd expect for at-least-once processing.

  • by Kwpolska on 11/12/24, 6:31 PM

    I’m looking at the product page [0] and wondering how those two statements are compatible:

    > Bufstream runs fully within your AWS or GCP VPC, giving you complete control over your data, metadata, and uptime. Unlike the alternatives, Bufstream never phones home.

    > Bufstream pricing is simple: just $0.002 per uncompressed GiB written (about $2 per TiB). We don't charge any per-core, per-agent, or per-call fees.

    Surely they wouldn’t run their entire business on the honor system?

    [0] https://buf.build/product/bufstream

  • by refset on 11/12/24, 3:12 PM

    > The Kafka transaction protocol is fundamentally broken and must be revised.

    Ouch. Great investigation work and write-up, as ever!

  • by didip on 11/12/24, 4:43 PM

    Has Kyle reviewed NATS Jetstream? I wonder what he thinks of it.

  • by philprx on 11/12/24, 3:57 PM

    I didn't find the GitHub project for bufstream... Any clue?

  • by kiitos on 11/13/24, 4:24 AM

    Great work as always.

    After reading thru the relevant blog posts and docs, my understanding is that Kafka defines "exactly-once delivery" as a property of what they call a "read-process-write operation", where workers read-from topic 1, and write-to topic 2, where both topics are in the same logical Kafka system. Is that correct? If so, isn't that better described as a transaction?

  • by c2xlZXB5Cg1 on 11/12/24, 3:22 PM

    Not to be confused with https://www.warpstream.com/

  • by zbentley on 11/12/24, 11:22 PM

    Erratum:

    > Transactions may observe none, part, or all

    Should, I think, read:

    > Consumets may observe none, part, or all

  • by pmdulaney on 11/12/24, 2:58 PM

    What is this software used for? Instrumentation? Black boxes?

  • by Bnjoroge on 11/12/24, 5:52 PM

    has warpstream been reviewed?

  • by bobnamob on 11/12/24, 3:42 PM

    > We would like to combine Jepsen’s workload generation and history checking with Antithesis’ deterministic and replayable environment to make our tests more reproducible.

    For those unaware, Antithesis was founded by some of the folks who worked on FoundationDB - see https://youtu.be/4fFDFbi3toc?si=wY_mrD63fH2osiU- for some of their handiwork.

    A Jepsen + Antithesis team up is something the world needs right now, specifically on the back of the Horizon Post Office scandal.

    Thanks for all your work highlighting the importance of db safety Aphyr