from Hacker News

Show HN: Sandglass – Distributed, scalable, persistent time-sorted message queue

by celrenheit on 12/11/17, 8:43 AM with 68 comments

• by menacingly on 12/11/17, 3:19 PM

  It triggers my eyebrow when a highly available message queue offers either exactly-once delivery or reliable time sorting.

  I'm not saying I wouldn't give this project a closer look, but I would much rather a product make it painfully obvious what compromises were made to offer its availability.

  My instinct is: either you need strict constraints on ordering and delivery, in which case you use rabbit, or you need at-least-once-no-matter-what semantics, in which case you use something else and make your app less fragile.

• by manigandham on 12/11/17, 12:09 PM

  Looks interesting, might be a good replacement for the more enterprise messages/queue systems that have all the typical ack/redelivery/scheduling features as seen here.

  It's worth mentioning the new Apache Pulsar messaging system which can replace Kafka with pub/sub and queueing semantics while providing better scalability and per-message acks, probably better suited to those who want a combined system.

• by adrinavarro on 12/11/17, 10:07 AM

  This looks promising.

  I'm currently dealing with a queuing-related issue.

  I have a series of tasks running across servers that consume from a queue and run a task.

  Often, these tasks die mid-execution (but can be resumed by any other server). So, the queue is a database, and the running tasks "touch" a timestamp in the database if they are still executing. When a database document hasn't been updated for a while, the "consumption query" makes it so that it is 'redelivered' to an available server listening to the "queue".

  Of course, this is subpar, but we haven't yet come across an elegant (and not too over-engineered) way to replace this.

• by eddd on 12/11/17, 11:07 AM

  These buzzword product descriptions are terrible. Documentation should say first, what does this piece of code do and when should I use it. The actual rationale for this project is buried down in the middle of the documentation with a sparse 3 sentences.

  > The first is to be able to track each message individually (i.e. not using a single commit offset) to make suitable for asynchronous tasks. > The second is the ability to schedule messages to be consumed in the future. This make it suitable for retries.

  That's a start - I'd love to see how did you solve the problem? How your solution compares to other similar products? Why should I care about stuff you are mentioning? It is not an issue of not mature product - I think one should start with defining exactly and precisely what is being solved here.

  When solving a technical problem you always have to tailor your solution to a specific set of requirements and it is never like: "distributed, horizontally scalable, persistent, time sorted message queue." So please stop using such buzzwords.

• by jwr on 12/11/17, 1:05 PM

  Aphyr's Jepsen or it didn't happen :-)

• by agnivade on 12/11/17, 12:12 PM

  Seems similar to faktory ? https://github.com/contribsys/faktory

• by nicois on 12/11/17, 10:54 AM

  I would think the imminent Redis streams data type would provide this better. It is battle-tested and allows great customisation to a range of use cases.

• by dvdplm on 12/11/17, 10:06 AM

  What does Sandglass use for persistence? Is it using something like Rocksdb under the hood or is the WAL and VL "homegrown"?

• by tmp123tmp123 on 12/11/17, 1:06 PM

  Nodejs sever? Segfault will cause data loss.

• by lloydatkinson on 12/11/17, 11:43 AM

  Why does everyone keep alluding to this replacing kafka? If anything this is more similar to RabbitMQ.

• by velodrome on 12/11/17, 10:53 AM

  How does this compare with NATS?

• by ninjamayo on 12/11/17, 9:22 AM

  Looking good. Question is how is it going to convince people to move over from Kafka.

• by yassinebenyahia on 12/11/17, 9:01 AM

  this is awesome, is it meant to replace kafka ?