from Hacker News

Show HN: An SQS Alternative on Postgres

by chuckhend on 5/9/24, 12:21 PM with 109 comments

  • by bastawhiz on 5/9/24, 3:00 PM

    > Guaranteed "exactly once" delivery of messages to a consumer within a visibility timeout

    That's not going to be true. It might be true when things are running well, but when it fails, it'll either be at most once or at least once. You don't build for the steady state, you build against the failure mode. That's an important deciding factor in whether you choose a system: you can accept duplicates gracefully or you can accept some amount of data loss.

    Without reviewing all of the code, it's not possible to say what this actually is, but since it seems like it's up to the implementor to set up replication, I suspect this is an at-most-once queue (if the client receives a response before the server has replicated the data and the server is destroyed, the data is lost). But depending on the diligence of the developer, it could be that this provides no real guarantees (0-N deliveries).

  • by pyuser583 on 5/9/24, 3:04 PM

    What advantages does this have over RabbitMQ?

    My experience is Postgres queuing makes sense you must extract or persist in the same Postgres instance.

    Otherwise, there’s no advantage over standard MQ systems.

    Is there something I don’t know.

  • by ltbarcly3 on 5/9/24, 4:03 PM

    Another one of these! It's interesting how many times this has been made and abandoned and made again.

    https://wiki.postgresql.org/wiki/PGQ_Tutorial https://github.com/florentx/pgqueue https://github.com/cirello-io/pgqueue

    Hundreds of them! We have a home grown one called PGQ at work here also.

    It's a good idea and easy to implement, but still valuable to have implemented already. Cool project.

  • by RedShift1 on 5/9/24, 7:14 PM

    This seems like a lot of fluff for basically SELECT ... FROM queue FOR UPDATE SKIP LOCKED? Why is is the extension needed when all it does is run some management type SQL?
  • by conroy on 5/9/24, 2:50 PM

    I'm curious how this performs compared to River https://riverqueue.com/ https://news.ycombinator.com/item?id=38349716
  • by kelnos on 5/10/24, 5:37 AM

    Perhaps we were all just not good at database'ing, but at a previous job, "using RDBMS as a queue" became a meme/shorthand for "terrible idea that needs to be stamped out immediately".

    Does Postgres have some features that make it not entirely unsuitable to use for queuing?

  • by dostoevsky013 on 5/9/24, 5:05 PM

    I’m not sure what are the benefits for the micro service architecture. Do you expect other services/domains to connect to your database to listen for events? How does it scale if you have several micro services that need to publish events?

    Or do you expect to a dedicated database to be maintained for this queue? Worth comparing it with other queue systems that persist messages and can help you to scale message processing like kafka with topic partitions.

    Found this article on how Revolut uses Postgres for events processing: https://medium.com/revolut/recording-more-events-but-where-w...

  • by anentropic on 5/10/24, 8:58 AM

    One of the appeals of doing MQ in Postgres is being able to submit events atomically in same db transaction as the stuff that raised the event

    Looking at https://github.com/tembo-io/pgmq/tree/main/tembo-pgmq-python ...how do I integrate the queue ops with my other db access code?

    Or is it better not to use the client lib in that scenario and use the SQL functions directly? https://github.com/tembo-io/pgmq?tab=readme-ov-file#send-two...

  • by cynicalsecurity on 5/9/24, 6:45 PM

    But why? Why not have a proper SQS service? What's the obsession with Postgres?
  • by lloydatkinson on 5/10/24, 9:01 AM

    For the longest time the common advice was that using a database as a message queue/broker was a bad idea. Now, everyone seems keen to use a DB for this instead of tools dedicated to this purpose. Why?
  • by pulkitsh1234 on 5/10/24, 10:09 AM

    A client is supposed to poll the queue for new items (i.e. issue pop requests in a loop), or is there some better event-oriented approach for this (via pg notify ?)
  • by thangngoc89 on 5/9/24, 3:00 PM

    I’m wondering if there are language agnostic queues where the queue consumers and publishers could be written in different languages?
  • by deepsun on 5/10/24, 3:53 AM

    > TIMESTAMP WITH TIME ZONE

    I'm yet to find a use case for "WITH TIME ZONE", in all cases it's better to use "WITHOUT TIME ZONE". All it does is displays the date in sql client local timezone, which should never matter for well done service. Would be glad to learn otherwise.

  • by bdcravens on 5/9/24, 5:58 PM

    Note there are a number of background job processors for specific languages/frameworks that use Postgresql as the broker. For example GoodJob and the upcoming SolidQueue in Ruby and Rails.
  • by rco8786 on 5/9/24, 3:27 PM

    This is neat. Would be cool if there was support for a dead letter or retry queue. The idea of deleting an event transactionally with the result of processing said event is pretty nice.
  • by justinclift on 5/10/24, 12:56 AM

    As a data point, there's a similar Go based project called Neoq:

    https://github.com/acaloiaro/neoq

  • by andrewstuart on 5/9/24, 7:45 PM

    Seems an unusual choice that this does not have an HTTP interface.

    HTTP is really the perfect client agnostic super simple way to interface with a message queue.

  • by airocker on 5/9/24, 5:23 PM

    I think it will be better if you create events automatically based on commit events in wal.
  • by ComputerGuru on 5/9/24, 9:54 PM

    This is strictly polling, no push or long poll support?
  • by junail on 5/10/24, 3:04 PM

    sounds interesting, is there a timeout? or batch processing?
  • by jilles on 5/9/24, 11:02 PM

    Can someone tell me what the usefulness of this is compared to RabbitMQ or Kafka?
  • by seveibar on 5/9/24, 3:11 PM

    People considering this project should also probably consider Graphile Worker[1] I've scaled Graphile Worker to 10m daily jobs just fine

    The behavior of this library is a bit different and in some ways a bit lower level. If you are using something like this, expect to get very intimate with it as you scale- a lot of times your custom workload would really benefit from a custom index and it's handy to understand how the underlying system works.

    [1] https://worker.graphile.org/