from Hacker News

Redis streams as a pure data structure

by itamarhaber on 3/22/19, 3:13 PM with 53 comments

  • by apeace on 3/22/19, 5:54 PM

    I get that the tennis match use-case is meant to be trivial and an example, but I don't buy it.

    > Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value.

    I think the sorted set would be a much better choice, because then you could still insert items in the past, like when that admin remembers there was a tennis match last week he never recorded. Same goes for modifying past values, or deleting values. These operations are trivial using a sorted set & hash, not so using streams.

    I'm excited for streams and I'm glad Antirez is taking time to blog and evangelize, but this article didn't convince me there's a compelling use-case for streams aside from the Kafka-like use-case.

  • by drewda on 3/22/19, 10:00 PM

    Just like it's useful to have both SQLite and Postgres available for smaller and larger data projects (and Spatialite and PostGIS for smaller/larger geo-data projects), it could be great to have Redis and Kafka for smaller and larger pipeline projects.

    Does anyone have good patterns for joining across entries from two or more Redis streams? This is one of the most interesting aspects of Kafka/Flink/Spark/Storm/etc. Would be useful to be able to develop with streaming joins in Redis playgrounds.

  • by skybrian on 3/22/19, 5:51 PM

    This seems pretty simple when events are logged as they happen with little or no latency and you can let the stream set the timestamp. I wonder, though, about the case where events may be buffered, perhaps due to an unreliable network? The time that the event occurred might be significantly earlier than the time it's inserted, and furthermore events are arriving out of order. It seems like things get much more complicated?

    Let's say tennis games are recorded on a piece of paper and entered into the computer later. What is different?

  • by nicois on 3/24/19, 12:32 AM

    I threw together a few words here about how we are using Streams combined with Sorted Sets to "upgrade" legacy databases to streams of data. Not revolutionary, but it could be interesting to some people. I can write more, if there's any demand: http://nicois.github.io/posts/databases-to-streams/

  • by _pmf_ on 3/22/19, 3:44 PM

    I wish we could standardize on using Redis as general interprocess transactional memory. I could drop 95% of our application code for our Embedded Linux platform by using stock Redis and stock SQLite, but of course there are political obstacles.

  • by skrebbel on 3/22/19, 7:15 PM

    Did anyone yet use Redis streams to store actual logs? Like server logs, application logs, etc.

    I understand that Elasticsearch is a common place to put logs, also because I assume that searching through logs is a common use case, but I wonder whether Redis has particular benefits for this use case. The data structure seems particularly tailored to it (but not so much to searching I guess).

  • by _pgmf on 3/22/19, 5:50 PM

    Streams are kinda cool but they have a distinctly different feel than the other data-types in Redis. They've got this invisible statefulness. Last ids, consumer group state, etc. I've tried implementing a couple little things with streams, and it's not necessary to use the consumer group stuff or whatever of course. I wonder why streams weren't made using the modules API, though? They seem just weird/different enough to warrant exclusion from the core data types, in my thinking. Anyways, just reading the title referring to streams as "pure" made me go wtf? Because there's a lot of hidden state in there.

  • by reggieband on 3/22/19, 6:50 PM

    I wonder how this compares to streams in Kafka or Kinesis. One of the main advantages of redis is that I see it used in many cases as a replacement for memcache (just a key/value store for bytes/strings) so it already exists in many infrastructures.

  • by erulabs on 3/22/19, 11:08 PM

    Streams are great! I've written a small library for Node which attempts to wrap some of the complexity (particularly for handling multiple connections for UNBLOCK calls, etc).

    So far I haven't used it outside of hobby projects for webGL games and such, but it's worked brilliantly, and no Kafka required for hobby async-streaming infrastructure!

    Hopefully it's useful to someone out there! https://github.com/erulabs/redis-streams-aggregator