from Hacker News

Ask HN: What stack do you use to track realtime events at scale?

by lgsilver on 6/8/15, 4:08 PM with 1 comments

We're about to embark on building our own realtime web events platform to track events on 20 large properties (~100k+ events/s). If you've done similar projects, what stack have you used? We're thinking of some combination of Logstash (piping logs in), Kinesis (organizing stream), and Storm (processing and aggregation), but need ideas :).

  • by eip on 6/8/15, 8:07 PM

    We use RabbitMQ and Storm. Rabbit is great but for our use case would just as easily use SQS or pretty much any other event bus.

    Storm is ok. It's not very DI friendly and the documentation is not the best. It only supports pull when for our use cases push would always be preferable.

    Twitter announced a replacement for Storm called Heron but no word on if/when it will be released. It's supposed to be much faster.

    Storm is nice in that it enforces patterns for message passing (spouts/bolts). So the code ends up being pretty clean and easy to understand.

    We use a small library on top of Storm to add Spring support.