from Hacker News

Show HN: Quickwit – OSS Alternative to Elasticsearch, Splunk, Datadog

by francoismassot on 1/7/24, 3:35 PM with 51 comments

Hi folks, Quickwit cofounder here.

We started Quickwit 3 years ago with a POC, "Searching the web for under $1000/month" (see HN discussions [0]), with the goal of making a robust OSS alternative to Elasticsearch / Splunk / Datadog.

We have reached a significant milestone with our latest release (0.7) [1], as we have witnessed users of the nightly version of Quickwit deploy clusters with hundreds of nodes, ingest hundreds of terabytes of data daily, and enjoy considerable cost savings.

To give you a concrete example, one company is ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit. They divided their compute costs by 5x and storage costs by 2x while increasing retention from 3 to 30 days. They also increased their durability, accuracy with exactly-once semantics thanks to the native Kafka support, and elasticity.

The 0.7 release also brings better integrations with the Observability ecosystem: improvements of the Elasticsearch-compatible API and better support of OpenTelemetry standards, Grafana, and Jaeger.

Of course, we still have a lot of work to be a fully-fledged observability engine, and we would love to get some feedback or suggestions.

To give you a glance at our 2024 roadmap, we planned to focus on Kibana/OpenDashboard integration, metrics support, and pipe-based query language.

[0] Searching the web for under $1000/month: https://news.ycombinator.com/item?id=27074481

[1] Release blog post: https://quickwit.io/blog/quickwit-0.7

[2] Open Source Repo: https://github.com/quickwit-oss/quickwit

[3] Home Page: https://quickwit.io

  • by dikei on 1/9/24, 5:03 AM

    > To give you a concrete example, one company is ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit. They divided their compute costs by 5x and storage costs by 2x while increasing retention from 3 to 30 days

    I guess that's to be expected. Almost anything is more storage-efficient than Elasticsearch, FTS is so expensive.

  • by figmert on 1/9/24, 3:11 AM

    Looking at the docker compose, this seems like a very complicated tool to run.

    You'd need Kafka, zookeeper, and Jaeger. All would need to be HA. Then also this service. Not mentioning postgres because in theory you can use aurora or the like.

    How quick have your current customers been able to get up and running so far? And how much maintainence have they needed?

  • by BrandoElFollito on 1/7/24, 9:03 PM

    What I am really missing in these (really nice) alternatives to ELK is Kibana and its Lens.

    I tried to tapping this with Grafana but never learned to have graphs as easily as with Kibana. Maybe I was not trying hard enough?

    Has anyone replaced Kibana with Grafana for non time based graphs?

  • by matesz on 1/10/24, 11:43 AM

    I am impressed with this work, especially given our current use of Loki with Prometheus and Grafana.

    Few questions come to mind. Firstly, is Quickwit compatible with any S3 compatible object storage, such as Cloudflare's R2? Are there particular considerations to keep in mind for this kind of setup? Secondly, do you see Quicwit being used for analytics, such as tracking daily visits or analyzing user retention?

    Your insights on these would be greatly appreciated.

  • by sidcool on 1/9/24, 8:19 AM

    When open source providers say 'alternative' to a commercial solution, do they consider the serious engineering needed to scale such systems? I mean I used SigNoz, and it's comparable to Datadog feature wise but nowhere close in performance and scalability.

  • by evanjrowley on 1/9/24, 2:47 AM

    I'm surprised no one has mentioned that it's mostly programmed in Rust.

  • by marsupialtail_2 on 1/9/24, 6:08 AM

    Glad this is getting some love. This is seriously good software. Have you guys supported generic substring search yet? I recall it was not supported as of a few months ago.

  • by jjtang1 on 1/9/24, 5:32 AM

    Congrats on the launch, we'll have to get you integrated with Rootly :). We can enable incident responders to fetch metrics while they respond to incidents in Slack!

  • by m1sta_ on 1/9/24, 10:39 AM

    How far away is the ES Query DSL compatibility?

  • by debarshri on 1/7/24, 4:29 PM

    I see that you persist the logs in cloud storage where are the indexes stored?

  • by wiradikusuma on 1/9/24, 8:13 AM

    How is it compared to Signoz, which is also open source?

  • by besus on 1/9/24, 12:34 PM

    That license is not FOSS. Users beware of the impending subscription costs to come.

  • by jcwayne on 1/7/24, 5:27 PM

    I find presenting this as an open source alternative to commercial solutions a little disingenuous when any commercial use of it also requires a paid license. Like many other cases it seems like the AGPL is functioning more as a trial license.