from Hacker News

ListenBrainz moves to TimescaleDB

by kingkool68 on 7/22/20, 10:10 AM with 96 comments

• by mfreed on 7/22/20, 2:22 PM

  Fun fact: TimescaleDB exists because we were using InfluxDB + Postgres for a previous IoT project and also found it unworkable (developer experience, query language, reliability, scalability and performance, operations, etc).

  We first built TimescaleDB as "Postgres for time-series" for our own needs and then decided to open-source it for others. :-)

• by 120bits on 7/22/20, 4:44 PM

  Interesting read and thanks for sharing.

  Not too long ago, I was asked to work on some analytics project and it required time-series data. I'm not a rockstar programmer and don't really know much about trends. So, I ended up googling and stumble upon InfluxDB. It felt like that right choice and I started playing with it. As the time passed, I realized that it might be a good software and I'm sure people love InfluxDB, but it wasn't the right choice for me. I didn't really like the docs, maybe its good now. And I had the same feeling about query syntax, it felt weird.

  I moved to TimescaleDB and never looked back. I have it production for almost 2 months now. 20 tables and over 100Million writes/week. One of things I really liked was staging, I don't use docker and or anything fancy. I have bash script that and it runs on centos box and all timescale extension and postgres database are packaged together.

  I was impressed by the timescale compression feature. I wasn't using it earlier because I had to be careful about what columns I need to segmentby. I would love to see some more features but I'm sure timescaledb team is already on it.

• by pgt on 7/22/20, 11:38 AM

  +1 on escaping measurement names. Quoting from their source code:

      def get_escaped_measurement_name(user_name): # ... comment omitted
          return '"\\"{}\\""'.format(user_name.replace('\\', '\\\\\\\\').replace('"', '\\"').replace('\n', '\\\\\\\\n'))

• by iliekcomputers on 7/22/20, 12:49 PM

  Hey! I've been working on ListenBrainz [0] for the past 3-ish years. Happy to answer questions if anyone has any.

  [0]: https://listenbrainz.org

• by RedShift1 on 7/22/20, 2:20 PM

  I'm in the process of moving from InfluxDB to TimescaleDB myself and can't wait to get rid of the hoops I have to jump through to get InfluxDB to answer some basic questions, mostly stemming from the fact that InfluxQL doesn't support boolean expressions. Something like 'SELECT MAX(temperature) > 10 FROM...' doesn't work.

• by akulkarni on 7/22/20, 1:48 PM

  (TimescaleDB co-founder). Thanks for the kind words! I feel especially proud about the first point "openness" - this is something we strive for both technically and culturally.

  For example, we have a pretty active Slack channel[0] where you can ask us anything. We've probably given away $$$$ of free support over the years ;-)

  [0] https://slack.timescale.com/

• by gregors on 7/22/20, 8:18 PM

  We too started off with influx but it wasn't a good fit mainly due to use having issues with high cardinality. I don't know if this is still the case with current implementations, but what it boils down to is if your data is searchable by a "user_id" really look elsewhere. That might be an oversimplification but that's the gist of it.

  I was fully ready to just roll my own partitioned table and gave TimescaleDB a shot. It worked well. There was a bug we ran into, but it was an existing one documented on github and was addressed pretty quickly.

  I still like influx, and would use it again but beware of the cardinality issues.

• by awinter-py on 7/22/20, 2:53 PM

  timescale is a postgres extension. 'postgres as a platform' is an interesting world to live in.

  postgres built-in RBAC is clunky or people would be relying on it, but an ecosystem of postgres plugins could include cleaner or smaller versions of this feature.

  Even things like complex migrations (github's gh-ost, for example) could exist as DB plugins.

• by decafninja on 7/22/20, 8:41 PM

  As someone who wants to pick up a time series DB to learn, what would be the best in terms of being the "industry standard"? InfluxDB? TimescaleDB?

  I'm familiar with some basics of kdb and use it often in my day job, but from what I understand that isn't widely used outside of finance?

• by osigurdson on 7/28/20, 5:10 AM

  In some cases it is difficult to define the table columns up front. Instead, a few tables: Object, Property, Time and Value (example below) are defined which make it possible to create new items on the fly. This works reasonably well up to a few billion records in the value table. However it does end up taking a lot of space (covering indexes/requisite memory are required for performance). It would be great to see a Postgres compatible solution that solves this problem in a more optimal way than a stock RDMS.

  Object objectId objectName other...

  Property propertyId objectId FK propertyName other...

  Time timeId time other..

  Value timeId FK propertyId FK value

• by silvester23 on 7/22/20, 12:29 PM

  > If you ever write bad data to a measurement in InfluxDB, there is no way to change it

  Correct me if I'm wrong, but I'm fairly certain you can just write data with the same timestamp again and it gets updated. Deleting is also easily possible.

• by rweichler on 7/22/20, 8:45 PM

  Figured I'd use this as an opportunity to plug my own service: https://eqe.fm

  Only works on jailbroken devices but it works well, has a local backup, and has been maintained (by me) for 2 years now.

  Server costs are $2.50/mo, so this will stay up as long as I am alive.

• by jbmsf on 7/22/20, 5:55 PM

  I'd love to hear more about how your data ingestion works. I'm thinking of implementing TimescaleDB myself, but in my initial read of the docs, the focus seemed to be managing the database, not getting data into the database...

• by iEchoic on 7/23/20, 1:03 PM

  Has anyone used Prometheus as well as TimescaleDB in production and have thoughts to share on those, comparatively?

• by thejosh on 7/22/20, 2:49 PM

  I really want to love timescaledb, it's great.. except for the minor issue of not being able to back up.

  https://github.com/timescale/timescaledb/issues/1835