from Hacker News

Show HN: LogTemplate analyse your logs with SQL

by CergyK on 11/23/21, 1:53 PM with 1 comments

• by CergyK on 11/23/21, 1:53 PM

  Hello HN, this a pet project I've been working on:

  It allows to visualize logs from raw files in a faceted manner in a sql database. It creates templates of recurring logs, extracting the 'moving' parts into columns values it stores in a column oriented DB For now it is possible to:

  - load a raw log file

  - provide a dissect pattern to extract meta fields such as the timestamp ('ts') and message ('msg')

  - Conduct analysis on the extracted data in grafana:

      - Aggregate (sum, avg, percentile) on metrics, group by facets (extracted strings)
  
      - Compute elapsed times between any given pair of events with sql (example to come)
  

  I came to work on this mostly after taking interest in templating algorithms and particularly drain: http://jiemingzhu.github.io/pub/pjhe_icws2017.pdf

  Let me know if you have remarks or if you think something like this would be useful to you!