from Hacker News

The log/event processing pipeline you can't have (2019)

by lemonwaterlime on 9/18/24, 3:12 AM with 1 comments

• by lemonwaterlime on 9/18/24, 12:57 PM

  This outlines an end-to-end logging philosophy on a worst-case setup (all embedded devices and having to accept whatever logging message format outputs they happen to get from kernels, drivers, vendors, etc.).

  The idea is that the techniques that work for a worst case setup where you have the least control of log message formatting will work for most every other logging strategy (e.g. apps where the team completely controls the logs, don’t have to worry about kernel panics, etc.)

  For instance one could imagine instead of writing logs to the OS kernel as the essay suggests, that one instead writes to RAM or something like Redis. In fact, writing to RAM is a strategy that many teams have found success with.