from Hacker News

Systemd by Example (2021)

by emigre on 2/12/24, 11:48 AM with 111 comments

  • by doix on 2/15/24, 2:17 PM

    Ha, systemd is basically my git. I'm always surprised by how many developers don't grok git despite understanding significantly more complex things. I keep thinking if they would just take a day or two, they would get it and then wouldn't complain about having to copy paste commands and "rm -rf && git clone" to fix their workspaces.

    And yet I do the _exact_ same thing with systemd. I don't get it, I don't like it, and every time I need to do _anything_, I'm googling, copy pasting commands into my terminal until something vaguely correct happens. The CLI options are indecipherable to me, like reading hieroglyphics.

    Deep down I understand that it solves some pretty complex things when it comes to system initialization, but something about the UX of using journalctl and systemctl is so bad (for me) that it creates a mental block that I cannot overcome.

    Luckily, nothing I do in my day job requires messing with systemd.

  • by girvo on 2/15/24, 9:33 AM

    I was finally impressed with systemd when I recently had to port our single-process multi threaded TCP/UDP server (listening on a heap of ports) at work to a multiple binary/process, single threaded asynchronous IO system.

    PartOf= let us still have one top level “lithium” service to start and stop which starts and stops all of the other services, while being able to use journalctl to view them as one log file via a wildcard!

    Super nice, it’s been quite lovely.

  • by hiAndrewQuinn on 2/15/24, 8:14 AM

    Systemd is a pretty neat little thing. It's what keeps me running Ubuntu and Debian servers instead of switching to Alpine, at least in those situations where I can't containerize what I want to do.

    The thing that got me into it was systemd timers. I really like being able to split up what should run from when should it run. `cron` works best if you desire simplicity above all else, and Slack used `cron` for a very long time before moving to something more custom. But I ultimately don't feel like the learning curve for systemd timers is that bad, and the benefits it can bring (like writing all your echo statements to journalctl instead of some random hard coded logfile in a script) feel very worth it to me.

  • by dang on 2/15/24, 6:55 AM

    Related:

    Systemd by Example - https://news.ycombinator.com/item?id=36817510 - July 2023 (11 comments)

    Systemd by Example - https://news.ycombinator.com/item?id=30071240 - Jan 2022 (69 comments)

    Systemd by Example – The Playground - https://news.ycombinator.com/item?id=30070159 - Jan 2022 (1 comment)

  • by gradschool on 2/15/24, 11:19 AM

    I wish I could make friends with systemd like y'all. I spent at least a week wrestling unsuccessfully with systemd (admittedly starting from complete ignorance) to make a service that synchronizes a local directory on a laptop with a remote directory on a server after a docker container is stopped but before the filesystems and network stack are torn down during poweroff. It was easy enough to accomplish by manually invoking a script prior to powering off but no amount of cajoling, poring over documentation, or reading and adapting examples from forum posts on my part would get systemd to do it automatically without conflicts, race conditions, or timeouts. Is this not exactly the kind of problem systemd dependences are intended to solve?

  • by braincat31415 on 2/15/24, 3:06 PM

    From the personal experience, systemd works quite well when it works, but sometimes I ran into problems with it that were very hard for me to solve. I admit I never became a systemd expert, but I think I had a decent working knowledge of it.

    This was the final straw though: https://forums.debian.net/viewtopic.php?t=154562

    I am back to sysvinit and happy.

  • by Emigre_ on 2/15/24, 9:54 AM

    The blog post series author then did an online playground [1] that lets you create systemd examples and experiment with systemd in the browser. It's really interesting.

    [1] https://systemd-by-example.com/

  • by shdh on 2/15/24, 9:37 AM

    A lot of people still probably hate on systemd, but it and journalctl are pretty nice to me.

  • by smetj on 2/15/24, 6:39 AM

    This doc format works best for me ... just enough to get you started and get results. Digging deeper can be done where required. More like this!

  • by andrewstuart on 2/15/24, 10:52 AM

    Systemd is one of the things that is really enhanced by ChatGPT.

    “Write me a service to start this program”

    Or

    “Write a socket activated service to…..”

    Or

    “Write a systemd timer that runs every ten minutes”

    Etc etc

  • by wrs on 2/15/24, 5:13 PM

    I haven’t been in the weeds with systemd for a while, but this post was very helpful in understanding how systemd actually works, which didn’t match my (or, I would think, nearly anyone’s) intuition as a user of it. I don’t know if there’s any other full technical explanation of how units, jobs, transactions, etc. are actually handled.

    https://blog.darknedgy.net/technology/2020/05/02/0/index.htm...

    >systemd’s developers and almost all public documentation revolve around a “unit-centric” definition of systemd (except when discussing bugs), but my position is that for systemd-the-service-manager the most important construct is not the unit but the job, and hence one ought to understand systemd in a “job-centric” way.

  • by SuperNinKenDo on 2/17/24, 12:41 AM

    Took me a while to read through so I could come back and comment.

    I've got to say, this is probably the best single explainer of the basics of systemd I've read.

    I've only recently started fiddling with systemd extensively since I had my Linux-coming-of-age prior to its widespread adoption. I've got to be honest. I've found the experience very confusing and disorientating. The man pages feel really terse, even by manpage standards, and all the information is heavily split up so it's quite difficult to build a mental model of the full system IMO.

    Somethint like this would have been magic before, but it was still really helpful to me now with the way it explained things, I feel like it took me most of the way from cargo culting, into more understanding.

    Great write-up!

  • by js2 on 2/15/24, 6:40 PM

    > Now we need to tell systemd to execute halt.service when it reaches halt.target.

    The halt service, along with poweroff, reboot, and kexec get special treatment:

    https://www.freedesktop.org/software/systemd/man/latest/syst...

  • by andrewfromx on 2/15/24, 10:04 AM

    funny, to me systemd == no docker, no containers, just a VM.

    it's my goto way to keep my programming running and have it be restarted if the vm reboots. I use VMs like "pods". I deploy code directly to the VM and run it there along with other programs. I scale up and scale down with: https://www.pulumi.com/

  • by CyberEldrich on 2/15/24, 8:14 PM

    “Systemd .. reminds me of every overly convoluted script a junior coder has ever written”

    https://www.reddit.com/r/linux/comments/bsunzw/systemd_vs_in...

  • by znpy on 2/15/24, 10:43 AM

    Back in the day when LinuxAcademy was still cool, I asked for a systemd course... and they did it. Can't find the link right now, but that was what made systemd really click for me.

    Weird that still there is no apress/o'reilly/packt/manning introductory book about systemd.

  • by zokier on 2/15/24, 9:31 AM

    Personally I find systemd man pages pretty great, they comprehensively explain pretty much every thing that you need to know. Sure, they are more of a reference than a tutorial and there is lot of material to read there, but still I'd recommend reading the relevant sections whenever you are working with systemd

    https://www.freedesktop.org/software/systemd/man/latest/syst...

  • by kosolam on 2/15/24, 8:04 AM

    I like it

  • by traverseda on 2/15/24, 12:19 PM

    I am not having a good time with systemd on my embedded device.

    Somewhat complicated scenario, I need to change the mode of a wifi driver (which I do via modprobe), use iw to create a second interface for that wifi device, and finally use networkmanager to create a wifi hotspot based on the host name of the device. We set the hostname based on the mac address of wlan0.

    There are a lot of places this can go wrong, and you need a lot of deep knowledge of systemd to get it right.

    I start with three services, one to set the hostname, one to create the wlan0_ap device, and one to start up the wifi hotspot using nmcli.

    Alright, simple enough, this is the kind of thing systemd is good at, right?

    Well no, (and pardon me if my memory isn't perfect on this, it's tricky to debug weird race conditions) sometimes setting the hostname fails. Why? Well sometimes wlan0 doesn't exist, there's a udev rule that renames wlan0 during start up. Apparently I can add the systemd tag to that udev rules, and it will create a `.device` dependency I can Require. Alright, use both require and after keywords just to be sure. Good enough.

    Huh, onehsot services are listed as stopped even though they've run. Do some research, find out I need the `RemainAfterExit` keyword yet. Sure, dependencies are working.

    Now wlan0_ap device creation sometimes fails. Why? No really, why, this is hard to debug. Well looking around at dmesg it looks like the problem is that networkmanager is trying to do something with the device at the same time I am. Looks like it's enumerating device capabilities at the same time I'm trying to create a new device? Oh well, I'll just add the retry keyword to my service.

    Oh, wait, I can't add the retry keyword to one-shot services? Really? Internet tells me to change the service type to simple. Sure, we'll try that. Now that wlan0_ap device exists, let's start our access point.

    Also it's still sometimes failing on the hostname step still? The set-hostname command is asynchronous I think? I'll add a line to the start_access_point script that waits until the hostname isn't the default.

    Everything is looking good. Oh wait, it's sometimes failing intermittently. What the hell, why doesn't wlan0_ap exist? Service type `simple` considers a service to be started when the command first launches, not when it exit successfully. I've introduced another race condition.

    How can I make a service that works like a oneshot, but that you can retry? What, write a shell script and use systemd-notify to say it's complete?

    Long story short I ended up just writing a traditional shell-script based init script. Maybe I'm thinking about systemd wrong or something.