from Hacker News

We achieved a 6-fold increase in Podman startup speed

by DrinkWater on 4/13/23, 6:31 AM with 189 comments

• by hghid on 4/13/23, 9:08 AM

  I'm all for improvements to pod startup times etc, but the general idea of putting more software into cars is not that appealing. I recently broke down in the highlands of Scotland in a fairly new car with the family - it was a horrible experience. It was made worse by the fact that there was nobody close that had a clue what to do with the car. The breakdown service arrived promptly, plugged the diagnostic tool into the car, proclaimed it broken, called a tow truck and left - two days later we arrived home. Had I been in a less complex car, a local garage could most likely have fixed the problem and sent us on our way. The sophistication and gadgets in modern cars are great until something goes wrong then they fail hard. Small local garages that used to be a life saver are next to useless now as they don't have the tools and knowledge to fix a mobile data centre.

• by rwmj on 4/13/23, 9:04 AM

  > If the backup camera or other sensors were to run as containers, we needed to improve the starting speed significantly.

  I think I see the problem already. Why does anyone think its a good idea to put everything in an embedded system into a container? Particularly as everything comes from a single vendor and so the usual argument about "but libraries are too hard!" doesn't apply.

• by com2kid on 4/13/23, 6:24 PM

  > we did was analyze what happens when Podman starts a container and why it takes so long. It turns out there was a lot of low-hanging fruit.

  I've done this analysis for lots of software before, Windows has a really nice tool called Process Monitor that I've used to find huge slow downs before. Point it at a process and it'll tell you the every bit of IO that the application does, and at that point you can just start digging and opening bugs.

  IMHO almost every piece of software of any significant size horribly misbehaves. Opening the same file again and again, writing out lots of tiny temp files, reading the same key from the registry (or some config file) over and over again , and worst of all, using fixed timeouts waiting for a task to complete instead of doing the work to make things properly event based.

  On that last note, timeouts instead of event callbacks (or some sort of signaling system) is the cause of so much slowness across our entire industry. If something you are doing on a computer takes a human noticable amount of time, and you aren't throwing around tens of GBs of data or waiting on a slow network, then you are flat out doing something wrong.

• by manojlds on 4/13/23, 10:19 AM

  Why does a car even need to run containers? It's a known hardware, so why containers? Feels like lazy engineering.

• by scientism on 4/13/23, 11:05 AM

  This should be part of the "How to Overengineer Anything and Everything" series.

• by kobalsky on 4/13/23, 7:30 PM

  podman rootless and startup speed was what lured me in, sadly after a couple of years I've switched back to docker, bit happily in rootless mode now too.

  podman works fine until it doesn't. My hypothesis is that it has some fundamental design philosophy that makes it brittle. Properly cleaning up doesn't exist in their vocabulary.

  For example, a cancelled download or image extraction can bring the whole thing down at the worst time, you have to hunt down the corrupted folder and remove so that anything works again.

  A failed compose startup can leave the network in a undefined state hard to diagnose and impossible to recover without wiping some folders within /run/user and killing some rogue processes manually.

  This is further cemented by the fact that a lot of minor issues are answered with: podman system reset, which reeks of rm -fr node_modules.

  docker was always a pleasure to work with, I still don't understand why I suffered with podman so long.

• by nickjj on 4/13/23, 12:55 PM

  I'm happy to see improvements like this.

  In 2018 I opened a github issue around container startup time[0] with Docker. A couple of things have changed since that issue but generally speaking we are talking about ~5s (containers) vs 150ms (no containers) to start a realistic process that depends on networking and other things you'd expect in a typical web app.

  [0]: https://github.com/moby/moby/issues/38077

• by anaganisk on 4/13/23, 12:50 PM

  Who said AI is killing dev jobs, now Devs have an alternative employment as a car mechanic. Drop kubernetes or VMs into the car and DevOps guys can also join. It would be so fun to hear "Umm your ingress seems to use older API, I have to update it for the gearbox to engage" and then see them run kubectl apply.

• by jensenbox on 4/13/23, 6:34 PM

  I am very likely one of only a few people but it really irritates me when the term "fold" is used when they really mean "times".

  Folding a piece of paper (just like binary numbering) 6 times will provide you with a stack of 64 sheets.

  They did not have a performance increase of 64 times.

  This is identical to the idea of stating "magnitude" as being the number of times based on 10.

  How wrong am I?

• by mkoubaa on 4/13/23, 11:06 AM

  All this because nobody wants to change the ELF loader to better isolate dependencies of binaries

• by mongol on 4/13/23, 11:00 AM

  I didn't know they used containers in cars. I guess it makes sense when you think about it but it always felt like more of an "enterprise" solution to me.

• by IceWreck on 4/13/23, 9:40 AM

  Why would you run podman inside a vehicle's computer. Cool nonetheless.

• by crest on 4/13/23, 8:11 AM

  Adding a `sleep $DELAY` to a startup script isn't something to brag about :-P. scnr

• by perlgeek on 4/13/23, 10:16 AM

  Do we know which cars specifically run their applications in containers?

• by shrubble on 4/13/23, 1:18 PM

  So it was poorly thought out or poorly designed all this time, then they fixed the obvious errors? That was my impression after reading the article...

• by hamdouni on 4/13/23, 11:42 AM

  Nice to see improvements in podman startup and how the team achieve it...

  Constraints (car env) drive creativity !

• by polskibus on 4/13/23, 9:14 PM

  Can podman be used instead of docker on Windows for minkube? Would it work faster than docker in such case?