by ilikepi on 2/17/24, 8:42 AM with 341 comments
by haunter on 2/17/24, 4:13 PM
And you can get them dirt cheap nowadays, has proper casing and cooling etc https://psref.lenovo.com/syspool/Sys/PDF/ThinkCentre/ThinkCe...
I have one right next to me, i5-8500T, 32GB RAM, 2x SSDs and currently 5W at idle with powertop auto-tune https://wiki.archlinux.org/title/powertop
by blacklion on 2/17/24, 9:51 AM
First advice must be to mount FS in read only mode, mount /var in memory and forward al logs to one nide, which may be not RPi but something with proper UPS and nut running. Power loss becomes absolutely bening if your FSes RO or temporary.
It is overkill if you have one RPi but author claims that he uses multiple RPis all around a house.
Also, good idea to have A/B system partitions and upgrade system with full partition rewrite and changing active one. Thus way your system will have one good system partition in any case, even if new version has fatal bugs, and recovery become trivial.
I'm using several small/single board PCs im different roles in such way for 20+ years with great success.
by hcfman on 2/17/24, 10:58 AM
Incidentally, that early commercial product was a home security product with a very small amount of home automation. I released this into open source with a new name in 2021 and now runs on the Jetson series SBCs (https://github.com/hcfman/sbts-install). Except then including high end YOLO models as triggers.
Because it was intended to be a standalone product it supported https with a GUI wrapper around all of the certificate operations. This still exists in my open source version, making it easy to use self signed certificates for intra-device rest calls.
But I've kept and expanded upon the multi-partition memory overlayFS approach and the installation of this system first asks you to install the sbts-base system, which installs the multi-partition memory overlayFS so that other's can use this as their own base systems.
by MPSimmons on 2/17/24, 11:12 AM
It's worth looking into for the cost savings on initial purchase and ongoing power draw.
by hn8305823 on 2/17/24, 3:25 PM
It's not a hack, it's best practice! Just like important servers in a data center should have some kind of out of band connectivity (IPMI, remote controllable RPDU outlets, etc), Important servers in remote difficult to reach locations should have some kind of watchdog script. The script should of course be tuned to the specific use case, considering the impact of a reboot vs downtime until reboot. At the very least it could log adverse events for later investigation.
A simple bash watchdog script was the very first thing I did when I deployed a remote RPI. Not just for wi-fi issues but for any of the dozens of things that could break and be fixed with a reboot.
by mvip on 2/17/24, 11:29 AM
by rcarmo on 2/17/24, 9:39 AM
by yjftsjthsd-h on 2/17/24, 9:21 AM
by lqet on 2/17/24, 10:47 AM
They both have SanDisk 2 GB cards in them. I vaguely remember naively thinking along the lines of "less space => less bit density => better reliability".
by 1vuio0pswjnm7 on 2/17/24, 7:50 PM
I just boot NetBSD kernel with embedded filesystem, e.g., INSTALL kernel or custom kernel. SDCard can be removed immediately after boot. Optionally chroot to attached storage. This runs for weeks, months or years. Have not experienced any of the issues cited by the blog author. Only issue I find is with the power connector when using a case; the connection can be brittle, e.g., if using a replacement cable. Perhaps this has improved on more recent Pis. (But I could say the same about most computers. The cables and connectors are usually fragile. It's always cheap stuff.) If power is interrupted because of movement, then the Pi reboots automatically.
by vegabook on 2/17/24, 9:30 AM
a) Cable ethernet
b) SSD (via USB3.0 adapter on my RPI4)
c) Ubuntu Server LTS 22.04.
d) cheap UPS.
Mine runs Yggdrasil network, HAproxy, Caddy server, a couple of webservers in containers, and a TMUX instance that I log into almost daily to write code (slow computer reveals bad code much better). Since I put it (and my router) on the UPS, in the last 2 years it has literally never gone offline other than a couple of times I rebooted it for firmware upgrades.by HankB99 on 2/17/24, 2:36 PM
The Zeroes run Raspbian configured with the read-only filesystem option. I have found it necessary to uninstall `unattended-upgrades` because the overlayfs employed for read-only root caches disk writes in RAM and the update/upgrade process exhausts RAM. For the same reason I disable swap. It makes no sense to swap to RAM on a 512GB system.
Upgrades are tedious since they require disabling overlayfs, rebooting, upgrading, rebooting, and enabling overlayfs. I wrote Ansible playbooks to perform these tasks. (https://github.com/HankB/Ansible/tree/main/Pi)
I have a Pi 4B performing as a file server and running Debian (not Raspbian) It boots from an SD card so that the entire HDDs can be used for a ZFS pool. To reduce wear and tear on the SD card I have mounted `/var` to a ZFS filesystem. I should probably use `tmpfs` for `/tmp`.
I use a Pi CM4 to run HomeAssistant and that boots and runs from an NVME SSD where durability is less an issue.
by zh3 on 2/17/24, 10:35 AM
by borlox on 2/17/24, 10:10 AM
They state the endurance on thousands of hours of FHD video, but what assumptions do they make in bitrate etc?
Can‘t they state total TB written or drive writes per day or something sensible?
by psanford on 2/17/24, 5:02 PM
But I also think that gokrazy's simplicity and design helps it be just a solid, reliable foundation to build on top of.
[0]: https://gokrazy.org/
by ChuckMcM on 2/17/24, 8:00 PM
I got there from having SD cards (nearly[2]) always be the failure point. Everything else has been pretty reliable when used within tolerances.
[1] It does "boot" from the SD card but that acts kind of like a third stage bootloader which loads and boots the "real" OS (FreeBSD) from the NAS device.
[2] I have had one fairly spectacular looking "melt down" of a no-name USB power supply wallwart PSU which, to appears to have also put something like 12V directly across the USB power pins (my best guess at what the secondary winding of the xformer in the wall wart was putting out on the 'low' side)
by sdflhasjd on 2/17/24, 10:32 AM
Also, creating a readonly root out of an existing disto is a bit of a pain, my preference is to use a distro (like TinyCore) that's already a readonly root.
by agilob on 2/17/24, 9:16 AM
by FerretFred on 2/17/24, 9:34 AM
by userbinator on 2/17/24, 10:31 AM
by cjdell on 2/17/24, 12:10 PM
Buildroot was surprisingly easy to use. Use a menuconfig to pick what you need and a burnable image for your SD card comes out the other side. Think I only spent an hour on the whole project.
by archerx on 2/17/24, 12:20 PM
I think the tinkerboards are better than Pi’s especially the ones that come with 16gb of onboard flash storage. However you don’t get all the niceness of PiOS but have to use TinkerOS (which was less than barebones when it came out) or Armbian which is nice but not built specifically for the tinkerboard.
I have a few friends who complained about Pi’s corrupting SD cards and it also happened to my only long running Pi so there is something going on.
by hcfman on 2/17/24, 9:41 AM
https://github.com/hcfman/sbts-aru
https://hackaday.com/2023/12/30/localizing-fireworks-launche...
With one command it for all Pi’s for both Raspbian and bookworm it:
* Shrinks the file system (Gee, how does it do that with just one disk ? ;-) )
* Creates new partitions
* Installs a memory overlayFS
* Installs and configures the system as an audio recorder with micro second time accuracy
* updates /etc/rc to do a forced repair of the data and config portions, in case they were damaged. This avoids system hangs waiting for human interaction with fsck
For the partitioning scheme it creates a swap partition, not as a wow but as an enabler if you really need it to install some large software.
It creates a small config partition. The idea here is that you keep it read only and remount it read write if you need to change config then remount it read only again.
And finally a data partition, which in this projects case is where the audio files are written.
I maintain a version of an overlayFS boot for the Pi but it needs revisiting for bookworm. The easiest way to use do this is to install the sbts-aru and then just don’t use it. Then everything is done for you in one command. And that version works for all Pi’s.
I also do this for the Jetson SBCs. But I need to revisit this for the Orin series. I have it working here for myself and friends but need to update the installer. Note, due to kernel behavior changes with Orin the older Pi like overlayFS code will not work. But I solved this and will release it when I release the Orin release of sbts-install soon.
I’ve been using memory overlayFS like installs for years for long running Pi systems.
by Namidairo on 2/17/24, 10:07 AM
While one could argue that you should figure out the source of your device freezing in the first place;
Nothing is better than having to ask someone to power cycle your Raspberry Pi while you're away.
by askvictor on 2/17/24, 11:30 AM
by WaitWaitWha on 2/17/24, 7:46 PM
I used to build my machines and loved it. Some are still chugging along after over 10 years. They were worthwhile before the mini PCs reaching comparable capabilities at lower cost.
One thing I still cannot seem to find is a good site that compares various vendors in similar fashion as "PC builder" sites do of build components.
Any suggestions?
by spintin on 2/17/24, 12:28 PM
A few weeks back the first SD card to fail got so corrupted it failed to reboot!
My key learning is use oversized cards, because then the bitcycling will wear slower!
I'm going from 32GB to 256/512/1024!
That said "High Endurance" cards are a scam, they fail way quicker than regular cards!
All SD cards except SanDisk have latency problems. There is no competition.
If you can get SLC SD cards use them for workload instances = no db or file storage.
by goodburb on 2/17/24, 9:25 AM
I'm guessing it's SLC/MLC.
Had a Transcend 32GB in 2016 die after a year.
The biggest issues with set-and-forget setups is software upgrade for security or other reasons, jumping major versions ends up breaking things. Compared to cloud which is (usually) regularly updated.
by teleforce on 2/17/24, 9:50 AM
Yes with the new PCIe expansion in the latest RPi 5 you can have external SSD for example, but if you decided to use it for other purposes as well like extra Ethernet port expansion then you cannot use it for booting anymore.
by thehias on 2/17/24, 2:02 PM
by cattlepi on 2/17/24, 6:33 PM
Been running pis (mostly 3b+) for eons with this solution and at this point i can say it's bulletproof.
The key is to minimize the sdcard wear and tear (it uses an overlay filesystem with squashfs as base and tmpfs as write top layer) and to keep zero stare on the device. You can build the image starting from normal raspbian. You can also update it over the network.
As far as usage in the wild the largest "deployment" (i know about) is at around 1000 pis.
by planb on 2/17/24, 10:52 AM
by demomode on 2/17/24, 9:18 AM
by tzs on 2/17/24, 3:13 PM
It is quite possible for updates to not break a running system but make it so that it will break on the next reboot. E.g., a dynamic library gets updated in a way that breaks a server process. It doesn't affect the running server because it still has the old library loaded.
Next time you boot your server process doesn't start.
These kind of problems can be annoying to deal with, especially when your system has an uptime of years and for all you know whatever change broke it could have been in any one of dozens of updates you've applied over that time.
by lxgr on 2/17/24, 2:48 PM
This makes me sad. I don't doubt that there are scenarios in which having IPv6 connectivity makes things worse, but these days, the opposite is more common, so I don't think "disable IPv6 just in case" is a good blanket recommendation to make anymore.
"If disabling IPv6 fixes your issue consistently, consider disabling it" would achieve the same outcome, without potentially causing problems/inefficiencies down the road.
by masto on 2/17/24, 3:27 PM
If there's a really really good reason, step 2 is to get rid of the SD card. Personally, none of mine even have an SD card inserted (ok, one does). I use network boot/NFS for everything. Some people attach other kinds of SSDs.
The one I lied about is a reverse telnet server that has been quietly doing its thing for 4 years without a hitch, apart from the time I had to replace the (PiJuice) backup battery because it was looking a little swole. But I should take the time to at least have a backup of the card ready to swap out when it fails.
by hcfman on 2/17/24, 10:50 AM
Also nice, is you can make your data or other partitions be encrypted. I've done this before. On the Pi 5 you can use the standard encryption as there's hardware support. On earlier Pi's you can use the encryption used for android. This does means there's a manual step in the startup for you to enter your encryption password.
by blackfawn on 2/17/24, 1:32 PM
Yet I've been feeding ADSB-Exchange and FlightAware from a Pi Zero for years and never had SD card problems.
I really like Alpine Linux for the Pi, running in it's own read-only mode where changes must be committed to disk. But unfortunately, Pi-hole isn't compatible with Alpine (at least last time I checked)
by vagab0nd on 2/17/24, 2:34 PM
EDIT: Also, make sure the power supply is sufficient. I was using a cheap adapter and had random errors and reboots.
by jnaina on 2/18/24, 3:11 AM
I use a USB network card, and a high-quality SD card. Other than that, no other special configs (except for another spare SD card with a full system image clone).
Rock solid performance and uptime over 8 years.
by guenthert on 2/17/24, 9:34 AM
by TacticalCoder on 2/17/24, 1:02 PM
I've been running RPi for years (including a VoIP server on a RPi 1!). The two tricks if you want a really long running Pi are: SD cards mounted read-only and ethernet.
FWIW I ve got a Pi running the unbound DNS resolver and it just works. It is not an art. There s a reason businesses are scooping up millions of Pi: they just work.
P.S: I ve got an army of NUCs too.
by jimmyswimmy on 2/18/24, 12:15 AM
by anonym29 on 2/17/24, 4:02 PM
I've had an RPi4 running continually for close to a year and never had a single network connectivity issue between the RPi4 and my router, connected via a 30m cat5 cable in my 75m² condo.
by akira2501 on 2/17/24, 9:31 AM
by tempaccount1234 on 2/17/24, 9:55 AM
The biggest problem is loss of wifi, after a few months one will lose wifi, but keep working - it’s constantly recording data so a reboot is not a good idea. I’d prefer a solution where I could just reset the wifi, but all attempts to script that reliably so far failed.
by nodesocket on 2/17/24, 5:32 PM
by rldjbpin on 2/19/24, 9:40 AM
for context, my workload was bursty in nature and nowadays it is mostly idle. but i never had any issues with the SD card, and i only upgraded to a new one for getting more capacity all this time (only twice).
keeping the dust out and managing the temperatures would go most of the way. i have served files directly from the sd card but it is always better to mount an external drive for this, while providing enough power to the board to power usb devices. limiting debug logs for stable applications can also help avoid write cycles, but using sd card on a pi has been a similar workload to using one in older smartphones for storing media.
by geon on 2/17/24, 8:12 PM
by yokoprime on 2/17/24, 10:19 AM
by Alifatisk on 2/17/24, 9:50 AM
by not_the_fda on 2/17/24, 1:35 PM
by niederman on 2/17/24, 9:28 AM
by csydas on 2/17/24, 10:38 AM
i don’t really get what to use a rpi for but i guess not important, it was just a nice series of articles
by weinzierl on 2/17/24, 2:03 PM
by tunnuz on 2/17/24, 11:13 AM
by RobotToaster on 2/17/24, 1:16 PM
by LeoPanthera on 2/17/24, 10:33 AM
by Temetra on 2/17/24, 10:07 AM
by tgsovlerkhgsel on 2/17/24, 3:11 PM
by jokoon on 2/17/24, 3:50 PM
Everytime there's a RPI I cry about it.
by guenthert on 2/17/24, 9:29 AM
by cjfd on 2/17/24, 3:35 PM
by Zopieux on 2/17/24, 6:50 PM
by vikmals on 2/17/24, 7:43 PM
Why is the author not considering using an SSD instead of an SD-Card here?
by pythonsnake on 2/17/24, 9:43 PM
Mouser is your friend.
by exfil on 2/17/24, 12:57 PM
by husam212 on 2/17/24, 8:39 PM
by asmor on 2/18/24, 10:35 AM
by Animats on 2/17/24, 11:24 PM
by greenie_beans on 2/17/24, 3:57 PM
by demondemidi on 2/17/24, 10:43 PM
What are you talking about?
It is literally zero effort. Just set up crontab reboots in case of power outage. That's it.
I've had a pi running as a BLE gateway / security cam for over 4 years with zero intervention.