from Hacker News

How Copyover MUD Servers Worked

by _jackdk_ on 2/11/25, 1:19 AM with 82 comments

by ydant on 2/11/25, 2:50 AM
This is a fun blast from the past. One of my first real programming experiences was working as a (the) programmer on a MUD and I remember learning about this technique somewhere (I think just reading about how forking worked) and then figuring out how to implement it on our MUD.
It felt absolutely magical to be able to hot deploy changes without kicking everyone off of the server - but felt even more magical to have gotten it to work.
The MUD community was a fun early introduction to open source. People (many of them probably "kids" like I was at the time) sharing various patches and features. It felt so cool to release something and have other people use it and provide feedback. Like the author says - at some point the MUD itself became a lot less interesting than the programming.
by Erwin on 2/11/25, 12:08 PM
That was an amusing post to see pop up here, as I believe I came up with the "copyover" name when I copied Melvin Smith (aka "Fusion") idea about "hot reboot" from his "MUD++" code base to the popular Diku-based MERC/Envy etc. bases -- that was 2000 or probably earlier. Whether Melvin originally got the idea from somewhere else I don't know.
That version just used exec, and closed all files but network descriptors already logged in, the mapping of fds -> login names was saved in a file. When the new copy started up, it would log the users on existing file descriptors. Today, using explicit file descriptor passing (so you don't accidentally keep files open) or a long-running proxy would be preferable.
Back then C/C++ were often used by the developers, and we were at best CS students. There were surprisingly few segmentation faults, but I remember a few mysterious memory corruptions...
by jonmarkgo on 2/11/25, 2:29 AM
I actually run a MUD that stills uses this copyover method (primarily C code) for major code changes and migrations. We have about 100 players online at any given time! We've actually built in some more modern automatic copyover triggers and hooks related to recovering from crashes, capturing backtraces, performing database migrations, and sending/receiving systemd signals (among a lot of other modernization upgrades).
by mmastrac on 2/11/25, 1:25 AM
This is a pretty interesting approach to replacing your own executable, though it's really akin to a spicy spawn where you (the parent) exec rather than they (the child) exec. Not to minimize the coolness of this approach -- it's certainly a good way to do it, and doubly so on older Unix.
These days you could probably spawn your child process, test-boot it and then mmap bits of it back into your process and then unmap your old code pages once you're certain it boots. Or given how cheap cycles are, test-boot it and then throw that away and spawn the new executable if you're happy with the results.
by em-bee on 2/11/25, 4:08 AM
this is why languages such as LPC were developed. LPMuds allowed developers to code rooms and objects while the game is running and simply reload them whenever they wanted without admin intervention. if loading failed then the old version was kept and you'd just fix your code and try again. it was incredibly robust and powerful. for a player to get access to new versions of objects or rooms they had to drop the objects and pick them up again or reenter the room so that the object references could be updated to the new versions.
even better, LPC has been rewritten into pike, a general purpose programming language that retains the same capability. in the roxen webserver i can write and reload modules at runtime. this works efficiently because the lifetime of any object instance is limited to each http request. when a module is reloaded http requests already in progress are not affected, only new ones.
in the object storage server open-sTeam also written in pike a more advanced method was developed using proxy objects that can update object references and thus allow the updating of code without breaking the references. i am still using that to host my own websites.
to this day i have yet to find any other language with this power. smalltalk can do it and i believe lisp too, but that's it.
by borlak on 2/11/25, 7:45 AM
The way my mud did it, and I believe most muds (considering I took my implementation from another popular codebase), was not by forking. Sockets are just files, so all we did was save everything including players, execl() another instance of the MUD, and exit. Same port wasn't an issue with SO_REUSEADDR.
On boot as it's loading players, their file descriptor ID was saved, so it loads it up and resumes talking to the socket. It also loads the ID of the server socket descriptor. No need to send information to a new process as it just loads the previous game state.
If you're curious: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...
And: https://github.com/borlak/acmud/blob/7c2442dfccfc28364fc399a...
by empathy_m on 2/11/25, 7:19 PM
I spent a fair chunk of the mid-90s to early 2000s on a moderately busy MUD with a group of people who, it turns out, were all about the same age. No idea in retrospect how I juggled this with school/friends/work - I guess kids just have a ton of free time.
Kind of drifted away for a couple decades during college and after, as other things filed up the time.
I came back decades later, after going to a memorial service for a friend who died untimely of a serious medical condition, and seeing that a bunch of the people there were from her online community. They talked about the MUSH she hung out on had been a real lifeline when she was bedbound for immune-system reasons -- it was really, really cool to see, and I went back and checked out my own place and met up with folks again.
During 2020 a TON of people all had the same idea and all logged in to my place again. There was a brief resurgence of activity (from dozens of people online to a hundred+). Very few new players, but very cool to see people who were all, more or less, the same cohort -- just grown up now. Folks have slowly drifted away again in the past couple years, and that's fine too. I'm glad it's there.
It's nice to have these subcritical, human-scale online communities. Not everything has to be a subreddit or even a 10,000+ person discord - you can just hang out on a server!
by AStonesThrow on 2/11/25, 4:43 AM
The MUDLine: timeline of MUD history, by Lauren P. Burka (1995)
https://www.linnaean.org/~lpb/muddex/mudline.html
See also: Burka's MUDDEX (1993), curating listings of the erstwhile MUD servers as NSFnet ruled the USA
https://www.linnaean.org/~lpb/muddex/index.html
Burka was known to her fellow players as "ashne" - stylized in lowercase - on Tinymud Classic & Islandia, hosted by CMU prof Jim "Fuzzy" Aspnes, among other servers
https://en.wikipedia.org/wiki/James_Aspnes
several interesting hacks we collaborated on:
A TCP port concentrator, implemented by "leet". This add-on front end multiplexed connections with separate processes, allowing us to overcome the 64 file descriptor hard limit, per process, on Unix. Nearly 256 players could participate!
Na Choon Piaw, while working for Bell Labs Research on the East Coast, added a programming language that resembles Forth, releasing TinyMUCK 1.x and TritonMUCK test bed server. Piaw worked with a VAX server, 7800 I believe.
https://en.wikipedia.org/wiki/TinyMUCK?wprov=sfla1
also, Jon "Stinglai" Blow from Berkeley designed a string interning method for TinyMUCK 2.x that saved plenty of memory, by deduplicating ASCII strings in memory.
Me? I assumed many interesting names over the years. Primarily "ChupChup" or his plural counterparts, "chupchups", or also "Lucretia", the statuesque goth woman wearing "boots of the deepest black". But in Real Life, (RL), I remain Robert Earl.
by jisnsm on 2/11/25, 2:11 AM
Irssi has a command that does what I assume is something similar to this to upgrade in place without disconnecting https://irssi.org/documentation/help/upgrade/
by skulk on 2/11/25, 2:16 AM
No need for past tense! The most active MUD today (AFAIK, 250 logged in at most times) uses a version of this to automatically reboot every 2 days.
by cdr on 2/11/25, 3:00 AM
I played on a fantastic MUD as a kid. It was based on the code for some other MUD, and the admins were unhappy about some things, so they decided to write their own MUD engine from scratch. They got as far as establishing a network connection to a client - all network code written in scratch from C - before they burned themselves out and quit, which pretty much killed the MUD too.
by jeffrallen on 2/11/25, 7:24 AM
It is also fun to engineer servers which can be upgraded without ever stopping listening, even for a nanosecond on the listen socket. Would be a fun senior C programming job interview question, but unfortunately those don't exist anymore. Only kubernetes pod wranglers do, where the answer is "destroy it all and create it all and hope that the ingress somehow hides the mess".
Sigh.
by 7373737373 on 2/11/25, 9:32 AM
https://en.wikipedia.org/wiki/MOO (as for example used in https://en.wikipedia.org/wiki/LambdaMOO) allowed users to modify the world too, but only those who were given the wizard bit.
A big challenge lies in designing a system where every user has creation and modification rights, without breaking the system. Systems based on the https://en.wikipedia.org/wiki/Object-capability_model for access control and those that can monitor and set computational resource limits internally can prove very useful there
by tito on 2/11/25, 4:40 AM
I looked up my favorite mud awhile back. Just this post of other people looking for it for over 20 years!
https://groups.google.com/g/alt.mud/c/Hn5EOqDfcyg
by lokl on 2/11/25, 3:16 AM
MUDs were the most enjoyable social networks I ever experienced, and the least harmful to society.
by jasonthorsness on 2/11/25, 2:47 AM
Is there any kind of generalization of this approach that can be used for hot-swapping other servers with persistent connections?
by cess11 on 2/11/25, 7:24 AM
Building MUD:s is really fun.
https://github.com/michaelprograms/nightmare-residuum is a fine starting point, and I'd recommend playing discworld.atuin.net 4242 to get some ideas.
by CobrastanJorji on 2/11/25, 4:13 AM
Maybe a dumb question here. Why is a child process needed? Can the parent process not open a pipe, write its own state into the pipe, and then call exec, allowing the new binary to read from the pipe? Will exec destroy the pipe if there is no child process?
by teddyh on 2/11/25, 7:58 AM
Doing exec() to a new process but keeping the file descriptors relies on none of the file descriptors having the CLOEXEC (close-on-exec) flag. This flag is, IIUC, considered best practice for all file descriptors now, and in Python, all file descriptors are now by default created “non-inheritable” (i.e. with the CLOEXEC flag). A working model would instead be to create one inheritable file descriptor, do the fork() (and exec the new process in the child), and then sendmsg() all the normal file descriptors to the new process.
by alernon on 2/11/25, 3:39 AM
In the off chance someone from Iconoclast MUD reads the comments. Hi Fam!
by noneeeed on 2/11/25, 2:16 PM
That was a lovely short read. I didn't play muds much, I didn't get an internet connection until they were starting to look old hat, but I did play a Discworld one for a while at uni.
It's stuck me recently how much I miss that kind of programming. Everything I do now is hidden behind so many layers of abstractions, toolkits, libraries and environments that it all feels a bit divorced from what's actually going on. I need to come up with an excuse to do a bit of low-level coding again.
by upghost on 2/11/25, 2:28 AM
Spoiled web developer here. Since I have no way to test this, I have some questions! It seems like this whole thing could've been simplified to calling exec. Whats with the whole pipe/fork thing if you are just going to kill one process, clone the other (with exec), and the kill the original. Is the pipe crucial to transferring the contents of memory, or something?
by KingOfCoders on 2/11/25, 5:54 AM
What I loved about MUD, was meeting people and scripting quests. I was using TinyFugue TF back then to speedrun quests.
For a long time you could reach me at stephan@mud.de :-)
by alernon on 2/11/25, 3:39 AM
If any admins from Iconoclast MUD are reading comments. Hi Fam!
by DeathArrow on 2/11/25, 9:06 AM
So they didn't just fire up pods in Kubernetes with new code and shut down the old ones? And they kept state in memory instead of some fancy distributed databases? And it worked?
by Suzuran on 2/11/25, 1:36 PM
Nowadays if I did anything like this I would get screamed at to kill myself for being unsafe and having code smells and using the wrong tools.
by rkagerer on 2/11/25, 6:25 AM
And yet we often still have to reboot our modern PC's after software / OS updates...