by maidenhead on 11/26/12, 6:35 PM with 200 comments
by ChuckMcM on 11/26/12, 9:25 PM
Sure he doesn't know what he doesn't know, but he has decided to fix that. Which, if you know your history, is not a whole lot different than Linus back when he was calling out Minix for being crap.
The challenge here is that the barrier to speaking on the interwebs is quite low so you can make a fool of yourself if you're not careful.
Jean Labrosse, who wrote uC/OS (which everyone called mucos), in his original thesis statement made many of these exact same arguments. And like your author he made some choices that he felt were reasonable, only to learn through experience that perhaps they weren't a well thought out as he had hoped.
I am a huge fan of people just saying "How hard can it be?" and jumping in. Enjoy the ride, they can surprise you if you under estimate them.
So assuming this person notes that they are getting a ton of traffic from HN, and comes here to investigate, my three suggested books are :
Operating System Concepts [1], Operating System Implementation [2], and The Design of UNIX [3]. Preferably in that order. Any decent college library should have all three in the stacks.
[1] www.amazon.com/Operating-System-Concepts-Seventh-Edition/dp/0471694665/
[2] www.amazon.com/Operating-Systems-Design-Implementation-3rd/dp/0131429388/
[3] www.amazon.com/Design-Operating-System-Prentice-Hall-Software/dp/0132017997/
by readme on 11/26/12, 9:55 PM
-----------------------------------------------------------
My research
What I’ve found out so far:
Boot sequence: BIOS;
Master Boot Record (MBR);
Bootloader – the program that takes it over from MBR and loads your Kernel;
Kernel. How to write your own MBR and write it to Disk on windows.
I’ve written a small utility in Visual C++ that allows directly to read/write from disk (download here, source included for Visual Studio 2010 Express);
How to write bare bones C kernel entry point.
How to write “naked” functions on Windows in Visual Studio
Missing link - I still don’t know how to properly step from MBR to Bootloader to Kerlen, that is – write your own MBR code that would load bootloader, pass the execution to bootloader that would load and pass execution to bare bones C kernel:
What exactly is Global Descriptor Table (GDT) and Interrupt Descriptor Table (IDT), and how it looks in C and Assembly? How and when, and again how, if when is later (for example in Long Mode or Protected Mode) to set up all this GDT and IDT stuff. They say you have to set it up before the kernel. Then they say you can set it up with dummy values and set it up later in kernel. Then they say, that to set it up, you have to be in Real Mode, so your kernel (which might be way over 1Mb of real mode space), needs to switch between modes. And then if your kernel is over 1Mb, you can’t access memory locations after 1Mb, and so on… It’s confusing, but I’m going to find it out and post it here later on.
How to handle Interrupts in C? Will they perform as callbacks that await some return values or do I have to use inline assembly to process them correctly;
Is it possible to write MBR in C? I do understand that you still have to set ORG to 7c00h, and use some specific assembly instructions, but if they could be wrapped in C using inline assembly and C entry point can be glued with few lines of assembly code, why not?
by untog on 11/26/12, 8:51 PM
"I spend most of my days in the world of PHP, JavaScript (I love jQuery) and a little bit of HTML, CSS and ActionScript 3.0. I’ve done some hobby software development and also wrapped my head arround languages like C, C++ (except for templates), C# and Java."
It sounds like you don't have the experience required to make an OS. I certainly don't either (I'm no C-head) so I am in no position to snark, but you're going to fail in this endeavour.
That doesn't mean it's pointless, though- I think it'll be a tremendous learning experience in getting to grips with the core of how computers actually work. So, good luck. Just don't go thinking you're going to make the next Linux out of this.
EDIT: It's also important to note that the author didn't submit this to HN. He didn't say "take this, HN amateurs!", he just posted something on his blog that someone else picked up.
by Xcelerate on 11/26/12, 9:49 PM
Second, a lot of grunt-code can be found in open source projects, so most of the tedious/time-consuming programming can be eliminated if he chooses to follow this option.
Drop preemptive multitasking, cache-optimization, modes, virtual memory, and networking and there's not too much left.
And finally, it doesn't take that long to understand the GDT. Mine became corrupted once so I took a day to learn how it worked -- fixed that crap in a hex editor.
So no, he's probably not going to invent the next highly polished OS that handles every edge case and has been rigorously tested against bugs, but then again I don't think it's unreasonable to see a simple little functional OS.
by kamme on 11/26/12, 9:12 PM
by robinh on 11/26/12, 8:13 PM
by steve8918 on 11/26/12, 10:45 PM
A really, really good book for this that I've read is "Developing Your Own 32-Bit Operating System" by Richard Burgess. It starts you from the beginning, and walks you through all the steps in writing a really basic OS.
It's old and out of print, but it's definitely the best one I've seen.
Edit: I just found the website, they are offering the book free here:
by drewmck on 11/26/12, 8:41 PM
by wreckimnaked on 11/26/12, 8:35 PM
Not to disappoint you, but you should try doing some more low level programming or dabbling with some existing OS code to have an idea how this kind of programs look like. Maybe having a look at Minix for a reference of simple OS?
Have you thought about targeting ARM? Its architecture may be way less trickier than most Intel CPUs.
Well, good luck with that. Worst case scenario, you'll end up reading lots of interesting resources.
by forgottenpaswrd on 11/26/12, 8:31 PM
Too ambitious. Doing that requires millions of dollars and tens of thousands of man hours to make. How do I know? I do electronics and low level programming work and I am really good at it. Just understanding the bugs that manufactures put in hardware and then solve in software(because it is way cheaper) takes a ton of work.
As I suppose he is not super rich, he will have to convince people to join their project, a la Linus.
Good luck with that!! I really wish a clean-no backwards compatible OS were real, I will add native OpenCL, OpenVG and OpenGL to the list, but my arachnid sense tells me a person that does not use Unix will have a hard time getting traction with geeks.
by polymathist on 11/26/12, 11:36 PM
by robomartin on 11/26/12, 9:04 PM
A quick read of the "About" page is probably in order:
What to say?
"Someone holding a cat by the tail learns something he can learn in no other way" --Mark Twain.
Here's the tip of the tail:
http://www.amazon.com/Embedded-Controller-Forth-8051-Family/...
http://www.amazon.com/Operating-System-Concepts-Abraham-Silb...
http://www.amazon.com/Performance-Preemptive-Multitasking-Mi...
http://www.amazon.com/Design-Operating-System-Prentice-Hall-...
Have fun.
by mvzink on 11/26/12, 8:48 PM
by gusc on 11/27/12, 2:21 PM
http://gusc.lv/2012/11/im-writing-my-own-os-intermission/
And thank you again for inspiration (even the cynicism is inspirational ;)
by mimog on 11/26/12, 8:39 PM
by chewxy on 11/26/12, 10:07 PM
Sometimes the HN crowd surprises me. We pride ourselves in being hackers, most often idealistic (bitcoins and patent law change anyone?) but when a singular person shows idealistic ambition, we immediately engage in poppy cutting.
Sigh.
by endlessvoid94 on 11/26/12, 10:09 PM
This is the exact definition of hacking, if you ask me.
by munin on 11/26/12, 9:00 PM
Generally this is a bad idea because without any external motivation, you lose interest and stop working. With external motivation is worse, because you can burn out and become a catatonic shell of a person, staring absently into space for the rest of your life.
Just some FYIs:
> On the side note - It’s 21st century, but our PCs are still booting up as old-fart Intel 8086.
You should read about EFI (http://www.intel.com/content/www/us/en/architecture-and-tech...)
You should also read all of the lecture materials from good universities OS classes. In those classes, you basically do this. Some classes are more guided than others. Some places to start:
- CMU: http://www.cs.cmu.edu/~410/
- UMD: https://www.cs.umd.edu/~shankar/412-F12/
UMD uses a toy operating system called GeekOS that the students extend. You might find browsing its source code useful (http://code.google.com/p/geekos/)
Good luck!
by sourc3 on 11/26/12, 9:18 PM
Good luck to the author, nonetheless it will be a good learning experience for him.
by robomartin on 11/27/12, 1:18 AM
The linked article does NOT talk about a one-semester school project or a quick-and-simple learning OS.
No, the article talks about a web developer with no real experience writing low-level code not only wanting to bootstrap every single device driver but also ignoring years of accumulated knowledge and code libraries to write an OS that boots directly into graphical mode, does not take advantage of POSIX and more.
There's nothing wrong with the "How hard can it be?" approach to learning. I've done this many times. And almost every single time I came away with "I sure learned a lot, but what the fuck was I thinking?". The last time I pulled one of those was about fifteen years ago and the "three month project" took nearly two years.
What he is talking about is more complex than writing the Linux kernel from scratch because he wants to re-invent everything. Here are some stats on the Linux kernel:
http://royal.pingdom.com/2012/04/16/linux-kernel-development...
Even if his project was 10% of this it would still be a grotesque miscalculation for a single developer, otherwise employed and without the experience to back-up some of what he is proposing.
If, on the other hand, the post had suggested something like this it would have been far more reasonable an idea:
"Hey, I just spent a year implementing everything in the Tanenbaum book. Now I would like to start from that base and enhance the OS to make it do this...".
Let's compare notes in a year and see how far he got.
by readymade on 11/27/12, 1:51 AM
by bitteralmond on 11/26/12, 11:56 PM
I'm sure the idea of building a modern OS that is straightforward and written in a simple, popular language like C (and possibly Python later for higher-level stuff) will appeal to a wide range of people who will all want to help. I'd love to see this project happen, and if the day comes where Gusts is calling for help, I'll be right there in line to help him make this.
by happywolf on 11/26/12, 9:21 PM
by mrng on 11/26/12, 9:45 PM
Oh. OK, then.
by agumonkey on 11/26/12, 8:56 PM
If I had crossed the desire threshold to start that project (#1 project in my mind since I left college) I'd leave the C ecosystem altogether, design a typed ,functional ,binary friendly, modular, subset of C (and probably be forever alone). Something in the groove of http://en.wikipedia.org/wiki/BitC, even though its talented author concluded it wasn't a successful path.
by desireco42 on 11/27/12, 2:47 AM
I would prefer he decided to fork linux and change things he didn't like, then start from skratch. However, there is a great value starting from scratch. I wish I had a life :) to join him and figure out things together, it would be a blast, how many times in your life you have a chance to work on actual modern OS.
I believe it is totally possible for him to accomplish what he started, if knowledgeable people would join him and work with project together. Today with amazing tools, it is good time to create a new OS that would have modern tooling.
I wrote recently on my blog about a need for developer distribution of linux. Strangely this is still missing. http://softwaredevelopmentinchicago.com/2012/10/17/ubuntu-al...
It is great that we are discussing this. That is how things start.
by nnq on 11/27/12, 1:31 PM
...and when you reach to the GUI part, do the same for C++, use the latest version and language features: I've heard that VS2012 lasts upgrade got closer to it, but google around before settling on it
...or to keep it simpler: better use GCC compilers (since the Linux kernel is built with it, you should find enough compiler specific docs and related tools too)
by jff on 11/26/12, 8:26 PM
by zanny on 11/27/12, 5:50 AM
1. Targeting a modern architecture is good, but if I were being this ambitious, I would wager having such a backwards compatable burdened architecture like x86_64 (even when it is highly performant just through raw funding dollars) I would still rather start at square 1 on some risc 64 bit virtual 48 bit physical word system. Go even further, and design such a hardware ecosystem with heterogeneous computing built into the foundations - have arbitrary numbers of ALUs and FPUs and have different pipeline structures allowing for various degrees of SIMD parallelism across some tightly integrated weak cores and more heavily pipelined and bulkier serial cores, and have an intelligent enough instruction set to allow for scheduling (or even better, the hardware itself) to recognize parallel tasks and execute them with varying degrees of parallelism. Take AMD Fusion or Tegra to the next level and instead of having a discrete gpu and cpu on one die mash them together and share all the resources.
2. I'd kick C out. If I'm going with a new architecture, I need to write the compiler from scratch anyway. I might consider LLVM for such a system, just because the intermediary assembly layer is intentionally lossless and allows for backwards language compatability with everything under the sun right now. But ditch C, take modern language concepts from C++, Python etc, and cut down on the glyphic syntax and try rethinking the distribution of special characters (I think pointer<int> c makes more sense than int (star)c, for example - go even further, and provide 3 levels of verbosity for each concept, like pointer<int32> c, ptr<int32> c, and &:i32 c). I would definitely want to fix standard type sizes at the least, having things like i32 integers instead of the int type being 16 or 32 bit, etc, with some more modern niceities like the D style real float that uses the architecture restricted maximum FPU register size).
3. Screw UEFI, mainly because it is a design by consortium concept - it is inherently cumbersome because it was a commitee project between industry giants rather than a revolution in booting. I do like cutting down on legacy interfaces, I'd go even further and try to minimize my system to (in theory) one serial transport and one digital, maybe 4, with unidirectional and bidirectional versions of both, and maybe support for some classic analog busses (like audio, which doesn't make much sense to transport in digital format, although I haven't looked into it much). Everything plug and play, everything attempting to provide power over a channel so you don't need additional power connectivity if you can avoid it. For the BIOS, I would replace it with some metric of scan busses for profiles -> incite some kind of device-wise self test -> provide device information in memory to the payload binary, to allow memory mapping and all the other goodness. Maybe even have the bios itself act as a sub-kernel and provide the mapping itself. Maybe even fork the kernel, and treat it like some kind of paravirtualized device environment where the bios never overrides itself with the payload but instead stays active as a device interface. Saves a lot of code redundancy between the two then. It would of course have an integrated bootloader and the ability to parse storage device trees for some bootable material. Maybe have file system standards where each partition has a table of pointers to loadable binaries somewhere, or maybe stick them in some partition table entry (obviously not a FS expert here).
4. Screw URIs, go straight for a kernelwise VFS that can reference everything. I'd love to see /net/<IP address>/ referening the top level of some remote servers public resources. You could have a universal network protocol where each connection is treated as a virtual mount, and individual files (and everything is a file, of course) can dictate if they use streamed or packet based data off some network transaction about the base protocol. So instead of having http://google.com, you could use /net/google.com/ which when opened does DNS resolution in the VFS to 74.125.224.72 (well, ipv6, obviously - we are talking about a new OS here, so 2001:4860:8006::62 - and as a side note, I would never try to get rid of IP as the underlying transport protocol - as insane I might be about redesigning hardware and rethinking stuff people much smarter than myself came up with, I know you will never ursurp IP as the network trasport everyone uses to connect the world ever). And then when you open google.com/search, you open a remote file that interprets the "arguements" of ?q=baconatorextreme on the extension into the returned page file that you access.
I agree with getting rid of Unix directories, they are outdated crappy, and all their names make no sense. However, /bin is meant to be system required binaries to boot, where sbin is root utility binaries, /usr/bin is general purpose executables that might not be local to the machine and might be a remote mount, and /usr/local/bin is the local machines installed binaries. Of course these polcies are never abided by, and they still have /etc, /usr/games, and a bunch of other folders to make life a hassle.
That's enough rates for a HN comment thread though, I'll stop and spare y'all :P
by olalonde on 11/27/12, 3:40 AM
by pshc on 11/26/12, 8:19 PM
If you strip out loadable module support and such, is it possible to boot without the usual POSIX support structure? Without filesystems?
by husam212 on 11/26/12, 9:37 PM
by guilloche on 11/27/12, 9:37 AM
As a developer, I have similar feeling on softwares including OSes, and I started a fresh vector editor project(Torapp guilloche online designer http://www.torapp.info), I know a vector editor is much simpler than an OS, but it is also pretty complicated. When designing the editor, I learned a lot and changed designs multiple times. I am sure that guy will learn and even if he can not complete an OS, he may leave some well-designed code base for other people to start with.
by robomartin on 11/27/12, 6:07 AM
In no particular order:
1- http://www.amazon.com/C-Programming-Language-2nd-Edition/dp/...
2- http://www.amazon.com/The-Answer-Book-Solutions-Programming/...
3- http://www.amazon.com/The-Standard-Library-P-J-Plauger/dp/01...
4- http://www.amazon.com/C-Traps-Pitfalls-Andrew-Koenig/dp/0201...
5- http://www.amazon.com/Expert-Programming-Peter-van-Linden/dp...
6- http://www.amazon.com/Data-Structures-In-Noel-Kalicharan/dp/...
7- http://www.amazon.com/Data-Structures-Using-Aaron-Tenenbaum/...
8- http://www.amazon.com/Mastering-Algorithms-C-Kyle-Loudon/dp/...
9- http://www.amazon.com/Code-Complete-Practical-Handbook-Const...
10- http://www.amazon.com/Design-Patterns-Elements-Reusable-Obje...
11- http://www.amazon.com/The-Mythical-Man-Month-Engineering-Ann...
12- http://www.amazon.com/The-Programming-Language-4th-Edition/d...
13- http://www.amazon.com/The-Standard-Library-Tutorial-Referenc...
14- http://www.amazon.com/API-Design-C-Martin-Reddy/dp/012385003...
15- http://www.amazon.com/The-Linux-Programming-Interface-Handbo...
16- http://www.amazon.com/Computer-Systems-Programmers-Perspecti...
17- http://www.amazon.com/System-Programming-Unix-Adam-Hoover/dp...
18- http://www.amazon.com/Memory-Programming-Concept-Frantisek-F...
19- http://www.amazon.com/Memory-Management-Implementations-Prog...
20- http://www.amazon.com/UNIX-Filesystems-Evolution-Design-Impl...
21- http://www.amazon.com/PCI-System-Architecture-4th-Edition/dp...
22- http://www.amazon.com/Universal-Serial-System-Architecture-E...
23- http://www.amazon.com/Introduction-PCI-Express-Hardware-Deve...
24- http://www.amazon.com/Serial-Storage-Architecture-Applicatio...
25- http://www.amazon.com/SATA-Storage-Technology-Serial-ATA/dp/...
26- http://www.amazon.com/Beyond-BIOS-Developing-Extensible-Inte...
27- http://www.amazon.com/Professional-Assembly-Language-Program...
28- http://www.amazon.com/Linux-Kernel-Development-3rd-Edition/d...
29- http://www.amazon.com/Version-Control-Git-collaborative-deve...
30- http://www.amazon.com/Embedded-Software-Primer-David-Simon/d...
31- http://www.amazon.com/Programming-Embedded-Systems-C/dp/1565...
32- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...
33- http://www.amazon.com/Operating-System-Concepts-Abraham-Silb...
34- http://www.amazon.com/Performance-Preemptive-Multitasking-Mi...
35- http://www.amazon.com/Design-Operating-System-Prentice-Hall-...
36- http://www.amazon.com/Unix-Network-Programming-Sockets-Netwo...
37- http://www.amazon.com/TCP-Illustrated-Volume-Addison-Wesley-...
38- http://www.amazon.com/TCP-IP-Illustrated-Vol-Implementation/...
39- http://www.amazon.com/TCP-Illustrated-Vol-Transactions-Proto...
40- http://www.amazon.com/User-Interface-Design-Programmers-Spol...
41- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...
42- http://www.amazon.com/Designing-Interfaces-Jenifer-Tidwell/d...
43- http://www.amazon.com/Programming-POSIX-Threads-David-Butenh...
44- http://www.intel.com/p/en_US/embedded/hwsw/software/hd-gma#d...
45- http://www.intel.com/content/www/us/en/processors/architectu...
46- http://www.intel.com/p/en_US/embedded/hwsw/hardware/core-b75...
47- http://www.hdmi.org/index.aspx
48- http://en.wikipedia.org/wiki/Digital_Visual_Interface
49- http://www.amazon.com/Essential-Device-Drivers-Sreekrishnan-...
50- http://www.amazon.com/Making-Embedded-Systems-Patterns-Softw...
51- http://www.amazon.com/Python-Programming-Introduction-Comput...
52- http://www.amazon.com/Practical-System-Design-Dominic-Giampa...
53- http://www.amazon.com/File-Systems-Structures-Thomas-Harbron...
54- ...well, I'll stop here.
Of course, the equivalent knowledge can be obtained by trial-and-error, which would take longer and might result in costly errors and imperfect design. The greater danger here is that a sole developer, without the feedback and interaction of even a small group of capable and experienced programmers could simply burn a lot of time repeating the mistakes made by those who have already trenched that territory.
If the goal is to write a small RTOS on a small but nicely-featured microcontroller, then the C books and the uC/OS book might be a good shove in the right direction. Things start getting complicated if you need to write such things as a full USB stack, PCIe subsystem, graphics drivers, etc.
by Geee on 11/26/12, 8:38 PM
by olgeni on 11/26/12, 8:44 PM
Actually it has a lot to say, but in this case it just appealed to the fifth amendment.
by merlish on 11/26/12, 8:44 PM
by jiggy2011 on 11/26/12, 10:17 PM
Seriously though, good luck.
by exDM69 on 11/27/12, 8:32 AM
Here's a few notes about his plans:
> Target modern architecture
> Avoid legacy, drop it as fast as you can. You can even skip the Protected mode and jump directly to Long mode
So if you want to "skip" protected mode, you'll have to write a pile of assembly code to get there. x86_64 is a lot more work than 32bit x86.
> Jump to C as soon as possible
You only need a few pieces of assembly code to get an operating system running: the boot code and the interrupt handler code. The boot code and the interrupt handler are just small trampolines that go to C code as soon as possible.
In addition to the boot and interrupt handler code, you occasionally need to use some privileged mode CPU instructions (disable interrupts or change page table, etc). Use inline assembler for that.
Anyone who (in this thread) suggested using something else than C seemed to be fairly clueless about it. Of the choices you have available, C is the simplest way to go. Everything else is either more work or more difficult.
> Forget old interfaces like PCI, IDE, PS/2, Serial/Parallel ports.
You're most likely going to have to deal with PCI bus at some point too, although many devices don't use the physical pci buses on motherboards, some devices still hook up to the pci bus. Look at the output of "lspci" on Linux, all of those devices are accessed through PCI. This includes USB, PCIe, SATA, IDE, Network interfaces, etc.
Again, using the modern buses is a lot more work than using the old ones and it partially builds upon the old things.
> Why does every tutorial still use such an ancient device as Floppy?
> Avoid the use of GRUB or any other multiboot bootloader – make my own and allow only my own OS on the system
Most hobby operating systems are just half-assed stage 1 bootloaders. Just get over the fact that you'll have to use code written by others and get booted.
Popular emulators (bochs, qemu) can boot multiboot kernels directly so you'll save a lot of time there too.
You need to get booted in an emulator and running under a debugger as quickly as possible. Operating system development is so much easier to do with a debugger at hand. Failures generally cause a boot loop or hang the device so there won't be a lot of diagnostics to help with issues.
So my advice is: set up Qemu + GDB + multiboot, and get your kernel booted in a debugger as early as you can.
I won't go into commenting his wacky ideas about VFS structure or APIs. It's nice to make great plans up front but by the time you're booted to your own kernel, a lot of the naïve ideas you started with will be "corrected".
Happy hacking and do not listen to the naysayers.
PS. here's my hobby OS: http://github.com/rikusalminen/danjeros
by dindresto on 11/27/12, 6:31 PM
by capkutay on 11/27/12, 2:18 AM
by grundprinzip on 11/27/12, 8:20 AM
by ww520 on 11/27/12, 1:49 AM
by dysoco on 11/26/12, 8:48 PM
by capkutay on 11/26/12, 11:50 PM
Also, prepare for about 6 months of hard yet rewarding, given that you put in about 50 hours a week ;)
by DannyBee on 11/27/12, 4:49 AM
by emeidi on 11/27/12, 11:34 AM
by djhworld on 11/27/12, 1:35 PM
by monochromatic on 11/27/12, 5:31 AM
Say you're writing your own OS: ok, sure...
by christina_b on 11/27/12, 2:16 PM
by ommunist on 11/26/12, 11:14 PM
by drivebyacct2 on 11/26/12, 9:21 PM
by frozenport on 11/26/12, 10:38 PM
by aeip on 11/26/12, 7:18 PM