The fuss over kernel design

Posted 13 May 2006 at 22:07 UTC by lkcl Share This

Recently there has been a fuss over monolithic and micro kernels - specifically the direction of the Linux Kernel development. Free Software is about "freedom of choice", and we should be able to choose to compile the Linux Kernel as either a monolith or a microkernel. To help accelerate this process, could someone please steal Linus' laptop, install l4linux overnight on it and give it back to him?

All joking aside, a leader's job is not to stop people from being able to fall upon their own swords should they so wish. Amongst the free software community, a projects' free software leaders should take this even more to heart. We are not beholden to companies or to shareholders: we are beholden to a hell of a lot more than that: our users, directly and immediately and in our faces. There is no financially-driven greed to stop us serving their needs.

Only recently did Linus himself say (not in these exact words) that Gnome, by virtue of it treating its users like idiots (and taking away options), that only idiots will use it - and so he prefers KDE (for what it's worth, so do I prefer to recommend KDE to ordinary users).

It is interesting to note that in areas of expertise where specific experts excel, when such experts (Linux/Kernel) are at the mercy of other experts (Gnome/KDE) they still have to voice their concerns in order to advocate that things get done a certain way - they cannot hope to spend their time learning another codebase which may have taken years to develop (which is something that always irritates me about the "don't like it? go away and write a patch. code talks; talk walks" standard response to enthusiastic and naive developers - but that's another story).

In short: as is always worthwhile mentioning: expertise in one area does not mean sufficient knowledge or expertise in another.

Now it is my turn to raise my voice: I would be remiss if I did not point out where I can see and feel that something could, in my opinion, be better. I'm not asking you to blindly act: I'm asking you to listen, and to consider.

First, I should say: although I've delved several times for several months at a time into the linux kernel source (the skyminder and selinux) and even into KDE (superkaramba), I haven't the up-to-date technical expertise and memorised code knowledge (nor the time and money to become such an expert) of either the Linux/Kernel team nor of the Gnome/KDE team, but I can still see very clearly where a decision of a leader is restricting people's options (we won't go into the Samba Team strategic-decision-restricting saga - not here, not now).

What am I referring to? What am I talking about? What justification do I have for such bold statements?

I am referring to a decision, based on "technical merit", to make the Linux Kernel a monolithic design - a decision made (if I recall the archives I read correctly) in about 1991, when context switching was costly, and device driver support was scarce (so the amount of code was relatively small - and clean)

Several experts, most notably those associated with the University of Karlsruhe and the University of Southern Australia, and also with Andrew Tannenbaum, do not agree with this technical decision, and in the case of the l4linux team, they are putting their weight behind a research project: to port the Linux Kernel to L4 (currently, the current l4linux kernel port of the linux kernel version 2.6.16).

My personal coding and design preferences - which are based on elegance, purity, beauty and other ephemeral things AS WELL as more practical matters such as maintainability - are irrelevant inasmuch as I have no "clout" or "say" in the Linux Kernel design decision process BUT my instincts tell me that there is something going wrong (or more specifically, that there is room for too much to go wrong), when you have so many millions of lines of code in kernelspace.

Of particular relevance is some nightmare stories from Microsoft's decisions to move things from userspace to kernelspace. There are two that I know of. The first was a decision made for the NT 3.51 to NT 4.0 development. Windows 95 had just taken over from Windows for Workgroups; it was faster than NT 3.51, and the NT team were getting flak for it.

I should point out that the entire screen driver codebase ran, in NT 3.51, in userspace. The screen subsystem could (and sometimes did) crash - leaving the system unusable as a desktop but perfectly fine as a server, where experienced administrators could (and did!) remember the keyboard shortcuts, closed their eyes, and typed blind to administer their machines, until it was convenient to reboot it. Remote Desktop systems like Citrix/Winframe were easy to write, easy to debug... you know what's coming next, don't you.

Yep - you guessed it: the NT 4.0 team were told to move the GDI into kernelspace, for "speed" reasons. I remember one of the NT 4.0 betas crashing on me if I moved a window fractionally off the top of the screen, as it overwrote outside of the SVGA memory-mapped region....

By the time it came round to NT 5.0 (renamed Windows 2000 because Nortern Telecom aka NorTel own the Trademark "NT") - the GDI was so complex that it was near-impossible for Microsoft to successfully develop, with their own code, a Remote Desktop system - so they called in (and bought) the people who did Citrix (or their competitors, I forget which).

It didn't stop there: Microsoft even tried, at one point, to move all Printer Drivers into the NT kernel. Millions of lines of third party code; tens of thousands of inappropriate device drivers now all running kernel-side instead of user-side. It didn't take long for them to back out of that decision.

Bearing in mind that NT is based upon the MACH Kernel, it's been hacked about very badly - as political decisions are enforced upon them to wring extra speed out of machines, to the detriment of maintainability and product stability.

Personally, I don't care about 15% faster or 15% slower and I'd rather run SE/Linux than not (but I use Debian so I can't even have _that_ overhead on my machines). I'd rather run a slower microkernel and lose yet another 10% performance than have a monolithic kernel on my machines.

A microkernel speaks to me in terms of elegance, in manageability, maintainability and practicality - AS WELL as the standard microkernel design issues such as verifiability, message-passing - not to mention the need for near-full-blown RPC systems which can make your kernel development so much easier (RPC is another area of expertise which is sorely underestimated to the detriment of projects which roll-their-own hand-grown RPC subsystem).

So it comes to this: Linus is NOT a microkernel expert: he has not, to the best of my knowledge, worked with microkernels day-in, day-out (certainly not to the extent of the l4linux developers). He is a monolithic kernel expert and advocate. This means that we, as "ordinary" Linux users, are beholden to someone whose responsibilities do not entirely match with their area of expertise.

We are therefore at an impasse.

The experts who advocate microkernel designs are not part of the mainstream Linux development. The experts who are part of the mainstream Linux development are not going to just "hand over" their power and responsibility (by converting Linux to a microkernel design) just like that - and it would be irresponsible of them to do so: they are the guardians of a system which quite literally runs millions of systems now, and will run on hundreds of millions - possibly billions of computers - over the next few decades.

So, instead: it should be left up to users to make their own decisions, and to help them make that decision, by providing them with easy (or easier) options.

The solution is therefore not to say "the Linux kernel must be a microkernel design" or to say "the Linux kernel must be a monolithic kernel design". The solution is to have a compile-time option for BOTH. This gives users the freedom to choose. It also takes away some of the burden of porting every single kernel release off of the Universities who are maintaining l4linux.

I'm not saying that it wouldn't be hard work - it's going to take a hell of a lot more than a "send me a patch" job, because the l4linux team use a modified version of glibc, a modified version of grub (or it did the first time i checked a year ago but that appears not to be the case, now at time of writing), a modified version of libstdc++, a modified version of OSKit, different targetting options for different versions of the L4 Microkernel - the list goes on and it is, in itself, a massive undertaking.

Incorporating the efforts of the l4linux team into the mainstream Linux Kernel will require a radical rethink of the way Linux Development is done: it will shake things up.

First and foremost it will require the experts who maintain the Linux kernel to, instead of being the ultimate push-guardians of patch acceptance, to step outside the box and proactively take control and responsibility for some code, rather than to have it pushed at them - and to "tweak" it to suit the present development tree. Secondly but just as importantly, it will require splitting the present Linux kernel into two sections - the "main" bit of the kernel, and its device-drivers.

However - this is not such a bad thing. OSKit is, itself, derived from the Linux Kernel source code (and at the time of writing, the l4linux team have updated OSKit to 2.6.16 so they are usually about... 4 to 6 weeks or so behind the linux kernel release cycle?) So, porting OSKit BACK to the mainstream Linux Kernel means that the Device Drivers (which is what OSKit is - a way for people to create their own kernels and they can get all the device drivers from OSKit) and the actual "Kernel" bit get a clean separation - you could even have different people specialising in maintaining the OSKit bits and the remaining "kernel" bits.

If you split the Linux Kernel itself into actual "kernel" code and you take away the bits that are in OSKit, you end up with only about... what.... 50k to 80k lines of code? That can't be a bad thing, can it? Clearly defining interfaces separating kernel from device driver development?

What's that you say, Linux kernel developers - you don't like the OSKit interface? So change it - make it better. Contribute to the development of OSKit, rather than side-line it. Embrace it and extend it rather than shun it because you say it's not your responsibility: it IS your responsibility, and it's especially your responsibility to ease the burden of time and maintenance of people who use your code (and OSKit _is_ your code - with a two to six month lag on it as the maintainers and users of OSKit constantly have to update it behind every kernel release).

Embracing OSKit and taking responsibility for its maintenance, and basing the Linux Kernel on it as an everyday release is the first step to making development of l4linux alongside current-monolithic-linux a reality rather than a research project.

Once that main hurdle is crossed, things can go nearly back to the way they are normally run, except that now the l4linux team could send you patches at the "edges" - in their own #define CONFIG_L4LINUX sandbox which you wouldn't have to worry about as much or take significant responsibility for (other than to refer overkeen hackers gently off of LKML and onto the L4Linux mailing lists).

Your job is to enable things to happen. Your job is NOT to stand in the way of progress. Your job is to embrace all the possibilities - and at the moment you are not fulfilling your duty as our key ambassadors of the free software movement.

For example: yes, I could go to and download a precompiled kernel - but I couldn't possibly hope to do apt-get install l4linux-2.6.18-1-686 next week. Yes, I could download the source code - all multi-megabytes of it - and spend several days compiling it all up. The last time i tried, and I am no stranger to compiling large projects, I spent several days on it and ran into many difficulties (you need to compile up - and use - a specially patched version of glibc, libstdc++ and grub, amongst other things, which of course will conflict with the debian - and redhat and suse - versions).

In other words - it's a sideline. It's a side-show. and only by placing your support behind their efforts will anyone else possibly hope to benefit from the l4linux teams' work, which is to prove that a microkernel version of linux is a robust, viable, useable and exciting option (especially when you consider the Virtual Machine versions of the L4 microkernel kernel).

Some decisions are not just about what is technically "the best". Some decisions are not just about what is technically "the fastest". Some decisions are about what is strategically "enabling". Some decisions are about opening doors to possibilities.

Are you going to leave the microkernel doors shut - and if so, for how long?

off-topic: Free software is not about choice, posted 14 May 2006 at 01:03 UTC by atai » (Journeyer)

OK, this is not about the topic of kernel design. But free software is about software freedom, the ability to copy/borrow software for everyone's use. It is not about "freedom of choice."

it's a long story., posted 14 May 2006 at 01:24 UTC by lkcl » (Master)

atai, it's a long story, and i will be at ukuug giving a talk on which this, and other free software projects' strategic decisions (or lack of), will be a very relevant example.

you are absolutely right - many "free software" people have grasped the fact that "source code" is available, great, you can work with it, that's fantastic.

and, as you know, it takes great skill to work with that "source code".

however - what many "free software" people have NOT grasped is, exactly as you point out, that with the ABILITY to program - and understand the "source code" of "free software" - comes a significant responsibility: a duty to serve other people - the users of that "free software".

in short, my talk will be a modern-day geek-relevant illustration of the "parable of the talents".

Huh?, posted 14 May 2006 at 06:17 UTC by bi » (Journeyer)

I don't get it. It's the L4Linux people who make their source extremely hard to compile, so let's blame its slow development on Linus?

If I were Linus..., posted 14 May 2006 at 10:49 UTC by badvogato » (Master)

Are you going to leave the microkernel doors shut - and if so, for how long?

Yes. Linux is a trademark of Linus. According to The book of Luke, I am no expert on Microkernel. So it's fair to say, I can only be my own gatekeeper. Make yourself a 'Minus' if you like. Make a door in the wall if you like, and if it points to a larger space, people will follow you...

"hard to compile"...., posted 14 May 2006 at 12:16 UTC by lkcl » (Master)

I don't get it. It's the L4Linux people who make their source extremely hard to compile, so let's blame its slow development on Linus?
dear bi, thank you for responding.

unfortunately i have to say that i don't quite understand what you want me to say, to clarify things, for which i sincerely apologise.

which bits of what i say make you believe that i would even _want_ to advocate that linus is in some way "at fault" for what you believe to be "slow development" of l4linux?

if i implied in some way that you _should_ "blame linus" then i am extremely sorry for having given that impression.

Make a door in the wall if you like, and if it points to a larger space, people will follow you..., posted 14 May 2006 at 12:22 UTC by lkcl » (Master)

dear badvogato, thank you (truly!) as ever for your esoteric comments.

one small correction: i'll be standing at the door. following gateway keepers/makers isn't a good idea (it's a very odd vocation). going _where_ a gateway keeper/make points out that there is a door is entirely up to the individual, and is a completely different proposition from specifically following in the footsteps of the gateway keeper/maker themselves (not recommended).

thank you for listening folks: passing-for-fairly-normal-strange-behaviour-service is now resumed :)

Personally, I love the name loL4, posted 14 May 2006 at 15:43 UTC by badvogato » (Master)

luke, you ought to put that name in the spotlight, instead of Linus.

folks, check out

I bet $4, i see the golden spherical computational GO in loL4 implementation.

Unworkable, posted 15 May 2006 at 23:45 UTC by slamb » (Journeyer)

Cutting the baby in half leaves two dead half babies. Likewise, you can't have a compile-time switch between two fundamentally different designs. If you did, you'd find that all of the following are true:

  • Your system has all of the complexities of design A.
  • Your system has all of the complexities of design B.
  • Your system is an unreadable mass of #ifdefs.
  • Your system does not work in design A mode or design B mode.

This is not specific to the macrokernel vs. microkernel debate. Sometimes you just have to make a choice and go with it. Your choice will have advantages and disadvantages, and that's how it is.

Linux is fundamentally a monolithic kernel. There are many, many design choices that stem from this. Even L4Linux is essentially monolithic at present. See this message mentioning their goals. Apparently they want to make it more microkernelish incrementally. If they're successful, I think you'll find that their design diverging totally incompatibly from Linux's.

Oh, and free software users do have a choice:

  • use Linux
  • use L4Linux
  • use HURD
  • ...

And if they're unsatisfied with those choices, perhaps they need to become free software developers. Freedom doesn't mean other people have to do your work for you.

many options, posted 17 May 2006 at 21:16 UTC by lkcl » (Master)

dear slamdb,

these are worth reading:

(yes i tried it)

Re: many options, posted 18 May 2006 at 03:13 UTC by bi » (Journeyer)

Hmm, isn't AST also the professor who wrote that

[...] 5 years from now [1992] everyone will be running free GNU on their 200 MIPS, 64M SPARCstation-5.

and that

Writing a new OS only for the 386 in 1991 gets you your second 'F' for this term.

I hope with Minix 3, whose "stability" actually hinges on the memory protection features that aren't present on the 8086, AST is finding the crow delicious. Oh wait, maybe Minix 3 isn't "only" for the 386, or maybe now since it's 2006 not 1991 he gets a free pass. Damn, I hate backpedalling.

But since Minix 3 is a choice, I say, use it! And while we're talking about "freedom of choice", shouldn't you be demanding at AST's door that he include a compile-time option in Minix 3 to allow it to build as a monolithic kernel?

*cackle*, posted 20 May 2006 at 21:46 UTC by lkcl » (Master)

nice point - i like it. yes, i think he should. let's hope he has time available to make that possible, as a demonstration. the thing is, though, that as an experienced computer scientist advocating the strengths of microkernel designs, he's very unlikely to. oh well :)

hey, did you notice also: there's minix2 available for L4KA?

i tried minix3 in a qemu session - it worked - qemu struggled, however. i think i really need to actually put it onto real hardware: the install of xfree86 took forever to unpack into the 1gb drive-in-a-file and eventually qemu went splat.

however - i can officially say i actually tried it :)

Why?, posted 19 Jun 2006 at 17:15 UTC by prozac » (Journeyer)

Personally, I don't care about 15% faster or 15% slower and I'd rather run SE/Linux than not (but I use Debian so I can't even have _that_ overhead on my machines). I'd rather run a slower microkernel and lose yet another 10% performance than have a monolithic kernel on my machines.

I know not much about Kernel design. I am a User and not a Developer. However, I have a perspective on software design of pragmatism.

What will the benefit be by making this (or any other) change?

I have delved in OS software dev. years ago, and still write some Perl/PHP stuff. And there are only three reasons to change software, or three reasons to take a particular development/design path. And all are about efficiency in these areas:

  1. Speed and/or memory usage.
  2. Configuration, modularity; support of "add-ins" and modules.
  3. User interface and/or API presented to the external world.

The hard part of programming is to be effient in more than one area. For example, support for many CPUs would be (2). I am assuming micro/monolithic is (1). KDE is, in my limited experience, "good" at (3) but "really, really bad" at (1).

When running on a 500Mhz Intel box with 256MB of memory, 15% faster is really really REALLY VALUABLE when using X!

you're right about modularity, posted 20 Jun 2006 at 19:44 UTC by lkcl » (Master)

hi there,

yes you're absolutely right about the modularity.

imagine being able to write a video application - no i didn't say a friggin device driver - that captures part of your screen and re-presents it as a video device.

this kind of application is available for windows as a proprietary product from

can you _imagine_ the fuss over even _writing_ such a device driver let alone getting it into the kernel.

if, like in the GNU/Hurd, you have an API (using RPC and message-passing to communicate via the video "interface" from userspace to kernelspace and from there the kernel re-presents the data to userspace) which allows you to do this kind of thing, you don't NEED the involvement - or permission - of the kernel developers.

all you have to know is which version of the video-driver-API is "current" - and write your APPLICATION to that.

oh, is your application out-of-date? _not_ a problem: you can have a "wrapper" application which sits in between your use of the "older" kernel-video-API and the more "modern" API doing a translation on your behalf.

oh, what's that you say: you don't _need_ the permission of the linux kernel maintainers to write a device driver? hellooooo, can you spell "maintenance nightmare" as you try to keep up with the patch levels as the linux kernel evolves, if your patch _isn't_ accepted into the mainstream kernel?

now multiply this nightmare by... oh... how many device drivers and APIs are there in the linux kernel now?

so yes, you are absolutely right: a lot of headaches simply go away if you have a micro-kernel design where your device driver APIs are all RPC-based and use message-passing. you get some more (versioning) but even that can be dealt with - by careful planning, inclusion of version numbering in the APIs, and the use of "legacy-wrapper shoe-horns".

speeding up KDE...., posted 20 Jun 2006 at 19:46 UTC by lkcl » (Master)

1) use prelink 2) set export KDE_EXECS_SLAVES=1 in /usr/bin/startkde (check this!) 3) set export KDE_IS_PRELINKED=1 in /usr/bin/startkde

this stops kde dicking about with a whole stack of rubbish that should never have been coded into it, that is prelink's job in the first place.

requires kde 3.4 and above.

Re: you're right about modularity, posted 21 Jun 2006 at 14:00 UTC by bi » (Journeyer)

imagine being able to write a video application - no i didn't say a friggin device driver - that captures part of your screen and re-presents it as a video device.
can you _imagine_ the fuss over even _writing_ such a device driver let alone getting it into the kernel.

No need to "imagine", just look at how things actually are. How many people are running Hurd or developing specifically for it? How many people are writing drivers for the Hurd, in contrast to Linux? If it's so easy to extend the Hurd with more drivers in a stable way, then how mature is the Hurd compared to Linux right now? Why do people choose to bear with the "imaginary" "fuss" of coding Linux drivers instead of enjoying the "imaginary" ease of coding Hurd drivers?

hurd mentality, posted 26 Jun 2006 at 01:38 UTC by lkcl » (Master)

1) the hurd has not reached critical mass. the successful - and design-limited - linux kernel ensures that the hurd will not reach a critical mass - neither of developers nor users.

the gnu/hurd kernel project must therefore "live off the crumbs" from the linux kernel project by utilising "OSKit", which is basically the device driver code from the linux kernel project (the millions of lines minus about 80,000 or so lines for the actual kernel bits, basically)

2) you don't specifically "need" to "develop" for the gnu/hurd - you just write POSIX apps and they just "work", right?

that's me being pernickerty - i know what you mean.

here's the thing: the number of free software POSIX/UNIX people who will consider doing mad - and actually quite useful - things like writing a userspace video-capture and representing it for use as a video feed are quite small; the last thing on such people's minds will be "ohh, what's the best OS to use to do that?".

they'll go "how can i do that with linux?" they _won't_ go "how can i do that with FreeBSD" or "how can i do that with the GNU/Hurd".

they'll go "christ that's difficult and pointless under the current linux dictacted-regime".

i won't dictate to the linux developers what needs to be done. i'm quite happy, however, to point out that the current development regime is very restrictive, though.

Re: hurd mentality, posted 27 Jun 2006 at 04:28 UTC by bi » (Journeyer)

Which raises the question: why didn't the Hurd reach critical mass? Since the Hurd was started earlier than Linux, and it's easier to develop drivers for the Hurd than for Linux, so if your "imaginations" are any guide it should be Linux who's feeding off the crumbs of the Hurd!

why is windows so much more popular?, posted 27 Jun 2006 at 22:16 UTC by lkcl » (Master)

you could ask the same question of any "successful" OS.

why was GEM/DOS ousted by windows 1.0?

why was OS/2 ousted by windows 3.1?

if ever you find out the answer to your question please let us know.


Re: why is windows so much more popular?, posted 28 Jun 2006 at 11:48 UTC by bi » (Journeyer)

Maybe because, erm, the winners tend to focus on real-world issues instead of doing lots of "imagine..." thought experiments? Strange that, after you refer to an article which stresses the importance of testing standards against reality, now you're advocating something only because it sounds nice in fiction.

benefits of "nice to have", posted 4 Jul 2006 at 11:45 UTC by lkcl » (Master)

if we didn't imagine, nothing would get created, bi.

here's the nice thing about "advocating", bi: you can think ahead, do "what-if" thought-experiments; it's part of what makes us human.

i can see the linux kernel creaking under the weight of its own restrictive design, and the problems it's causing are only going to get worse, not better, as the number of new devices and new drivers will only increase.

keeping coordination amongst all those drivers is making life really difficult.

and there's a well-known, well-implemented and real-world solution.

Do you mean Nice To Have, or Not Nice To Not Have?, posted 4 Jul 2006 at 18:37 UTC by bi » (Journeyer)

Well, there are a few problems with this spate of "imagination":

  • You use imagination not to explore what's possible, but to decide what's impossible.
  • You make up imaginary problems faced by imaginary developers coding imaginary drivers, while totally ignoring the costs and issues behind setting up the alternative microkernel infrastructure in the first place -- which, I imagine (!), will be an extremely high one.
  • ESR did something similar -- he predicted that the kernel will careen out of control its own weight if Linus Does Not Listen To His Sage Advice. Well, the kernel hasn't gone out of control yet, but ESR sure has.

okay, posted 10 Jul 2006 at 19:02 UTC by lkcl » (Master)

fine, bi - have it your way. you are right. nothing can be changed, because you are right.

Bi's Law of Sarcasm, posted 11 Jul 2006 at 03:14 UTC by bi » (Journeyer)

"As an online discussion on politics or philosophy grows longer, the probability of people trying to end it merely by sarcastically restating their opponents' claims approaches one."

I think, instead of coding drivers for rhetorical devices, it might be better to simply follow the "don't like it? go away and write a patch. code talks; talk walks" line of action. It's not a bad idea, really.

absolutely., posted 18 Jul 2006 at 15:12 UTC by lkcl » (Master)

yes. you are entirely right. nothing can be achieved except by following the status quo. there is nothing that i can teach or advocate to you.

Re: absolutely., posted 22 Jul 2006 at 19:10 UTC by bi » (Journeyer)

Nothing can be achieved if you are doing nothing to achieve anything.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page