Older blog entries for lupus (starting at number 20)

Mono on the Nokia 770 OS 2006

Starting with Mono version 1.2.1, the Mono JIT supports the new ARM ABI (also called gnueabi or armel). This is the same ABI used by the 2006 OS update of the Nokia 770 and it should be good news for all the people that asked me about having Mono run on their newly-flashed devices.

The changes involved enhancing the JIT to support soft-float targets (this work will also help people porting mono to other embedded architectures without a hardware floating point instruction set) as well as the ARM-specific call convention changes. There was also some hair-pulling involved, since the gcc version provided with scratchbox goes into an infinite loop while compiling the changed mini.c sources when optimizations are enabled, but I'm sure you don't want to know the details...

This was not enough, though, to be able to run Gtk# applications on the Nokia 770. When I first ran a simple Gtk# test app I got a SIGILL inside gtk_init() in a seemlingly simple instruction. Since this happened inside a gcc-compiled binary I had no idea what the JIT could have been doing wrong. Then this morning I noticed that the instructions in gtk_init() were two bytes long: everything became clear again, I needed to implement interworking with Thumb code in the JIT. This required a few changes in how the call instructions are emitted and at callsite patching. The result is that now Mono can P/Invoke shared libraries compiled in Thumb mode (mono itself must still be compiled in ARM mode: this should be easy to fix, but there is no immediate need now for it). Note that this change didn't make it to the mono 1.2.1 release, you'll have to use mono from svn.

As part of this work, I also added an option to mono's configure to disable the compilation of the mcs/ directory, which would require running mono in emulation by qemu inside scratchbox. The new option is --disable-mcs-build. This can also be useful when building the runtime on slow boxes, if the building of the mcs/ dir is not needed (common for embedded environments where the managed assemblies are simply copied from an x86 box).

There are not yet packages ready for the Nokia 770, though I'll provide a rough tarball of binaries soon: the issue is that at least my version of scratchbox has a qemu build that fails to emulate some syscalls used by mono, so it's hard to build packages that require mono or mcs to be run inside scratchbox. I'm told this bug has been fixed in more recent versions, so I'll report how well jitted code runs in qemu when I'll install a new scratchbox. This is not the best way to handle this, though, because even if qemu can emulate everything mono does, it would be very slow and silly to run it that way: we should run mono on the host, just like we run the cross-compiling gcc on the host from inside scratchbox and make it appear as a native compiler. From a quick look at the documentation, it should be possible to build a mono devkit for scratchbox that does exactly this. This would be very nice for building packages like Gtk# that involve both managed assemblies and unmanaged shared libraries (the Gtk# I used for testing required lots of painful switches between scratchbox for compiling with gcc and another terminal for running the C#-based build helper tools and mcs...). So, if anyone has time and skills to develop such a devkit, it will be much appreciated! Alternatively, we could wait for debian packages to be built as part of the debian project's port to armel, which will use armel build boxes.

This afternoon Jonathan Pryor pasted on the mono IRC channel an interesting benchmarklet that showed interesting results. It came from Rico Mariani at http://blogs.msdn.com/ricom/archive/2006/03/09/548097.aspx as a performance quiz. The results are non-intuitive, since it makes it appear that using a simple array is slower than using the List<T> generic implementation (which internally is supposed to use an array itself).

On mono, using the simple array was about 3 times slower than using the generics implementation, so I ran the profiler to find out why.

It turns out that in the implementation of the IList<T> interface methods we used a special generic internal call to access the array elements: this internal call is implemented by a C function that needs to cope with any array element type. But since it is an internal call and the JIT knows what it is supposed to do, I quickly wrote the support to recognize it and inline the instructions to access the array elements. This makes the two versions of the code run with about the same speed (with mono from svn, of course).

The interesting fact is that the MS runtime behaves similarly, with the simple array test running about 3 times slower than the IList<T> implementation. If you're curious about why the MS implementation is so slow, follow the link above: I guess sooner or later some MS people will explain it.

Mono for ARM/Nokia 770 binaries
I made tarballs of binaries for use on Linux/ARM systems, including the Nokia 770. And, yes, Gtk# apps work fine on it:-).
Happy hacking, you'll find them here.

Mono on the Nokia 770
After the Mono JIT port was done using a desktop little-endian ARM computer, Geoff just recompiled it and run it on a Linksys NSLU2 (which runs a ARM processor in big-endian mode).
That was pretty cool. I wonder if it is as cool as running mono on a Nokia 770 (no recompilation necessary, just copied the binary from my Debian box). Here it is running our hello world app.
Many thanks to the fine folks at Nokia for sending me a prototype so quickly.

Mono ARM port
The Mono ARM port is mostly complete: check it out from svn or from the soon to be released mono 1.1.9.
It has been bootstrapped on both little and big endian Debian ports (thanks to Geoff for enduring and submitting his Linksys NSLU2 to an extended compiling session to test big endian support).
Feedback from folks with Linux PDAs is appreciated: I'll see if I can prepare some binaries later today as well.

Memory footprint improvements in mono
At the end of March 2004, davidw pointed out that mono and Gtk# used a lot of memory for simple applications: in his mail he had 42196 KB of VSIZE and 11096 KB of RSS for a button in a window. Of course this was not good and at the time I told him to check again at the time of the mono 1.0 release or thereabouts and indeed we had some improvements: at the beginning of July the numbers were 31776 KB and 9916 KB respectively for VSIZE and RSS. The changes involved both the way Gtk# was built and the way mono loaded metadata information and how that was stored in the runtime data structures.

Today I ran the same test again to see what is the status, since we're going to release mono 1.2 in a few weeks/months. The numbers look better: 18912 KB for VSIZE and 8200 KB for RSS. These numbers are comparable to the existing perl and python bindings (17032/10028 the former, 15092/9164 the latter).

For reference, on my system the equivalent C program gives: 10568 KB for VSIZE and 4728 KB for RSS. The aumount of writable mapped memory is what counts, though, so I got some data about that, too, with a small program looking at /proc/pid/maps. The C program has about 935 KB of writable memory mapped, 290 KB of which is what looks like the relocation overhead in the numerous shared libraries loaded. Since memory is mmapped in page-sized chunks, even if a library has only a couple of relocations, it will make a whole page dirty. Ulrich Drepper has a paper on how to make shared libraries behave better that is a worthy read, but in some cases there isn't much we can do if we want to support plugin-like systems.

A very amusingly sad fact is to be noted about the libgmodule library: this is a very tiny lib (9 KB of text size). So, to be able to share 9 KB of readonly data, we waste 4 KB of writable data. Maybe it makes sense to just include the gmodule code inside the standard libglib library. libdl has a similar issue (7 KB of text, note that the wrapper library is bigger than the wrapped one, sigh): maybe some linker magic could be used to include libdl inside libc, it would increase the size of the latter by just 0.5%.

In the mono/Gtk# case, the writable memory mapped was 4.35 MBs. Of these, 1 MB is the stack setup for the thread that runs the finalizers: hardly a few KBs of this memory is ever touched, so the kernel doesn't actually commit this amount of data. There is another megabyte that consists of shared mappings from files, which is used by the io-layer to implement the win32 handle semantics. Usually, very little of this memory should be actually read or written, so it shouldn't matter much and the io-layer rewrite Dick is working on will hopefully get rid of most of the usage, too. If we subtract the amount of memory used by the C program itself, we get that the memory actually allocated by mono/gtk# is about 1.3 MBs, in the perl and the python case it's about 2.5 MBs. The 1.3 MBs include about 200 KB of GC heap size, 128 KB of memory allocated for the jitted code (of which just 20-30 KB are used) and the memory used by the mono runtime data structures: metadata from the IL images, jit bookkeeping etc. Of course we plan to reduce our memory requirements even further, since this is good both for the people using mono in embedded solutions and for the people using multiple mono desktop applications. Hopefully in another six months time we'll have as good improvements as we had in the last year:-)

A small followup for the previous entry about exception handling performance.
After the discussion, Zoltan fixed most of the big slowdown issue and now running the pie-thon benchmark, mono is just 20% slower than CPython while running all the tests (it was 2x slower before). Great progress.
What propmted this entry, though, is an interesting result while profiling the eclipse startup running with ikvm: 30% of the time is spent in the garbage collector but about 16% of the time is spent doing exception handling. I haven't investigated if this is an implementation detail of ikvm or the way eclipse works on other JVMs, too. So it turns out that improving exception handling performance helps also eclipse startup times.

The Nemerle guys point out that exception handling is not quite fast as they'd like in Mono and the other CLR implementations and assume it's a design issue of .Net.
I can't say about other implementations, but as far as Mono is concerned, we were busy writing a couple million other lines of code and exception handling speed is of little importance when compared to the amount of work we did and the amount of work we have ahead of us. That said, making some particular exception handling pattern work faster is easy and in fact I just committed a change that makes the posted test case 4 times faster. The trick is to either throw objects that don't derive from System.Exception or to throw already thrown exception objects: in both cases we don't need to get the stack trace info, which is what currently slows down exception handling.
Of course we'll speedup the code when we'll have more time to dedicate to fringe performance areas (the changes are not a 1-liner as the change just committed:-).
The code could be optimized more, for another 2x speedup, if someone is willing to dedicate the time.

Anyway, this issue reminded me of a similar issue that IronPython exposed: when Jim gave his talk at OSCon, there was an embarassing case of exception handling in the pie-thon benchmark where Mono resulted 100 times slower than CPython. A few days after we knew about the issue, I committed some changes to reduce the overhead to more sane levels (we are now about 8 times slower in that particular sub-benchmark and again most of the time is spent gathering stack trace info).

It's interesting to note how the Mono performance improved for the pie-thon bench since IronPython was released. In his slides he mentions Mono 1.0 was 23 times slower than CPython (2 times slower if the exception-handling bench is not considered). A few days later I posted at http://lambda-the-ultimate.org/ the numbers using mono from cvs (what was to become 1.1.x) and there were already some improvements, with mono 1.8 slower than CPython (without the dreaded b5 test).
Today I rerun the benchmark and the numbers have improved a lot: Mono is now 2 times slower than CPython including the heavy exception handling test. The nice related result is that mono is 10% faster than CPython if b5 is excluded. This amounts to a general 2x speedup in a little over 6 months.

This gets us back to the discussion about the design of .net/mono and exception handling speed. There is nothing in the general design that would make exception handling particularly slow, but of course the virtual machine is not built around the need to make exception handling fast, and for good reasons. As Jim quotes in his slides from the Modula-3 manual: ``Implementations should speed up normal outcomes at the expense of exceptions ... Expending a thousand instructions per exception raised to save one instruction per procedure call would be reasonable.'' One of the reasons IronPython could implement the Python semantics with good performance on Mono is because Mono and the .Net design allow for very fast method calls. Contrast this with, for example, the design of Parrot, where method calls and returns are built around the concept of continuations. This makes exception handling in parrot probably almost as fast as method calls, but normal calls are also made one or two orders of magnitude slower than in Mono.
We'll see more speedups in Mono's exception handling, but people should consider that exception handling will always be about two orders of magnitude slower than a simple method call (well, except for some special cases that we plan to optimize, where they'll be comparable:-).

Mono PPC JIT hacking
Yesterday was a sort of turning point: the mono JIT for PowerPC is ready for wider testing on both Linux and MacOSX. While there are still at least a few bugs, people are unlikely to run into them (but, of course, I expect the wider testing bit to turn on more obscure bug reports).
We can also run quite a few Gtk# programs (well, GtkDemo, the other samples and the monodoc browser: I'm sure some brave people will build and try MonoDevelop and report results). Especially important was testing the Fifteen Gnome canvas game, which I did throughly (I just needed to remember to run ssh -C, since I tried it from Miguel's G5 on the other side of the Atlantic, in Boston).
It has been a turning point because only one of the bugs which I considered release-critical was a JIT bug. The other issues were a bug in Gtk# which corrupted memory on x86, too and some endianess issues in PEAPI.
Now, it's time for fixing the more obscure bugs and then move on to do a bit of optimizations.

Mono vs JVM for the desktop
Both in the initial post from Havoc and in the replies some simplifications of the issues were made to keep the discussion focused and manageable, but when simplifications start blurring the data points, it's better to spell out the options we have available.
The issues to consider are technical, legal and strategic in nature and as Havoc points out each of the proposed changes to the way the Gnome desktop will be developed has stregnths and weaknesses in each of these areas. Let's see some of them.

Technical issues: as a platform, both Mono and the free software Java virtual machines are mostly on par maturity and stability-wise. gcj has currently probably the lead on portability (I guess, since it's based on gcc, but I have no hard data: anyone knows the status here?), while Mono currently has JIT ports to x86, sparc and ppc (though itanium, s390, amd64 and arm are in the works): things are moving fast on this issue in Mono, though, so I expect more ports to be completed by the end of the year.
From another point of view, Mono has better features to easily integrate with the existing codebase (P/Invoke and embeddability): I think this is important since the core libraries for Gnome are likely to continue to be written in C for the next few years.
Language independence has always been one of the strong points of Gnome and the architecture of the Mono virtual machine provides for a better foundation than the JVM: support for structures, function pointers, tail calls, etc, enable people to write compilers that target Mono and produce reasonably good code for a variety of languages. For example, while it is possible to run Java code in Mono (using IKVM) the opposite is not true. When you want to choose a platform that could be used as a base for Gnome, it's better to choose one which actually supports the use and the interoperation of different programming languages (either with compilers that target the virtual machine directly or with easy to use and implement bridges).
I won't go into the details of a C# vs Java programming language, but it looks like most people (who used to program in Java) found that C# fixed many of the issues they had with the language (though the new version of Java due out in the next few months implements some of them, too).

Legal issues: most of the legal issues are centered around the availability of the code under a free software license and on patents. The Mono code is available under the LGPL license (the runtime) and the X11 license (the class libraries). Some tools, like the C# compiler are GPL. GNU classpath is available as GPL+exception, so there doesn't seem to be any huge difference here, though Havoc cites legal issues with the use of GPL code and the proprietary JDK (to me this just means that the open source Java community needs to reimplement the relevant code as we have been doing in the mono project).
The patent issue is somewhat different: the major companies backing the CLR and C# standard agreed to RAND and royalty-free licensing of the patents needed to implement the standard. Libraries non included in the standard may have patent issues, though AFAIK no patent has been granted or claimed yet (this is a similar case as for the libraries we already use in Gnome development: no dying company claimed patents on them yet, but there is no guarantee that won't happen). This is a general issue of patents as has been pointed out multiple times, so it's not something specific to Mono. In fact there are likely several patents on Java and the JVM and no explicit license fortheir use has been granted.
So, for Mono, at least the core virtual machine and library is free from reasonable legal issues, while the JVM is not, as things are known today.
Of course, people see Microsoft as the devil and, as Havoc says, joining forces with Sun may give better results against Microsoft. I'll just note two things: both Sun and Microsoft are big companies with a variety of positions inside them. From one side, for example, Sun donated OpenOffice to the community (I consider it a contribution as big as the Linux kernel, the GNU tools and Gtk+/Gnome for the success of free software systems today) and from the other, they funded SCO, threatened JBoss etc. Microsoft, instead, is completely outside of our community; they, too, injected funds into SCO and they are competing against us. But at the same time they support the standardization of the CLR and C# as Sun never did for the JVM.
And rejecting a standard only because MS proposed it doesn't look like a rational move to me: I bet that today, 90% of the standard committees have MS representatives and a good chunk of the proposals come from them. Are we going to refuse to interoperate with the standard because of that? One of the reasons of the success of free software is that we tryed to implement standards whenever possible, to interoperate with other systems and to allow the migration from proprietary systems to open source ones.
So, in the end, it boils down to this: do you trust more someone who is ambivalent with our community and hasn't granted a license on its patents for our use, or someone who is outside, but did grant the patent license? Please don't answer that not all the possible patents on all the .Net technologies have been licensed, because I can easily reply that no patent license at all has been granted for the JVM:-) Also note that Microsoft is currently in a dominant position, but they may need to cave in on interoperability issues as a result of the antitrust case currently going to a (favorable for us) end in the EU.
I know some people will reply that they won't trust either Sun or Microsoft. What I think they are missing is that in the current patent situation any alternative technology has the same risks: it's not Sun or Microsoft the issue, but the patent system. Consider the issue this way: whichever technology we choose for the development of Gnome or of our applications and libraries, the patent risk is a constant. If instead you plot the patent and legal risk in a graph against the success of Gnome and free systems you'll find it's a steep, increasing line. If we want to be successfull in providing a free software mainstream platform we'll face patent and all sorts of legal threats (even if they are unsubstantiated, like the SCO case shows).
We'll be free of legal issues only if we become irrelevant.