Older blog entries for pphaneuf (starting at number 72)

Tribes 2 (gaming)

Boy, was it a long time that I played that game! A few days ago, I found out that there was a new patch for the game, so I updated. Wow! Not only did they fix the crashing bug in the community browser, but they added the one feature I always though was missing, some audio hit confirmation (now, like Quake 3, it goes "plink" when you hit someone)! So it is now the perfect team game. :-)

If that wasn't enough, I found out that NecroBones, my favorite server, is back up! Well, that made my day. I gotta find a bit of time to go there and play a bit with the old buddies...

XPLC

XPLC is getting growth hormones it seems. It's getting better and better, I'll have to make a new release soon!

XPLC

apenwarr started a CVS branch for WvStreams that adds support for XPLC to it, adding an interface for the core implementation of a stream, so that new streams can be dynamically loaded (for example, adding HTTPS support to a program only involves adding the SSL stream, and linking with SSL is not required for basic functionality).

This is really nice, because it's the first real-world code using XPLC, and it is pushing it to be better and more usable. [apenwarr] commented on how the whole object creating was too involved and how re-implementing getInterface over and over got tiring, so I made object instantiation a one-liner and made a generic getInterface implementation that only takes a (compile-time defined) array of supported interfaces.

Also, it happens that this library is popular and is in most Linux distributions (it is used by the popular WvDial program). So if it goes through, XPLC would come with just about every Linux distributions, thus furthering my world domination plans. ;-)

It is interesting to see the collision of two such libraries, seeing WvStreams become slightly stricter and XPLC slightly easier to use, while not actually taking ideas from each others, just by having them rub in a single binary and making it work.

Reposted that diary, it was about to fall by the wayside, lost in testing of my ColdSync Advogato conduit...

Life: Went to "100% Ads Festival" (used to be the "Ads Night") this weekend, my first time! Was really cool, I recommend it heartily!

One of my friends just got herself a job in Québec City (a three hour bus ride), which is a bit sad, since her boyfriend (also a good friend) just got contracted here in Montréal. Looks like he'll move there when he's done, I'll miss them! Its not like we won't visit, but you know...

I'm just back from their place, we watched "Chasing Amy", one of Kevin Smith's movies. I like it a lot, I feel like it resonates with some things in my life. For example, the two comic store guys, so nerd it hurts :-). We're so broken, but we can't notice (see "Goedel, Escher, Bach" for why).

Palm: After posting a question on the Plucker developer mailing list, it looks like I should be updating the creation time of the PDB in order for Plucker to invalidate its metadata.

Now, the problem is that ColdSync's generic conduit doesn't sync PDBs headers, quite understandably. Argh!

I had started doing a "sync" conduit to update the creation time, but ColdSync::SPC doesn't seem to have anything to set the creation time of a database. So I thought about just deleting the metadata, but this should only be done if the database has changed. Hmm, I could put a "sync" conduit on the database itself, and just delete the metadata if any records have the dirty attribute. I really don't know why ColdSync's "fetch" conduit interface doesn't allow making DLP requests?

While ColdSync looks fine from what I've seen, the Perl module to make conduits could use some improvements. It requires Palm::PDB for no good reason, and dies at the slightest provocation (the PDB doesn't exist for a "sync" conduit? what if I'm about to create it? or I don't need it?).

Work: I have been using this Palm hacking to learn about the mechanics of synchronization, but I'll have to go back at project related stuff now.

I have managed to find how to make a proper makefile (non-recursive, with a complete dependency graph) handle generated header files, including correct auto-dependencies! The trick is simple: have a dependency for every source file (the .c or the .cc file, not the .o file!) on a build-header .PHONY target. Then, make that build-header target depend on the header files that need to be generated (you can generally figure them out from their source file names) and let an implicit rule do the rest! This will build all of the header before they are ever used, which then allows for usual auto-dependencies to work correctly.

This will help both for XPLC (to support IDL source files) and for work stuff, I'm rather proud of this hack, however simple.

Now, to make non-recursive makefiles scale to a large project smoothly using an unpatched GNU Make... :-)

XPLC: Speaking of that, did just a bit of tweaking this friday, after looking into making a Debian package for it. I'm rather disappointed with this. A bunch of files in an unsightly directory (that I can't specify the name of) at the top level, rather sparse documentation, etc. RPMs require a file at the top level (and only for automated packaging), but its a single self-contained file, and you get to pick the name yourself (save for the extension, of course). Blah, I'll ask one of my coworker to help me with this next friday.

Life: Went to "100% Ads Festival" (used to be the "Ads Night") this weekend, my first time! Was really cool, I recommend it heartily!

One of my friends just got herself a job in Québec City (a three hour bus ride), which is a bit sad, since her boyfriend (also a good friend) just got contracted here in Montréal. Looks like he'll move there when he's done, I'll miss them! Its not like we won't visit, but you know...

I'm just back from their place, we watched "Chasing Amy", one of Kevin Smith's movies. I like it a lot, I feel like it resonates with some things in my life. For example, the two comic store guys, so nerd it hurts :-). We're so broken, but we can't notice (see "Gödel, Escher, Bach" for why).

Palm: After posting a question on the Plucker developer mailing list, it looks like I should be updating the creation time of the PDB in order for Plucker to invalidate its metadata.

Now, the problem is that ColdSync's generic conduit doesn't sync PDBs headers, quite understandably. Argh!

I had started doing a "sync" conduit to update the creation time, but ColdSync::SPC doesn't seem to have anything to set the creation time of a database. So I thought about just deleting the metadata, but this should only be done if the database has changed. Hmm, I could put a "sync" conduit on the database itself, and just delete the metadata if any records have the dirty attribute. I really don't know why ColdSync's "fetch" conduit interface doesn't allow making DLP requests?

While ColdSync looks fine from what I've seen, the Perl module to make conduits could use some improvements. It requires Palm::PDB for no good reason, and dies at the slightest provocation (the PDB doesn't exist for a "sync" conduit? what if I'm about to create it? or I don't need it?).

Work: I have been using this Palm hacking to learn about the mechanics of synchronization, but I'll have to go back at project related stuff now.

I have managed to find how to make a proper makefile (non-recursive, with a complete dependency graph) handle generated header files, including correct auto-dependencies! The trick is simple: have a dependency for every source file (the .c or the .cc file, not the .o file!) on a build-header .PHONY target. Then, make that build-header target depend on the header files that need to be generated (you can generally figure them out from their source file names) and let an implicit rule do the rest! This will build all of the header before they are ever used, which then allows for usual auto-dependencies to work correctly.

This will help both for XPLC (to support IDL source files) and for work stuff, I'm rather proud of this hack, however simple.

Now, to make non-recursive makefiles scale to a large project smoothly using an unpatched GNU Make... :-)

XPLC: Speaking of that, did just a bit of tweaking this friday, after looking into making a Debian package for it. I'm rather disappointed with this. A bunch of files in an unsightly directory (that I can't specify the name of) at the top level, rather sparse documentation, etc. RPMs require a file at the top level (and only for automated packaging), but its a single self-contained file, and you get to pick the name yourself (save for the extension, of course). Blah, I'll ask one of my coworker to help me with this next friday.

Sorry!

I was testing my Advogato conduit a bit, sorry for the inconvenience! I think it works fine now.

Palm

I should add a few LaTeX-style shorthands for common HTML in that conduit, it's already getting a bit annoying...

Palm

I started hacking a bit on ColdSync, setting up Plucker with it and getting the mail to work on my Palm. I think that I'll have to write a conduit for both of these.

plucker-build doesn't update PDB files, it just overwrite them, and since the version on the Palm and the one on my workstation are the same number, the new one is not uploaded. So I think I'll have a fetch conduit that will modify the existing PDB instead.

For the mail, ColdSync just has a conduit for sending mail. I'll have to make another fetch conduit to gather mail from my IMAP account(s). I find it nice that the two sides (send/receive) can be separate like that.

But at the moment, I am currently hacking on an Advogato conduit (I wrote this entry on my Palm)

XPLC

Oh my. Finally got around to doing the unit test and the implementation of the module loader. I think XPLC might actually be usable now!

Photo

Got myself a new camera, a Canon Elan 7e (EOS 30). Wow, is it fast! I'm used to the 1.5 fps, long shutter lag and long viewfinder blackout of my Rebel 2000 (EOS 300), this thing is really quick, and rather quiet too. Very impressed. Now I just need to get off my ass and scan some good stuff!

This is a piece that I wrote on my (almost never visited) home page, back when I had one. Since people have been hassling me with this a bit, here it is, so that it gets some visibility and I have an URL to give people!

One big "feature" of XPLC that will either attract or deter some people is our stance on threading: I do not smoke this.

Okay, let's be clearer: preemptive threading is mostly out of the picture, the possible exceptions being operations that cannot be done in a non-blocking manner otherwise (like using gethostbyname() on Unix) and multiple CPU systems (we do want to use all of the CPUs, so the plan is something like N or N-1 threads, where N is the number of CPUs in the system).

The reasoning is that when it isn't deadlocks, race conditions, resource contention between threads, it's context switching making things that should be fast go slow (by interrupting right in the middle of your nice hand-optimized assembler tight blitting loop, wrecking the cache and do some other stuff that will probably make you miss the video refresh and look even slower than it actually is (and God knows look is everything these days).

This is quite expensive a price to pay when all you really get is the illusion that multiple things happen at once! Careful programming will give you that same illusion, with a much lesser impact on performance, since it leaves you in control of the scheduling, letting you wait until after your blitting is done.

Some people will object that the "careful" part will make programming much harder than what it will give back in performance, and this looks true at first look. But what do you know about first looks? They are often wrong. Threads are tricky to get right. There are traps at every corners and believe me, you will fall into them. You will think your debugger smokes crack. You will cry for your mother.

Threads can be faster in many cases, but it will definitely take an expert to make them do so, and statistics say I can bet you are not and still make a ton of money. Those who will owe me money will find themselves putting in too many locks or too coarsely grained locking will see their performance go right down the drain.

All of that said, there is hope! Proud users of POSIXy systems will find GNU Pth (Portable Threads), a cooperative multithreading library, a very useful addition to their arsenal. I was intending to use that library, but (ironically) it has a portability problem, being quite dependent on POSIX features for its portability. The current development version has started support for a Win32 version, but it is dependent on the Cygwin compatibility POSIX layer rather than a native version. MacOS, BeOS and other operating systems would also be left out in the cold anyway. But other similar cooperative multithreading libraries exists, this is just one example!

Some links:

• This interview of Ingo Molnar on Slashdot has a very interesting answer about this at question #6.

• The State Threads library is another non-preemptive thread library that is specifically oriented toward I/O.

• Dan Kegel has a web page about designing a web server that could handle tens of thousands of clients. Among other things, threading is discussed, and it contains a large number of links to relevant material and papers.

• I found the slides for a talk that John Ousterhout (the creator of Tcl) made at the 1996 USENIX Technical Conference.

• My colleague apenwarr says the following: WvTask (part of WvStreams) uses just plain C++ and K&R libc to do co-operative multithreads with setjmp/longjmp. It's definitely not pthreads-compliant, but it should run fine on nearly any OS/compiler/library. I think I even tested it on Borland C++ 3.1 for DOS, once.

Heat Problems

Though we are now in a break of cool weather, Montréal has been through a pretty harsh heat wave in the last few weeks. I don't have A/C at home, and I am running a 24/7 server. The GeForce2 video card (which seems like it had an iffy cooling fan) died of some analog problem (it still works, but everything is distorded) that is common for these cards, apparently.

A bit later, I turned off the server to do some clean up in the back. When I turned it back up, the fans and hard drives came up, but it didn't boot. After some troubleshooting, found out that with the ATI video card I had put in (and which worked perfectly for a week), it wouldn't boot, and putting in an old S3 ViRGE that I had laying around, it would boot, but only after a real power cycle (unplug/replug from the wall plug or cycle the switch on the back of the power supply). I'm a bit afraid there is now something wrong with my motherboard.

I underclocked it from 1200 to 900 MHz, to spare it from some heat...

Life

Wow, I am amazed at how grumpy I became during the heat wave. Physically, I stood up pretty well, but psychologically, it makes me pretty shaky. I'll try to be more careful about that, particularly when dealing with others...

Work

Great news! Stéphane will join us at Net Integration. He's good, and he's nice working with, so I am looking forward to that.

Also, we just received a few new co-op students for the term, and one of them used to work on printing stuff for Corel, I think he's going to be helpful solving some things about WvPrint.

I converted WvStreams makefiles from recursive to non-recursive, with a speed boost from 30% faster compilation time for a complete build to a ludicrous speedup for an empty build (between 15 or 30 times faster). It's also been at least partly autoconfized. I wish automake made non-recursive makefiles! No, they're not available yet in the public release of WvStreams, but they'll be coming.

Jeff Darcy has replied on his weblog to an email I had sent to David McCusker, which the latter had put up on his weblog. Whew! Here's a continuation of this discussion.

The performance of smaller/simpler instructions is usually worse than the equivalent CISCy code (and I'm thinking of a much bigger granularity than CISC) because the instruction decoding overhead becomes a bigger percentage of the equation, everything else being equal.

Jeff asks why we should be worrying about the performance of interpreting code, when incremental compilation of those virtual instructions is the right thing to do when performance is what you want. He makes a good point about the level of abstraction and expressiveness being the important things, but I would point out that the in-memory representation of a Perl program plus the memory used by the native code implementation of these instructions is often smaller (and more cache friendly) than the fully spelled out native versions, and the quality of the virtual instructions implementations optimization is probably better than what an incremental compiler can do. In fact, the Perl interpreter has put all of the virtual instruction implementations in a single compilation unit (and they also tweaked the order of these inside that unit), so that they are in a contiguous area of memory. All of this adds up to code that's almost as fast as native, optimized C++ can be.

They can also be loaded from disk more quickly (though Perl itself doesn't take much advantage of this, as it compiles the code every single time).

For reference, the test program in "The Practice of Programming" (Kernighan/Pike) is a Markov chain generator. The timings and lines of codes were as follow (the first timing is Pentium II 400/Windows NT/VC++ and the second is MIPS R10000 250/Irix/can't remember the compiler):

• Plain C: 0.30/0.36 seconds, 150 lines of code
• Perl: 1.0/1.8 seconds, 18 lines of code
• C++/STL: 1.5/1.7 seconds, 70 lines of code
• AWK: 2.1/2.2 seconds, 20 lines of code
• Java: 9.2/4.2 seconds, 105 lines of code

Perl is even faster than C++/STL! I wonder how good of a job would a JIT compiler do with this? Then again, it is possible that the Java version did use a JIT compiler, and it didn't do it much good if it is the case!