Older blog entries for apm (starting at number 22)

I did some more fooling around with ACE, but it looks like it's too large and interdependant to try to use just the pieces I need. Too bad, overall I'm pretty impressed by it. After reading the ACE book I picked up Pattern Oriented Software Architecture - Volume 2 - Patterns for Concurrent and Networked Objects (POSA2). It covers a lot of the same things as ACE but from a more abstract perspective.

I've been prototyping a new design for the client-server network code in Suneido based on the ideas from ACE and POSA2. It looks like it'll be an improvement over the current code - cleaner, faster, and more portable.

Been "strongly encouraging" pair programming with my programmers. I don't think they're all totally sold on the idea - I'm not even sure I'm 100% convinced. But I have to say that the more we do it, the more I'm sold on it.

jhill - thanks for certifying me as master - I appreciate the gesture. :-)

Picked up the new C++ Network Programming Volume 1 - Mastering Complexity with ACE and Patterns by Schmidt and Huston yesterday. I knew about ACE but I'd never really looked into it too closely. Now I think I might. I'm always much happier to use a third party library if I know about the "philosophy" behind it. I'm always pretty cautious about adding dependencies on third party software to Suneido. It really is a "dependency" - if they have bugs or don't support what you need you can be in trouble. Even if it's open source, that doesn't mean you've got the time or understanding to fix it yourself. But at the same time, leveraging off other people's work can have major benefits. Suneido uses Scintilla for source code editing. We could have rolled our own, but we got a lot nicer component for a lot less effort. Of course, something like that is not too serious a dependency - it could be replaced fairly easily. If I rewrote Suneido's networking to use ACE, that wouldn't be so easy to change. My other concern is size. ACE is large! I don't want to double the size of Suneido just to get a little better networking! I'll have to play around and see what kind of size increase it would entail.

The other nice part about getting a "good" computer book is that it tends to really motivate me, not just in the area of the book, but all over. Maybe because it's "inspiring" to read about other people doing a really excellent job. It makes you want to raise your own standards, do better work, improve stuff. I started thinking about multi-threading Suneido's database server - something I've always been nervous about. And I thrashed out how to clean up the database history mechanism to be cleaner and more complete.

I spent a little time playing with VMware the other day. I really like the idea of being able to run multiple operating systems without re-booting. It would be really useful for testing. And it would let me run Linux as well as Windows. (Too much of my work is Windows based to run Linux most of the time.) In fact when I recently got a new computer I deliberately put more memory (512mb) and more disk (80gb) in order to have space for VMware. But ... as soon as I installed it, Windows XP started to act flaky. Strange delays, networking problems, video problems. All intermittent and unrepeatable. So, much as I like the idea, VMware had to go - Windows is flaky enough as it is. Besides, how much use would it be for testing if it's flaky? Who knows what the source of the problems was, but I don't have the time to try to track it down. Maybe it isn't fully compatible with Windows XP? Or my hardware? Another aspect that scares me away is that there doesn't seem to be any way to "turn it off". Even when you're not using it, it's got all kinds of stuff running - yuck! I wonder if using Linux as the host OS would be better? Maybe I'll give that a try at some point. It's too bad, the idea is cool.

I wonder if I'll ever make Master on Advogato? Without sounding too egotistical, I think my experience, expertise, and investment in open source is as great as many of the other people rated as Master. But I don't know anyone "important", and I'm not working on a highly visible project. (I know that "major project" is part of the criteria, but many of the masters don't seem to be working on "major" projects.) I guess in a way it's a weakness of the rating system - people have to know you. Of course, it doesn't really matter - it is just an ego thing - everyone likes recognition :-)

Enough thinking out loud - time to do a little work!

Got new computers at work and at home, nothing too special - Dell 1.6ghz P4's. They came with XP - so far it seems okay. I'm afraid I find it a little unnerving when it does things like set up new hardware without telling me. What was that Niven quote? Something like "any sufficiently advanced technology is indistinguishable from magic". I put 512mb of memory and an 80gb hard drive in my work machine - I'm planning to put VMware on it so I can test Suneido on different versions of Windows (and also run Linux). I still have some issues that only seem to surface on Win98 (which also happens to be the most common version right now, unfortunately). I only put 256mb and a 20gb drive in the home machine. I'd tell you these numbers boggle my mind but that would date me.

I also finally took the plunge and installed a Linksys wireless access point switch to connect my notebook and desktop at home. Only just installed it last night, but it seems to work pretty slick. Finally I can access the internet and print from couch :-) Of course, one of the reasons I prefered to use my notebook at home was because it was faster than the desktop - that situation is now reversed.

I've been pretty frustrated lately because I don't seem to be spending any time on "real" work. I've been trying to hire a programmer and a customer support person and we got a big pile of resumes this time. Even after weeding out the obvious rejects, we still had about 40 interviews to do. Pretty hard to get anything done in between interviews. And of course, switching computers at work and home is also time consuming. Hopefully things will settle down a bit in the next week or two!

Finally got around to taking a look at Eclipse - the general purpose IDE that IBM released open source. Read a bunch of the articles on the site. There are some pretty interesting ideas. Some aspects I liked, some I didn't.

• Although the Eclipse IDE is general purpose, it's written in Java, and to tailor it or extend it you use Java. Since I'm primarily a C++ programmer this is a bit of a negative.

• They chose to write their own (yet another) portable gui framework (SWT). I thought Java already had a portable gui? I find their approach very reasonable though. It makes me want to work on a portable gui for Suneido along the same lines.

• There's a good discussion of gui "resource" "disposal", i.e. how and when to free fonts, etc. and why finalization isn't a good approach.

• Eclipse has an interesting plugin architecture. It seems relatively simple yet flexible, with attention to performance. Plugins can define new "views", add to existing menus and toolbars, and even insert material into the documentation.

It makes me feel that Suneido's IDE is somewhat limited. I think it would definitely be worthwhile to incorporate some of Eclipse's ideas.

I've had fun the last few days getting Suneido to automatically change the mouse cursor to the hourglass wait cursor whenever it's busy. (This is on Windows.) You might think that would be easy, but it's actually fairly tricky. You can read about it on Codeproject.

I've wanted to do this for quite a while but been putting it off because I fully expected it to be frustrating. (And it was!) My thoughts after spending almost two days on this:

• Why isn't this built into Windows?

• Why hasn't anyone else done this? (Or if they have, why haven't they published it.)

• Documentation sucks. None of the critical information I needed was in the documentation.

• Thank goodness for news groups. I'm not sure I would have solved this without comp.os.ms- windows.programmer.win32

• Was it worth spending this much time on such a minor detail?

Despite the frustration (or maybe because of it?) I'm still pretty happy with getting this done. It's a little thing, but in many ways I think a good product is the gestalt of a whole bunch of such "little" things.

It sure feels good when you can make a specific change to some software and get a definite improvement!

Suneido uses a cost based optimizer for its database queries. For each operation (e.g. join, union), it estimates the cost of different strategies. The problem is that if you have a complex query with a lot of operations, then the number of possible combinations of strategies becomes very large - a "combinatorial explosion". So query optimization slows down a lot for large queries.

However, I had a feeling there was a lot of redundant work going on recalculating the same costs over and over in different combinations. If this was the case, then caching the calculated costs for each operation could save a lot of work. So I added a simple cache to each operation class, using a linked list. After calculating a cost for a certain strategy I added it to the cache. When asked for the cost of a strategy I checked in the cache first to see if it had already been calculated. This involved 20 or 30 lines of code in the base class for query operations. All the automated tests still ran - a good sign that I hadn't broken anything.

With some simple counters in the code I found that on the automated tests (mostly fairly simple queries) the cache was eliminating about 2/3 of the cost calculations. Not bad, but the real test would be complex queries. One of the more complex queries in our accounting package was taking about 2 seconds to optimize. With the cache it now took about .1 seconds - 20 times faster!

Not bad for about an hour's work! But the best part was that the structure of the code allowed me to make this change easily and locally without disrupting anything else. Yes, I could've included the caching when I originally wrote the code. (If I'd known at that point it was worthwhile.) But I'm a firm believer in doing simple versions first. If you try to include "everything" in the initial version, you'll never get it done, it'll never work, and you'll still have to change it later.

If you're interested, you can see the actual changes to query.h and query.cpp in CVS on SourceForge

Argh!!! I was in the middle of updating the Suneido website when the server started giving errors and I couldn't finish the update. Now stuff is "broken" and I can't fix it! DataPipe hosts our site, and for the most part they've been pretty good, but it's frustrating in this kind of situation to have to wait for their tech support to get around to looking at the problem. I'm tempted to set up our own server but I'm not sure our internet connection is up to it. We're in a research park so the bandwidth is shared. Can't really afford to get our own dedicated connection. Maybe someday.

I was feeling pretty good about things before this happened. The site has been up for over a year, and I've managed to average a release every month. And so far the releases have all been pretty fair quality, IMO.

Progress is always slower than I'd like, but I don't really feel like working more than the 60 or 70 hours a week I already put into it. Suneido has attracted a few regular small-scale contributors, but so far no one else has really got involved in a major way. Not that I had any naive notion that masses of open source developers would immediately flock to the project. "If you build it, they will come." doesn't really apply! There are a zillion open source projects and so far, obviously, Suneido hasn't convinced anyone it's worth major investments of time. Maybe that'll come, and maybe not. In the meantime, I'll keep plugging away. I just hope our website comes back!

Something that's been on my todo list for Suneido for a long time is cleaning up a few places in the client-server network code where there might be bad interaction with the TCP/IP Nagle algorithm. However, I'd never actually done any testing to see if it was a real problem.

I won't try to explain the Nagle algorithm in detail. I'd recommend "Effective TCP/IP Programmer" by Snader if you're interested. But in short, it's a standard technique used to improve the efficiency of TCP/IP. However, it assumes you alternate between send's and receive's, e.g. send a request, receive a response. If you don't follow this pattern, for example, you do two send's e.g. a header and then data, then the Nagle algorithm can slow you down to 200ms per send/receive, or only 5 "messages" per second - pretty slow. One "fix" is to simply disable Nagle, but then you lose the improvements it brings.

So, last night I fixed the code to combine multiple send's and this morning I did some quick benchmarks. On the client side, the places affected were output, update, and delete. Sure enough, with the old code I was only getting 5 outputs per second (ouch!) With the new code I'm getting 2500 outputs per second, or 500 times faster! Not bad for a few hours work! (NOTE: This only affects client-server operation across the network, not standalone use. Standalone I get about 8000 outputs per second.)

It just goes to show that knowledge is a powerful tool. If I hadn't read about this problem it could have been a long time before I figured it out. Of course, if I had any brains, I'd have made this change a long time ago!

Gave a slide show on our spring Shishapangma expedition. It's always a lot of work pulling a slide show together out of literally thousands of slides. Fun to look at them though. Some good pictures. We had a pretty good turn out and it seemed to go over well.

Still plugging away at Suneido. Activity has been a bit down this week, both downloads and the forum and mailing list. We just passed 2500 registered downloads (lots more unregistered). Had an inquiry from a Brazilian company wanting to put Suneido on their CD. We'd previously had a German computer magazine wanting to include Suneido on a CD but I'm not sure if it ever was.

I've been working on trying to finish Suneido's Version Control system. We've been using it in-house for probably 6 months, but I never seem to find time to finish it and document it. Maybe I'll be able to get some more done on it this weekend.

Should prepare a new release before the end of the month as well. And update CVS on SourceForge, although I don't think anyone's really using CVS yet.

One of our users was talking about deploying an application he'd written with Suneido. That would be a milestone. I don't think anyone has deployed any applications outside of our office. Hope it works well for him.

It seems like I'm spending more and more time doing "support" for Suneido. On the positive side, that means people are actually trying to use it. However, it also means I don't get as much done. A lot of the questions (but not all) would be answered by better documentation. I should be able to use some of my responses as a starting point for some documentation. I'm afraid I can't really get excited about writing documentation. I know it's worthwhile, and if I want Suneido to be successful, it's going to have to be done. But ... I'd rather be programming. I can't even really fantasize about someone else doing it for me, since there aren't many people who know it well enough (yet).

Did get a little bit done today on int64 support for the dll interface. It's still not perfect because Suneido numbers don't cover the full range that int64 does. At some point I should probably extend the range of Suneido numbers. Currently they use 4 "digits" where each digit is 4 decimal digits (0 - 9999), for a total of 16 decimal digits. To cover the full range of int64's I'd have to add one more "digit" to handle 20 decimal digits. Shouldn't be too hard, but unfortunately, some of the code assumes 4 "digits". And, obviously, I don't want to break something as basic as numbers. I do have pretty extensive automated unit tests though, so hopefully they'll catch most problems. It's not a big priority, but one of our users is trying to implement an interface to MySql and it uses 64 bit integers. He also wants float and double support in the dll interface. Shouldn't be too hard to add.

The big things I want to get done for the next release are the version control and the unit testing framework. Both are more or less written and we've been using them in- house for quite a while. But they need some cleaning up, polishing, completing AND documenting. There's that damn documentation again!

Time to bike home. Not even 6pm and it's already getting dark. That's the problem with living so far north. Of course, the long days in the summer are nice. At least it was warm today - the snow we got last week is mostly gone.