Older blog entries for alvherre (starting at number 4)

My plans

It's been a month since my last entry, so clearly I'm not a good blogger. Anyway:

I submitted the MultiXactId patch some time ago. Tom reviewed and committed it. We don't have the foreign key problem anymore! I was kinda ashamed by that problem, so I'm happy it's now solved. Fortunately, we figured a way to make the mechanism be used only when it's really needed, that is, only when a row is locked by two or more transactions at the same time. This way we don't even incur in a performance hit — the only point where the mechanism enters the picture is when previous Postgres' versions would block, so it's a net win.

Right now I'm looking at Heikki Linnakangas' two-phase commit patch. With some luck I can help clean it up some before it gets to Tom, so that when he gets it, we can quickly have a fully functional 2PC implementation in time for 8.1.

Additionally, I'm looking at completing my shared dependencies patch before 8.1 freezes. I let it rot some; I partly blame Stephen Frost's SQL role support patch. Of course it's a lame excuse — it'd be cool to have both things in. But they conflict anyway so I'm leaving this behind a little.

It worked!

Ok, I am amazed: the phantom Xid idea worked! Only I didn't name it phantom Xid because that name has a bad history. So I named it MultiXactId. The implementation is finished; it only needs some more testing, making sure it deals correctly with wraparound, and that it correctly truncates SLRU segments.

To recap: the idea is to allow a row to be share-locked. This means somehow storing multiple TransactionIds in the tuple header. This is done by using a shared counter similar to TransactionId, which I have dubbed MultiXactId. Each MultiXactId is associated with variable-length storage in SLRU, on which the set of TransactionIds resides. This MultiXactId is stored in the tuple's Xmax, just like a TransactionId is stored in case of a SELECT FOR UPDATE.

I just posted the patch; probably Tom will have some disagreements with the way some things are done, but he already expressed approval of the general idea, so I expect that eventually this is going to be committed.

I feel somewhat ashamed that it took me two weeks (counting from the last blog entry) to complete the implementation. I hope not everything takes me this much, or I won't be having a lot done.

After this I'll return to the shared dependencies patch, and then I don't know what my next project will be ... maybe I'll continue trying to figure out a way to do the translation of the docs, which is badly needed.

phantoms – a better idea

After wasting several days trying to come up with a way to make Postgres' lock manager spill to disk, I gave up and ICQed Bruce. This was mainly motivated by the fact that several people complained that the idea of having any process sleep on I/O waiting for the lock manager was a loser. After thinking on it for a while I cannot but agree. I even considered writing some sort of lightweight B-Tree handler on top of SLRU, so we could quickly find lock entries on disk by their LOCKTAG; this is better than using a plain sequential structure, but it's difficult to get right, it's lots of additional code, and it will still need I/O for getting locks :-(

So when I talked to Bruce, he simply said it was a bad idea to use the lock manager to lock tuples; we can't afford to have that many lock entries. In a way it was a disheartening idea. He wondered that maybe we should use some sort of Phantom TransactionIds, an idea on which hearing I almost freeze on terror – I had already heard of Phantom Xids from him, during the nested transactions project, and it was very painful (to me) to code and in the end it was an utter failure. (Althought coding that was a very interesting exercise from which I learned a thing or three – the patches and the disappointing comments from Tom are here.)

So I spent the next day thinking on phantom Ids and felt miserable ;-) because I didn't really see how would they apply to this case.

Turns out it's surprinsingly easy to have it all fit together. Phantom Xids in this case are just a means of indicating a set of TransactionIds; and we store the set on SLRU. So when somebody wants to lock a row, he creates a set; and when somebody needs to lock exclusively, he can sleep on every TransactionId of the set. The only tricky part is how to store variable length content on SLRU, but that is easily solved by using two SLRU areas.

So far I have coded a good deal of the whole thing; though it still doesn't do everything yet, I have now confirmed that the idea is perfectly workable and I'm sure it will have perfect performance.

This means we will have shared row locks for 8.1! I can't help but feel good about that. No more deadlocks in foreign keys. Cool!

posted the patch

Yeah, apparently what I had first thought was in fact the right idea. I'm talking about my shared-row-locks project. Due to some misunderstanding on my part, I figured that the simpler idea was wrong; so I tried a lot of other things, and of course, they were also wrong. So eventually I came back and tried the first thing again, and discovered that it works as expected (by me at least). So I posted the patch, and promptly received a comment from Tom which made me notice a gross mistake. Easily solved, but gross anyway ;-)

performance measure stupidity

I ran some performance testing to verify that my patch won't make people too angry at me. I was terrified to discover that it had dropped by 25% or so in pgbench. I spent an hour and a half looking at the patch searching for the culprit (I didn't want to compile with profiling enabled because my machine is somewhat slow) ... And then I realized that I had compiled the whole backend/access/heap directory with -O0. Recompiled, reran pgbench and now I see no measurable difference between pristine sources and my tree. That's fortunate at least. I still have to see how will the lock-spilling code perform.

28 Mar 2005 (updated 28 Mar 2005 at 15:04 UTC) »

Finally got around to creating a blog. Hopefully I'll be posting with some regularity here.

I'll begin with my current activities: I've been busy these days trying to make Postgres use shared locks, not exclusive locks, in foreign key checking. This is not as easy as it sounds, because the current locking system is limited in available memory, which means the current row-locking code does not use it at all; instead it marks tuples directly on disk, which in turn means we can't have more than one locker at a time. Hence exclusive locking. Moreover, the heap access method is not at all prepared to deal cleanly with multiple lockers. (I also want to get rid of duplicated code in the heap AM, which confuses me because the duplication is not verbatim and so it needs slight adjustments before refactoring can be done.)

So, I've been mostly reading existing code and dumping one solution after another. But eventually I'll come to it — in fact, I think I found a real solution this time that I expect to be able to try tomorrow.

Additionally, I submitted a patch to allow tracking dependencies on shared objects (users, groups, tablespaces). This means that we will be able to deny dropping a user which owns objects in other databases. IMHO this is not a very exciting new feature for the database, but for me it means correcting annoying and buggy behavior. Tom Lane found the patch missing in several respects; I'll be correcting it as soon as I get rid of the shared-row-locking issue. Hopefully I'll be able to finish both things before 8.1.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!