Recent blog entries for badger

Yesterday I deployed a new version of the PackageDB. This is becoming a rather regular event due to the fact that maploin is working very hard on a lot of important UI improvments. For this update we have the answer to one of our commonly requested features: ability to filter the package list by letter.

A few weeks back, at the prompting of Lyos Norezel, I updated the orphaned packages page in the packagedb to display the releases a package was orphaned for. Unfortunately, I managed to break sorting of the packages at the same time which made things a bit unusable.

Now that that's been pointed out, I've fixed things to be alphabetical again:

Flatten a list in a python one-liner

I sometimes need to flatten a list in a list comprehension or generator expression (usually because I'm doing work inside of a template where keeping temporary variables is somewhat painful). This has stumped me for a while but today I found itertools.chain() which worked wonderfully. Here's how I found out I could make use of it:

>>> import itertools
>>> data = [(1,2), (3,4,5), (6,7)]
>>> [i for i in itertools.chain(*data)]
[1, 2, 3, 4, 5, 6, 7]

New Election Software

As Fedora folk know, the Fedora Board Election is currently taking place. What people might not all be aware of is that the election software has been completely rewritten by one of our all-star contributors, Nigel Jones. Since I wrote the former election software, I'm well aware of its shortcomings: a manual process for setting up an election, no confirmation pages, limited styles of voting, only a single election at a time, and it just plain looked ugly. Nigel looked at all the problems and wrote a new app in TurboGears that fixed them. Not only does it solve all the problems we had before but it gives us a much more flexible framework for further enhancements. If we want new styles of voting or a new Fedora theme, it will be much easier to implement that with the new code.

So if you're on one of the Fedora IRC channels, give a warm thank-you to "G" for the excellent work he's done!

Why I Love Open Source

I've had a bunch of items on my TODO queue for the pkgdb that would be major enhancements to its usability but haven't been able to work on them due to higher priority things constantly coming up (FAS, figuring out why common operations are so slow, eliminating unicode crashes, mass branching, etc) This doesn't mean that I don't know there are usability issues with the pkgdb, though. And every time I see those issues I have to shudder and promise myself that I'll have time to fix them soon.

Well, for at least a few of those most requested features that's no longer necessary! Christopher Aillon (caillon) submitted a patch to add UI to the user package page to filter packages according to whether you own them or not. This is great as it means I'll be able to set the default filter to owner, approveacls, and commit and people will be able to easily change the settings themselves.

An even bigger enhancment is being worked on by Ionuț Arțăriși (maploin) who has recently started hacking on the PackageDB. He's adding search capability so that we can find packages by keyword a la yum search. Because of the infrastructure change freeze that we're enforcing until Fedora 9 is out the door, I don't have a test server running his code yet. If you want to try it out, though, it's available by checking out the bzr repository:

bzr branch
bzr://bzr.fedorahosted.org/bzr/packagedb/mapleoin-devel

Instructions for configuring a test server are in the README file.

Here's a few screenshots to whet your whistles:

Mugshot to the Rescue!

While doing other things after dinner I happened to see this entry go by in my mugshot stacker and thought, hmmm... I should look at that. One click later I was looking at the suspect changeset on fedorahosted's trac instance and confirmed that it would break bugzilla sync. A few minutes of editing and a fix was committed.

Moral of the story #1: When set up to watch the RSS feed for your project's commits mugshot can help get the right eyes on a problem quickly.

Moral of the story #2: If you have any doubts about a change make sure to mention them in the first line of your commit message.

Does it say good things about inkscape or bad things about the other programs we have available that it's what I use to model objects and relationships when I program?

Mmm... Matzo ball soup and roasted tomatoes.

Thanks to Mike McGrath and a host of other Fedora Infrastructure people, the new Fedora Hosted server has better support for Bazaar Repositories than the old one! Under the old setup we had sftp for authenticated checkout and http for anonymous checkout. In the new scheme we have smart servers replacing both of those.

Here's a bit of preliminary timing to show you the difference. Note that these times have several inaccuracies and should only be used as an extremely general indication of speed. A few caveats that I know of:

• These times don't test push, merge, and pull speed, which are the networked operations that you'll be using most frequently. They only show the inital branch operation (which, however, is the first impression that most people get of an SCM's speed.)

• rsync doesn't understand the bzr repository structure so I have to download the entire repository with rsync instead of just the branch I'm interested in. In this case, the difference is 4876K for rsync vs 4556K for the bzr protocol.

• Although I have a bazaar repository setup on the server, I don't have one setup on the client. In normal operation, I'd already have most of the data checked out in a repository in a different branch so a new branch would have to transfer a lot less data.

• This test is performed with bzr's dirstate-tags branch format. The latest bzr has a new "pack" format that is supposed to be faster. We currently have 1.0rc3 on the server and I'm running 0.92 on the client. I'll likely be experimenting with the new format after I get 1.0 installed on both sides.

Baseline rsync This runs rsync and then does a lightweight checkout in the directory (leaving us with a branch and working tree just like a bzr branch operation.)

rsync -a rsync://bzr.fedorahosted.org/bzr/packagedb/ .
cd fedora-packagedb-devel
bzr checkout --lightweight 

 
 real    0m16.548s

bzr over http This is equivalent to what we have had to use on the old server:

bzr branch
http://bzr.fedorahosted.org/bzr/packagedb/fedora-packagedb-devel
Branched 258 revision(s).                                  
                  
real    1m2.844s

Ouch!

A fresh branch using the smart server

bzr branch
bzr://bzr.fedorahosted.org/bzr/packagedb/fedora-packagedb-devel
Copying repository content as tarball...
Branched 258 revision(s).

 
 real    0m23.192s

Much better! This is only 7 seconds slower than using rsync.

bzr+ssh smart server for rw access:

bzr branch
bzr+ssh://bzr.fedorahosted.org/bzr/packagedb/fedora-packagedb-devel
Copying repository content as tarball...
Branched 258 revision(s).

 
 real    0m25.501s

The ssh protocol adds a few seconds to the total time but it's also quite speedy.

One interesting point is that the smart server times are much better than the ones I posted in October. This could be because we're dealing with a different server on a different network or it could be that we've gone from using bazaar-0.18 to bazaar-1.0rc3 on the server. I'll have to do more timings after we get settled into the new machine to tell more.

My last week was largely spent working on Fedora Infrastructure tasks: fixing a lot of annoyances, creating scripts to automate tasks, and helping to get the new hosted infrastructure up and running. This week has started off the same but I'm hoping to get some work done on some new authentication code for python-fedora before next week starts. J5, Ricky Zhou, and I have been talking about what it would take to get single sign on, certificate authentication, and openid working and I want to start modifying code to make that happen.

Note: I did get to cross one of my PackageDB tasks off without doing much of anything -- porting to RHEL5 turned out to only be a matter of getting a few packages branched for EPEL. No coding required.