Automated Python testing with nose and tdaemon

If you're testing your code at all (and you are, right?), it's awfully convenient to automatically have your test suite run whenever something in your project changes. This is particularly handy if you're doing test-driven development (TDD), where you'll be writing a lot of tests and need immediate feedback on them.

With Python this is made easy with the help of nose and tdaemon.

In a hurry? Jump straight to the summary!

Installing

Both tdaemon and nose can be installed via Cheeseshop (or "PyPI", for the suits), using pip:

  sudo pip install nose tdaemon

(You might not need sudo, depending on your setup).

An interlude: pip versus easy_install

You can use easy_install instead of pip. It should work just fine, for what we're trying to do here. But do yourself a favor and switch over to pip. It's as simple as

  sudo easy_install pip

Another interlude: whither tdaemon?

If you google "tdaemon", the first result is a github tree from Bruno Bord, tdaemon's author. The version at Cheeseshop (yes, I'll keep calling it "Cheeseshop", damn it!) lists John Paulett as author and points back to the github repository as its home page. Both versions are almost exactly alike, except that the Cheeseshop one has a slight enhancement (a command-line argument to ignore specific directories). John Paulett took tdaemon, added that feature and packaged it for Cheeseshop. So we'll use his version.

Getting cute notifications

So far, if you simply execute tdaemon on a terminal, it'll monitor the current directory and run nosetests whenever it detects a change. Which is fine, but I don't want to switch to the terminal all the time while I'm programming, if I don't have to. So let's arrange our environment so that we get visual alerts every time the tests are run.

On Mac OS X

I assume you already have Growl on your Mac. If you don't, install it, it makes your life easier (if you don't know, it should be at the bottom row on your System Preferences panel).

The NoseGrowl nose plugin provides Growl notifications. Unfortunately, NoseGrowl installation is currently broken: there is only an egg for Python 2.5 on Cheeseshop, and the source code on bitbucket has a bug on setup.py. But the latter is easy to fix:

  hg clone http://bitbucket.org/crankycoder/nosegrowl
cd nosegrowl/nose-growl
# This next command simply removes a specific line from setup.py
# If you want to, edit setup.py and remove it yourself
# (and blame OSX for not shipping GNU sed)
echo -e "/install_requires=\['py-Growl'\],$/d\nw" | ed setup.py
python setup.py install

By the way, NoseGrowl's author, Victor "crankycoder" Ng, told me a few months ago that he was looking for someone to take over for him. So, if you find the project useful, please consider talking to Victor and volunteering to maintain it. Might make him less cranky :)

On Linux (Gnome)

Michael Gundlach adapted NoseGrowl to use Gnome's notification system. His version is called NoseNotify and can be installed directly from Cheeseshop:

  sudo pip install NoseNotify

I don't know if there is a notification plugin for KDE. If you do, let me know and I'll add it here!

Putting it all together

Open a terminal window and cd to the root directory of your project (tdaemon recursively looks at everything down from there).

If you're on OS X, type:

  tdaemon --custom-args="--with-growl"

If you're on Linux, type:

  tdaemon --custom-args="--with-notify"

Since you're passing custom arguments to nosetests, tdaemon will ask you to confirm that this is the command you want to run. Simply type "y".

Now you can create a file called, say, "tests.py", add tests to it and watch what happens as you save it!

 

Summary

Install nose and tdaemon:

  sudo pip install nose tdaemon

On Mac OS X, install NoseGrowl:

  hg clone http://bitbucket.org/crankycoder/nosegrowl
cd nosegrowl/nose-growl
# This next command simply removes a specific line from setup.py
# If you want to, edit setup.py and remove it yourself
# (and blame OSX for not shipping GNU sed)
echo -e "/install_requires=\['py-Growl'\],$/d\nw" | ed setup.py
sudo python setup.py install

and run as:

  tdaemon --custom-args="--with-growl"

On Linux, install NoseNotify:

  sudo pip install NoseNotify

and run as:

  tdaemon --custom-args="--with-notify"

That's it. Happy coding!

Syndicated 2010-08-01 17:57:23 from Bits of rbp - isnomore.net

Yahoo! Open Hack Day Brasil 2010

On March 20th and 21st, Yahoo! Brazil brought us our second Open Hack Day. I'd been to the previous one, in 2008, and it was amazing! Our project even won at a newly-created category, aptly named "What the Hack?"

This year, I wanted once again to try a hardware hack, using whichever parts I could get my hands on. Not necessarily anything useful, though. That's what I love about the Hack Day. I can do useful stuff throughout the rest of the year :)

Yahoo! Hack Days

Before the 2008 Hack Day, I had pretty much written off Yahoo! as a company that was no more. I wasn't even particularly interested on the event, and only decided to go at the last minute. Boy, what a change in perspective. Of course, Yahoo!'s São Paulo team has some very clever people. But, more generally, I was very impressed with the data-gathering tools that Yahoo! had started offering. YQL is simply fantastic. It perfectly captures what the Internet is about, data-wise (I'm not saying it's perfect, but it embodies the right spirit). I hadn't had that much geek fun in a long time. So you can probably tell my expectations were high for this year's edition. Could Yahoo! deliver?

Not to fret, they knew what they were doing. Just put some 250 hackers in a fish bowl, give them food, coffee, wifi (surprisingly good, some silly proxy restrictions excluded), show them Monty Python, and wait for it!

Our hack, and our hackers

Even before the announcement of this year's Hack Day, I'd been toying with the (admittedly silly) idea of a firefighting robot, that lurked online waiting for people to report fires anywhere in the world. It would then bravely roll over to wherever the fire was, and put it out. Bravely.

Trouble is, it'd probably have to be one gargantuan robot. So I thought I'd settle for a more modest, Lego-built, Arduino-controlled one. With the Hack Day approaching, I suggested this project to a few friends and we created a Wave to discuss the idea. I was planning to use a box of old Lego pieces from my childhood (see mom? I told you I'd have eventually use those again!) and a couple of servo motors I had lying around, but Rodolpho brought his Mindstorms NXT into the picture, making the project much cooler (and, incidentally, much more manoeuvrable).

On Saturday morning I rode with Gola to the (really cool) auditorium of Senac University, where the previous Hack Day had already been hosted. There we met Rodolpho and Mobi, our original team. We had found out the day before that there would be a limit of 4 people on each team, this year. In 2008 our team had been comprised of... 12? 15? I never even knew. We'd simply started building weird, blinking stuff, and people had gathered around for the fun. We got, therefore, a bit disappointed with this edition's limit. But, since we weren't really expecting to win anything (and therefore disqualification wasn't an issue), we bent the rules a bit, and Werneck, Lucmult and Mauro hacked with us. Also, Aline joined us a bit later. Since Werneck and Lucmult eventually had to leave and didn 't return for Sunday, and Aline and Mobi didn't program, I think we were sort of in the clear. Technically. Sort of.

And then we started building.

What we built

The body of the robot ended up using mainly the Mindstorms parts. We had started building a larger, sturdier body with assorted Lego pieces on top of a rigid board, pulled by front-wheel drive with four wheels in the front and a loose trailing one. But, after a lot of testing, the loose wheel kept veering the robot out of track, so we rebuilt everything with a lighter, smaller frame, and a pair of caterpillars tracks. In hindsight, I think we should have kept the larger body (even if rethinking wheel traction), as it gave the robot more stability, which we would come to miss later. But we had to make a quick decision, and we worked with what we had. The NXT controller sat on the robot and communicated with an external server via bluetooth, relaying control input to the wheels and sending back odometry information.

This external server (actually, Rodolpho's notebook) continuously used YQL to query Twitter, Yahoo! Meme and news feeds (which I originally wanted to aggregate using Yahoo! Pipes, but we didn't have time for it), searching for people reporting fires - a simple string search, filtered by the Yahoo! Term Extraction API. Whenever there were reports, the server would send them through the Placemaker API to extract location information. It then determined which location in the world was in most urgent need of aid (by number of reports).

Now, the tricky part: the robot needed to be aware of a rectangular projection of the world onto the room, and to know its own location on it. I don't mean a visual projection. We did consider it, but realised that it would be unfeasible for the Hack Day. We toyed with the idea of printing (or drawing) a world map on large sheets and taping then to the ground, but the robot would surely slip, trip, or tear the sheets. So we settled for an abstract projection, and, based on information sent back from the robot, plotted its estimated current position in the "world", at each instant, using Yahoo! Maps and the Yahoo! Geocoding API (via geopy).

To locate itself, the robot relied on Mindstorms' odometry feedback, and on QR Code markers distributed around the room, read by an Android phone using Python (from the ASE) and the Barcode Scanner app. The Android phone sent the information encoded in each QR Code to a custom service built with Python's SimpleHTTPServer, which our server polled. The server then sent back new movement controls, according to the robot's current position and desired target. We tried to use Mindstorms' sonar and colour sensors, but they just wouldn't work reliably with the Python interface (which we needed to send commands programatically via bluetooth).

No one likes an impersonal robot, and ours, accordingly, had a face in order to convey emotions. We used an XO laptop as the head, and defined that the robot would have pre-determined emotions depending on the situation: it would be "at rest" when there were no fires going on, "worried" when it was going towards a fire, and "happy" when it had put the fire out. A Python script running in the background continuously searched Flickr (using YQL) for expressions associated with each of these emotions, and downloaded a number of related pictures. Another script queried the robot's current emotional state (set by the server), and displayed an appropriate random subset of these pictures. For every few of them, we'd display a face that Aline had drawn specifically for that emotion, to make up for the fact that we couldn't be sure if the pictures would depict it (people give the weirdest tags and descriptions to their Flickr uploads...).

We originally meant to use an Arduino to control a servo that would squish some water from a syringe, thus putting out the fire once the robot got to its destination. But all that proved too much for a mere 24h, and we had to settle for a manual pump.

A camera-shy robot, and a sleepy presenter

Of course, we'd tested (almost) everything, and the robot worked (almost) perfectly. (Almost) Great. But, when it was time to go up on the stage, we hit a few snags.

First, there was only one large screen. Of course, we'd known that, but it hadn't occurred to us that people would need to see not only the robot (which was tiny, especially at a distance), but also the map showing its movement towards the fire, at several different moments. Yahoo! had predicted even such an eventuality (seriously, guys, kudos!) and provided us with a way to switch between our server's screen and the feed from a video camera which Aline used to film the robot. But I didn't manage to coordinate the switch properly, and we didn't get to show the animated map.

Also, we had already noticed that the XO weighed a lot compared to the rest of the robot (remember what I said about the previous, sturdier version?), and that, in order to steer, we needed to balance it carefully. I didn't, and the robot soon went out of its path.

Finally, we had used so many different technologies and APIs on this project, and most of them simply vanished from my mind when I started presenting! Note to self: next time, write some hints on the back of my hand...

All I have to say for myself is that I had slept for only about 1 hour since the previous morning, ours was one of the last projects to be presented, and by then I was very sleepy and barely able to react. I should have switched from the camera to the map more often, so people could see what was going on. I should have grabbed the robot when it went off track and restarted the demo. I should have asked my teammates for help when trying to list everything at work on the robot. Shame on me.

But, all in all, I think it was a great project, and I'm very proud of it. I'm especially proud of my team, you rock much more than I was able to show :)

And, next year, I promise to take a nap before presenting!

Syndicated 2010-04-05 22:26:24 from Bits of rbp - isnomore.net

Google Wave invite giveaway result!

Thanks to everyone who participated in the Google Wave invite giveaway!

I've just drawn a random name from those who followed the giveaway rules (on Google Reader, "like" one of the other two posts on this site, and share the giveaway one). I've recorded the drawing process, so you can see it was done in all fairness:

*[I had embedded a video here, but some people reported problems with it. So now here's a screenshot, and you can see the video separately, if you want; sorry]*

Congratulations, Sviatoslav S, you get the Google Wave invite! Please add some contact information (at least temporarily) to your Google profile, so I can send you your invite. If you have any questions please send me an email or leave a comment on this post. I hope to hear from you soon!

[Update]

It turns out I do have a second invite! And, ironically enough, I found it quite hard to get rid of it...

Once I found out I had the extra invite, I quickly drew another name from the list:

Congratulations, Maurizio! You... Oh, wait. Apparently he's not following the rules of the giveaway. I can't really tell whether he never has or if he gave up after the first invite went out, but rules are rules. So let's draw another one:

Hooray! Brian was one of the first to participate when I announced the giveaway. So I contacted him. But he had already been invited by somebody else, and graciously allowed me to hand the invite over to another participant. So...

Right. Daniduc is already in Google Wave. He had actually "liked" and "shared" the posts before I announced the giveaway, and had already told me he wouldn't need the extra invite. Next!

Ah! I'd talked to Leonardo just yesterday, and he didn't have a Wave account yet! So congr... Ah, he got his from someone else this morning. Right. But the fifth one...

So that's what you're going to get, Badá! The strongest castle in these islands A Google Wave invite! I've already added you to the list, hopefully Google will send you the invite soon.

As for the others, thank you so much for participating!

Hope to see you all soon on Google Wave :)

Syndicated 2009-10-25 02:18:25 (Updated 2009-10-25 03:18:25) from Bits of rbp - isnomore.net

Giving away one Google Wave invite (or two)

[A quick note for the lazy: read at least the "But how?" section below. It explains what you need to do to qualify for the giveaway]

So, Google Wave is quickly reaching a stage of being actually useful for daily communication. I've used it for planning trips and dinner parties, coordinating server maintenance, doing collaborative translation and just generally chatting, with varying levels of success.

I got my invite to the sandbox (a developer preview, not integrated with the rest of Google's services) after attending Google Developer Day São Paulo. I played around for a while, but it was too unstable. A few weeks later it had improved considerably, so I started using it again. But too few people I knew were there, and its use as a communications tool was therefore obviously impaired.

A few weeks ago, then, Google promoted Wave to http://wave.google.com, and there was much rejoicing. As a sandbox user, I got an invite to it, and with it 8 more to give away. I distributed them to a few close friends, but ended up with one left!

And that's the one you might get :)

"But how?"

[updated about 1h after originally posting, to correct the rules]

It's really simple. After a long period of inactivity, I've resumed writing on isnomore.net. There are currently two posts (not counting this one): one about programming Python on the Android, and a recipe for Spaghetti alla Carbonara. So, access isnomore.net's RSS feed on Google Reader and pick the one you like best. Then "like" it (by clicking the proper button at the bottom of the appropriate post, on the Reader screen), and share this post (the one you're reading right now).

To make it clear: you should "like" one of the other two posts currently published ("Pushing up Python on Android" or "Spaghetti alla Carbonara"), and share this one ("Giving away one Google Wave invite (or two)"). You may also share and like others at will, but that's the minimum to qualify for the giveaway.

Important: if you "share" and "like" from someone's shared items, or some other address you might have (if you used to subscribe to this site's old feed, for instance), it won't show up on my count and I can't know you've done it! So the safest bet is to go straight to the feed on Google Reader and choose from there (like so). In any event, if you feel that there is any confusion, let me know.

Within three days (that is, on Saturday the 24th of October) I'll collect a list of the people who've liked any of the articles and pick one at random. If that person has also shared the article, they get the invite (otherwise, I'll just pick another one from the list, and so on). I'll post the results here on Saturday, and I'll also contact whoever was selected via their Google account (if you want me to use a different email, let me know in the comments or "Share with note" and leave a comment to that effect there).

"What if I share and like both articles?"

Well, you'll make me very happy, please do! But it won't improve your chances, I'm afraid.

"I already have a Wave account (or don't want one), but I'd like an invite for a friend"

Sure, no problem. If you are selected, just let me know the email to which I should send the invite.

"But I don't have a Google Reader account!"

Well, a Google Reader account is just a regular Google account. If you've never used Reader, just give it a try, it's really simple. If you don't have a Google account at all, you'll need one for Google Wave anyway, so you might as well create it now!

"If I'm picked, how long until I have a Google Wave account?"

That's not up to me. Google Wave invites actually mean Google will know that someone invited you when the time comes for them to offer a new round of accounts. But they don't say when that will happen, or even whether everyone on the current invite list will get an account on the next round. In practice, accounts have been handed out at a relatively high pace, but I can't be sure.

All I can promise is that I'll put your email address on my invitation list as soon as you confirm it.

"Wait, the title said or two!"

That's right. I might have an additional invite. It's a bit of a long story, but I can't be sure. So, if it turns out that I do, I'll pick two people. Otherwise, that's just one lucky winner :)

"But... But... I am your friend and didn't get an invite from you! I even asked on Twitter!"

Calm down, don't be offended. First of all, most of my invites went to the people with whom I exchanged the most emails and instant messages. Makes sense, right? Besides, some of the people to whom I had promised invites ended up not needing them (having received one by other means), so I didn't know from the start I still had invites left.

Finally, I specifically avoided giving them out to people asking around on Twitter (as opposed to asking me personally). Asking in broadcast mode meant that, by the time I invited you, someone else might have also done so. From my experience with the other people I've invited, multiple invites don't seem to make your account arrive any sooner. As Google Wave invites are currently somewhat scarce, that would be a waste of one, which would be a shame. And I want the highest number of people on Wave, the sooner the better.

"I have more questions! You can't predict and fake-quote them all!"

True. Leave me a comment!

And good luck! See you on the Wave :)

Syndicated 2009-10-21 03:39:36 from Bits of rbp - isnomore.net

Spaghetti alla Carbonara

Carbonara is widely known as a simple recipe, but for the longest time it seemed a bit mysterious to me. I'd watch a friend mix a few ingredients together, in what seemed destined to become some sort of spaghetti omelet, only to be presented with a beautiful - and delicious - meal. Then one day I came across David Leite's recipe. It seemed simple enough. Well, it seemed short enough that it'd be simple. So I tried it.

And, well, it wasn't bad, but came out a bit clumpy and dry. So I tried again. And again. And read some very helpful comments on Leite's article. And found some variations of the recipe. And experimented a bit. So now I'm finally happy with my Carbonara, and hopefully you'll find it as easy and rewarding as I do! :)

I used to make it with bacon, which is ok but sort of overwhelms all other flavours in the dish. I've recently purchased some excellent pancetta and let me tell you, Carbonara just isn't the same without it. Actually, apparently Carbonara is supposed to be made with guanciale, but I haven't managed to lay my hands on some yet (But I will. I will). So use pancetta if you can, but don't let that stop you otherwise. Bacon is fine. Or try both and let me know if you can tell the difference. And do drop a line if you use guanciale!

Another small but significant detail: I've found many recipes that use only Pecorino cheese. However, David Leite uses a 3-to-1 mix of Parmigiano-Reggiano (or Parmesan) and Pecorino. I usually cook Carbonara with Parmesan (it's more easily available around here), but the Parmesan/Pecorino mix is indeed more flavourful. However, I think using only Pecorino would, again, come out too strong and break the balance.

So let's get to it. The quantities are good for 3 people (or two, with repeats; at least that's how it goes around here). It takes about 30 minutes from start to finish, but allow a bit more on your first try.

Enjoy!

Syndicated 2009-10-19 14:10:41 from Bits of rbp - isnomore.net

Pushing up Python on Android

A few days ago, I put on the manliest voice I could muster and made an announcement to my wife: "Stand aside, woman! I am going to the gym!"

Needless to say, she was thoroughly unimpressed but somewhat amused when, half an hour later, she found me at the computer, programming.

Despite what she might tell you, I hadn't given up on exercising. You see, I had recently taken up the One Hundred Pushups, Two Hundred Situps and Two Hundred Squats programs. These involve a few sets of a varying number of repetitions each, with timed pauses between each one. So, for instance, on my first day I'd do 10 pushups, rest for 60 seconds, do 12 pushups, rest, 7 pushups, rest, 7 pushups, rest, and finally as many pushups as I can (but at least 9). These numbers of repetitions vary as you progress. My problem was keeping track of how many repetitions of which exercise to perform on any given day.

Now, there are nice PDFs with the whole exercise program on each site, but they're supposed to be printed. On paper. How low tech! Some people use spreadsheets, but... Meh. So I decided I should turn to my Android phone for help.

There is an iPhone app for One Hundred Pushups et al, and at first I considered writing an Android app to match. And I still do, but, of course, that's a full-on project, one that would definitely not be usable in time for me to exercise that night. So: pragmatic program, must be running in a very short time (my wife was laughing out loud, by then) and improve as need arises. This looks like a job for... Python!

Python is not (yet?) a first-class citizen on the Android, but it's a respectable second-class one, thanks to Damon Kohler and his Android Scripting Environment. ASE lets you run several interpreted languages on the Android, amongst them Python, Lua, Perl and JRuby. However, these are limited on what they can access on the Android API. More specifically, you can't build arbitrary user interfaces or create new activities (though you can invoke existing ones).

Still, having a Python interpreter on your mobile can be handy. I needed to input three sets of repetitions (one for each exercise program), and have Android let me know how many repetitions to do next, and for how long to rest between them. I'm still meddling with this code (trying to weigh making it better versus building a proper app versus actually, you know, exercising), this is just a quick hack I cooked up to get going, but it's growing on me. Anyway, here it is:

from time import sleep
import android

droid = android.Android()

# How long to rest between repetitions
rest = 60
# Warning before starting next round
wake = 10

# One line per exercise set, each number of repetitions separated by spaces
# For instance (pushups, situps, squats):
# 10 12 7 7 >=9
# 9 9 6 6 >=8
# 19 24 19 19 >=27
user_input = droid.getInput("Series", "Describe all repetition sets in your series:")
series = [i for i in user_input["result"].splitlines() if i.strip()]

def interval(theres_more=True, rest=rest, wake=wake):
    droid.makeToast("Rest for %d seconds..." % rest)
    sleep(rest - wake)

    if theres_more:
        droid.makeToast("Ready? %d seconds to start!" % wake)
        droid.vibrate(500)
    sleep(wake)

    if theres_more:
        droid.makeToast("Go!")
    droid.vibrate(3000)

for s in series:
    droid.getInput("New series!", "Ready?")
    sets = s.split()
    l = len(sets)
    for n, repetitions in enumerate(sets):
        droid.getInput(repetitions, '(press Ok when finished)')
        interval(theres_more=(n+1<l))

droid.makeToast("w00t! Congratulations!")

(you can also download it here)

As explained in the comments, it initially expects lines containing the number of repetitions on each set. So, if I'm undertaking pushups, situps and squats (respectively), I might input:

10 12 7 7 >=9
9 9 6 6 >=8
19 24 19 19 >=27

Of course, ">=9" is not a number, but the script will use whatever you input there as labels for prompting you to perform your repetitions.

You'll notice that the script uses getInput for displaying messages when it expects the user to press "Ok" (even though it doesn't expect any typed input at all). That's because, currently, getInput is the only graphical widget provided by the the Python proxy for the Android API. But more on that later.

So, try it out (if you're willing to exercise at all, or if you're just curious), and let me know what you think! Did it help you exercise?

Syndicated 2009-10-07 19:08:31 from Bits of rbp - isnomore.net

Bleeding hell, it's been a long time (more specifically, since a time when advogato didn't have password recovery)... "Update advogato" >> GTD.Inbox

Four posts, all from 2000. Ok, we can all see where this is heading. I officially pronounce this advogato blog dead. There should be more frequent posts at http://isnomore.net/blog

Ok, a bit late for that, but oh well...

Everything happened at Comdex: I lost my luggage on the plane to Chicago, the booth wasn't ready, the other girl from Conectiva that went there had trouble with her reservation at the hotel... But, when we finally got things ready, it went ok. All that trouble kept me busy with solving problems and less with enjoying he show, but I think it was a positive experience...

I'm almost on my way to Comdex Sprint (Chicago - USA)!
Last week I was asked on wednesday if I'd be available to go to the US the next saturday, for CA World and stay there until Comdex. Wow! Can I pack first? ;)
Then they (Conectiva HQ) told me someone else would go to CA, but I'd still go to Comdex. Cool :)
It seems I'm going tomorrow night. Yes, I still don't know.

See you at Comdex :)