Advogato blog for quad

bzrshelve, a punchline to a bad joke

Tue, 17 Jun 2008 09:06:03 GMT

The joke has been long coming.

Back when I was still on reddit, a short meme hit where someone wrote a little hack that made frontpage. The title is what must have sold it, as there wasn’t any there - there.

“Using Git as a versioned data store in Python” aka gitshelve.

A few days later, of course, hgshelve came into existence.

It’s telling that the Bazaar community never got into the action. I can imagine good arguments for both that scene being too small, or too busy getting work done.

Fortunately, I have no such issue. Behold: bzrshelve.

And the only DVCS that can get the source is svk.

Happy Key Revocation Tuesday

Tue, 17 Jun 2008 09:06:03 GMT

Almost one month ago, Florian Weimer on behalf of the Debian Security Team announced one of the worst security vulnerabilities in recent history. I won’t go into a technical description of the problem itself. But, it’s interesting to note how Debian both succeeded and failed, how this vulnerability broke the “patch to stay secure” model, and how it personally impacted me.

On Debian…

First, Debian is an all volunteer organization that created and maintains the largest integrated body of code. Ever. The Debian “operating system” is far larger than Microsoft Windows or Mac OS X - they can barely be compared. That a security vulnerability could lay in any package undiscovered for years is unsurprising.

But, once discovered, Debian’s security team promptly released an update of the affected packages fixing the flaw. In the same announcement for the update, there was an included link to a page that promised to have instructions on how to actually close the holes. That page wasn’t filled in until over a day later.

Of course, the wiki page had helpful information within 30 minutes.

Are you saying getting the security update didn’t fix my computer?

Yes. The problem wasn’t a matter of fixing the user’s software but fixing their data. The security keys they thought weren’t. The software to make new keys was provided; but, any Debian user that wasn’t subscribed to the right mailing list wouldn’t have known about the further action necessary. (Though, to be fair, the OpenSSH package at least warns about vulnerable keys on update.)

In fact, the average Debian user would be hard pressed to find any mention of the vulnerability. It wasn’t a front page news item. OpenSSL, and all dependent packages, fail to provide any alert on upgrade. Worse, the Certificate Authorities still haven’t revoked certificates for compromised keys. That means the SSL aura of trust has been devalued even more.

It would be an interesting, and expensive, experiment to see how many CAs will EV sign one of the compromised keys.

On me…

Meanwhile, tonight, I finally finished with “key rollover” on all my affected services.

tara: No services effected. (Too old.)
steak: No services effected. (Too old.)
megan: SSH, SMTP / IMAP, XMPP
resa: SSH
Personal keys: EECS, wsunix, Planet EECS, tara, megan, nearlyfreespeech

Gosh, I hope I got everything. Each of those only took about five hours apiece.

Of course, some people did make it easier. I already shouted out to the wiki page earlier. But, of everything and everyone who should have been doing their jobs, one group stood out and another one embarrassed itself:

From: “NearlyFreeSpeech.NET Member Support”
Subject: [NearlyFreeSpeech.NET] Potentially weak ssh key detected
Date: Wed, 14 May 2008 12:30:00 -0400
Hello
You are being contacted because an ssh key vulnerability in Debian-
derived Linux systems has been detected that may affect you.
…

Wow. Thanks!

From: “XMPP CertMaster”
Subject: XMPP SSL Certificate revoked, 09:12 pm 13 Jun 2008
Date: Fri, 13 Jun 2008 21:12:48 +0300
This mail is intended for the person who owns a SSL Certificate from the XMPP Intermediate Certification Authority (http://www.xmpp.net).
Your certificate with serial number 890 has been revoked for the following reason(s):
- The holder / owner of the certificate requested revocation.

You can’t blame the XMPP Federation. They don’t actually run a CA, they subcontract. I hope Peter isn’t paying much… as I’d say him having to write a notice of the vulnerability was not his money’s worth.

Free shiz

Tue, 17 Jun 2008 09:06:03 GMT

I feel kind of like a jerk of a friend, not mentioning this earlier.

Microsoft has this deal called “cashback from Live Search.” The idea is that you search using Live, and from select stores you’ll get sweet savings.

One of those select stores is eBay. And, the sweet savings is 35% off for up to $250. Three times.

Use Live Search.
Search for “zune”
Click the “Live Search cashback” link in the Sponsored Sites for eBay.
Ok, now you have 60 minutes to buy whatever on eBay. some terms and conditions apply...

Meanwhile, I ♥ you, in particular.

In, quite possibly. my most boring post yet.

Tue, 17 Jun 2008 09:06:03 GMT

About wasting less time and blogging more... it’s easier said than done.

Adding personalities was an experiment. I got a view into the drama of A-list and B-list bloggers. Nice for them; but, mostly an incredible waste of time for me.

In hindsight this should have been obvious. No one, except me, is interesting 100% of time. This is an existence proof for my LiveJournal.

But, some people are excellent filters for interesting things. Anarchaia, the first tumblelog, is what made me think there was value in subscribing to personalities. Well worth reading if you’re smart or a little computer sciency.

None of this helps me. I need to feed my mind with the latest happenings. My environment is my inspiration.

Except, I have constructed a perfect echobox of Web 2.crap. Ok, new rules:

No more visiting Facebook. Or Reddit. Or Hacker News.
Google Reader has comics. And people I know.

When combined with the previous purges, I’m left with close to no input from the Internet. IM and e-mail. Did I mention I switched IM accounts? Did I mention I unsubscribed from mailing lists?

And, yes, it’s difficult to rationalize being subscribed to people’s blogs when I won’t read my friends page.

This should leave plenty of room for real life. And blogging about it.

I will never be a software architect

Thu, 8 May 2008 10:11:55 GMT

Disclaimer: this may be be a Seattle area phenomenon.

I have “software architect” on my resume, and it pains me. Wikipedia has a great article on what a software architect may or may not be. But, in my world, a software architect has the knowledge, insight and responsibility to make educated decisions about the scope and direction of a team-developed software project.

That was a mouthful.

Software architects pick frameworks. They find previously existing packages for functionality just before the rest of the team realizes they need it. And, they plan and communicate how all the moving parts will come together. They’re really-really smart.

Everyone wants to be a software architect. At Seattle’s Startup Weekend, no less than a third of the developers signed up as architects. And why not?! The act of creation - from art to programming - is egotistical. If you’ve ever referred to yourself as a “software engineer” with a straight face, then you’re advertising the capability to plan non-trivial projects.

You’re a liar.

Software is big. You just won’t believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it’s a long way down the road to the chemist’s, but that’s just peanuts to software.

With all apologies to Douglas Adams. Software projects are the most complex machines created in the history of invention. You’re telling me that you can do better than Leonardo Da Vinci, Thomas Edison, or the Wright Brothers? Because each of those iconic figures were geniuses driven to create simpler machines than a web application. And each was wrong up front.

This isn’t a fair comparison. We have Photoshop, Digi-Key, and kit airplanes. Also, Rails!

Those inventors were forging into unknown territory. Customizing a CMS or integrating SAP ERP into a SOA are known quantities. It could be argued the architect exists for the partially ambiguous problems.

My response is a question oft head in agile circles. I learned it from working in open source projects, corporate giants, startups and contracting. It’s a kōan:

“How will your program work in six months?”

The job of software architect is an answer. Is it the right one?

There is value in understanding a problem domain.
But, the stakeholders in a project tautologically have that.
There is value in making the hard decisions.
But, that is why we have team leaders.
There is value in planning your design.
But, software structure inevitably resembles its team’s structure.

… and so on.

The software architect exists because of the cultural need to have someone be responsible for these aspects. But it isn’t possible to satisfy these responsibilities and simultaneously attend to the details that inform future decisions. Architecture astronauts just don’t have the time to be any more grounded!

Instead? Go slow. Let the programmers make the decisions. Feed them knowledge and constraints. Try to develop a consensus among the actual stakeholders. And accept everyone’s input. That quiet intern? They go home and spend all their spare time playing with tools that handle 80% of the job.

I’m not arguing for agile development practices.

I’m arguing for considered diligence. Plan a little. Work a little. Rinse and repeat. Never let yourself slip into the tunnel-vision that comes with long cycles.

Because if your team cannot make responsible architectural decisions, then no one can save your project.

Renewed literacy

Thu, 8 May 2008 05:11:14 GMT

My commute includes two bus rides through the worst of Seattle traffic. My evenings rarely leave me wanting to stare at a computer screen. And my weekends are spent with my friends.

Thanks to this new work-life balance, I have rediscovered books.

A few years ago, I top-loaded my Media to Consume note with the ALA’s list of most frequently banned / challenged books. After those come a large set of philosophy texts included half from curiosity, and half because “why should only liberal arts majors swoon the ladies?” Finishing it off are instructionals on the practice of programming.

Since January, here is what I will admit to having finished:

“A Christmas Carol,” Charles Dickens
“All Quiet on the Western Front,” Erich Maria Remarque
“An American Tragedy,” Theodore Dreiser
“Beautiful Code,” Andy Oram and Greg Wilson (Editors)
“Emotional Design: Why We Love (or Hate) Everyday Things,” Donald Norman
“Moneyball,” Michael M. Lewis
“Parliament of Whores,” PJ O’Rourke
“The Omnivore’s Dilemma,” Michael Pollan

I’m finishing, starting, and working through every exercise: (respectively)

“The Design of Everyday Things,” Donald Norman
“The Practice of Programming,” Brian Kernighan and Rob Pike
“Structure and Interpretation of Computer Programs,” Abelson and Sussman

All were time well spent. That’s why I provide convenient links. Because I care. Even though, I use the library.

What are you reading?

A 'nother month.

Wed, 7 May 2008 10:18:14 GMT

Every morning I wake up, roll over, and reach for my laptop. It sits on the floor beside my futon. I unplug it and pull it to my lap. Before leaving my bed, I have already started on my only daily habit: I “catch up.”

This process is carefully refined and practiced. I glance over accumulated IMs, review new e-mails, and open my feed reader. These tasks are ordered by the time they take.

I can read and reply to a dozen pending IMs in a minute or so. Sorting and responding to e-mails is another ten. It’s the feed reader that consumes hours.

I have found my information limit.

I’m merciless with ignoring IMs and removing buddies since the great shakedown. Instituting rules dramatically reduced the cognitive load of e-mail. But, my feeds were becoming difficult to stay current with.

I evaluated what I found interesting. I reviewed previously saved and shared articles. Then, I cut. Video games, tech news, and politics were all cut. Friends, a few web comics, and local music were kept. Aggregators like reddit were deleted, but Planets like Parrot remained. Personalities like Tim Bray, why, and Joel not only survived but were added in bulk.

My direction became clear. Notice that Facebook, Twitter and other attention networks aren’t in that above list. When “catching up,” I’m writing test programs to understand concepts, noting ideas that pop in my head, and spamming my friends with exciting links. Attention networks provide me with little gain except a personalized tabloid.

Thus, I’m dropping them, for varying values of “drop:”

Facebook: I hate it so much. I wish it didn’t have messaging. But, until it dies, there are two reasons to keep my account: profiles and events. But now, I receive no notifications.
Twitter: I used it as an SMS note-taking service. Then, I started communicating on it. But, it’s just so awkward! So, no more following and back to note-taking.
LiveJournal: I’m not reading my friend’s page anymore. Sorry.
All the rest: Gone. Let me know if you see any stray accounts.

Hopefully, I’ll be blogging more.

How Scott hosts e-mail

Thu, 24 Apr 2008 03:08:22 GMT

I’ve been on the Internet a long time.

> ;$network.MOO_Name
=> "LambdaMOO"
[used 2 ticks, 0 seconds.]

> @age me
Quad first connected on Tue Oct 31 17:07:28 1995 PST
Which makes us 12 years, 5 months, and 10 days old.
However, for official purposes our age is 12 years, 3 months, and 27 days.

And, in that time, I have accumulated a few e-mail addresses. I’m proud to say that, with a few exceptions due to legal complications, every one of them still reaches me. But, this means I invest quite a bit of effort into my infrastructure.

I have a VPS running Postfix / Fetchmail + Procmail + SpamAssassin + Dovecot. I use mutt and (increasingly) Thunderbird to read and write. It’s a well oiled machine pushing a 6 gigabyte spool.

How Stuff Gets In

The Postfix configuration is bog standard. megan.quadhome.com is the authoritative name for the server. My domains are all virtually aliased to UNIX accounts.

For relaying my mail, the settings are straight-forward. No relaying without authentication. No authentication without TLS.

For the addresses whose domains I don’t directly control, that’s where Fetchmail steps in. I have a .fetchmailrc listing my accumulated servers, accounts and passwords. A crontab entry on @reboot starts the daemon.

How Stuff Gets Munged

I used to use virtual addresses. scott_BLAH@scott.tranzoa.net for anything sketchy. But, I found the effort made no difference in my inbox.

Now, when an e-mail comes in, it goes through a Procmail filter that separates mailing list traffic into their own dedicated boxes. After that, everything remaining is fed into SpamAssassin. I use spamc / spamd with bayes_learn_journal enabled to keep things fast.

As incredible as it sounds, occasionally SpamAssassin is wrong. Two folders named “Ham” and “Spam” exist for those situations. I appropriately file the miscategorized mail and the following script ran @hourly solves the problem:

#!/bin/sh
#
# learn-mbox
#
# An fancy wrapper around SpamAssassin's sa-learn.
#
# Learn an mailbox and then delete it.
#
# Lock to ensure we don't clobber anything.
#

MBOX="$1"
MODE="$2"

if [ -z "$MBOX" ]; then
  echo "Usage: $0 [MAILBOX] [ham | spam]" >&2
  exit 1
elif [ ! -f "$MBOX" ]; then
  echo "$0: '$MBOX' does not exist." >&2
  exit 1
elif [ ! -s "$MBOX" ]; then
#  echo "$0: '$MBOX' is empty." >&2
  exit 1
fi

if [[ "$MODE" != "ham" && "$MODE" != "spam" ]]; then
  echo "$0: '$MODE' is not a learning mode. ('ham' or 'spam')" >&2
  exit 2
fi

lockfile-create $MBOX
lockfile-touch $MBOX &

sa-learn --mbox --$MODE $MBOX > /dev/null
echo -n > $MBOX

kill %1
lockfile-remove $MBOX

How Stuff Gets To Me

No Hotmail, Eudora, or Squirrelmail for me. I used Pine for the first years of my online life. After the licensing dispute, I switched to mutt and never looked back. It had all the features I needed.

Time marched on, and different features became more important.

Now, I use a combination of Thunderbird and mutt. The former provides a richer experience. The latter is a safety net for when I’m on random computers.

mutt is on the server, so it accesses my mail directly. But, Thunderbird is an IMAP client. And, Dovecot provides those necessary IMAP services.

Dovecot is also configured with out-of-box defaults with one exception. My IMAP passwords are different from my UNIX passwords. Dovecot provides TLS-only SASL authentication with hashed passwords. Postfix also works with Dovecot to share the same authentication method.

The practical upside is when Mallory finds my mail passwords, she can’t destroy my server and backups.

The challenge from Denver.

Thu, 24 Apr 2008 03:08:22 GMT

My friend Mike drunk-dials me one evening and leaves a voicemail. He’s out in Boulder for TechStars 2007. Apparently, some friendly harassment over drinks between companies was pushed to the next level. EventVue’s team bet Mike a dinner and some cash that a hack couldn’t be slipped in on their website.

~ Who ya gonna call? ~

I get started Thursday afternoon with a whois/ping of the server, and basically do my homework to make sure all the registration information is what it should be. What can I say - even though I’m being given an account on their server, I still like to feel comfortable before I (possibly) break the law.

Rules of the contest are to find a site modification hack. This has been defined as:

XSS
SQL Injection
Remote Root

I plan on focusing on XSS attacks as they’re easy and have the least potential to cause long-term damage. SQL injection investigation can result in inconsistent database states, and a remote root means a painful security audit for someone who isn’t me.

Their development web server is protected using HTTP authorization - plaintext. I haven’t been given a username and password yet. Therefore, I send Mike a text message and wait to get some permissions.

In the mean time, I refresh my memory on various PHP artifacts. It was mentioned that magic quotes are enabled as a security precaution. A mental echo tells me that the feature is a false sense of security option and that most deployments have it turned off. I read documentation to refresh my memory. For the uninformed, it’s a mechanism where incoming GET and POST data is unconditionally escaped. It’s generally disabled on servers because of the headaches it causes in repeated escaped data being passed from page to page. It also offers limited protection for SQL injection, as it s often easy to bypass in cases of alternate delimeters.

30 minutes pass.

Mike sends me a username and password via text message. It isn’t the most secure password, but whatever - I don’t plan on running a dictionary attack or anything.

I logged into the development site and it’s a slightly more broken version of their normal front page. And, I apparently still need an invite. Another phone call to Mike…

30 more minutes pass.

I receive further login details and immediately am greeted with an inauspicious beginning. In their login page, the authentication fields are pre-filled with the incorrect credentials I had supplied earlier. I don’t have Javascript enabled yet (NoScript) and I planned on taking a look at the cookies later but… I decided to look then.

There were only session IDs. Their server is storing the username and password cleartexts keyed to the session ID and then pushing them back to the client in the HTML. If I find a XSS, then I can steal anyone’s username and password by requesting their login page.

Also, my username and password still don’t work.

While I wait for further details from Mike, I suss out the beginnings of a POC. The login page is XSS’able via its authentication fields. I can cull passwords via an XSS against it and then XMLHTTP’ing the password scraped from the DOM back.

Though, it is destructive on the username, but I think that can be worked around.

20 minutes pass.

I’m finally in the site. It was a matter of a “beta.” vs. “dev.” URL. I take a look at “Account Settings” and they’re kicking back the username and password there too in cleartext. So, the login page XSS doesn’t need any trickery to work around.

Their search page uses some odd search-and-replace mechanism on the query quoting. I can’t figure it out too much, but a simple XSS of:

/search?q=%22%3E%3Cscript%3Ealert(1)%3C/script%3E

Works just fine. But, I still want to find an injection hole in order to make something self-replicating.

The profile page is where they spent their lock-down time. Every field has aggressive HTML stripping and magic quotes applied. This makes for some ugly formatting bugs, but I can’t immediately push an XSS through there. The HTML filter is something along the lines of:

regexp_replace(“\<^\w*>”, “”)

I feel that there should be some trick to using magic quotes and their inconsistent use of stripslahes to bypass it all. Specifically, they strip on some output (profile page) and not on others (profile edit page). I’m surprised they just don’t use htmlspecialchars and be done with it.

An hour passes.

I called Mike to let him know I win. While I think my earlier XSS attacks were enough, I finally found a on-site modification. Changing the user’s name to a quote injected with an onload event worked. It triggers on all other users when they visit the Community Page too.

Does this mean I win a free trip to Boulder, and Munchy’z tomorrow? Sweet deal.

This was first posted 2007-07-06 but taken down because EventVue was nascent. It’s back now, for keepsies.

Wii!

Thu, 24 Apr 2008 03:08:22 GMT

I’ve been remiss on this.

7368 9770 0615 7417

But, this has got to be the most inefficient method of getting my friend’s codes.

Let’s try something even worse.