Advogato blog for connolly

25 Oct 2007

Thu, 25 Oct 2007 00:52:26 GMT

Remembering Modula-3

Systems Programming with Modula-3 has been on my wishlist for years, but after reading the feedback from Tim Bray's Wide Finder project, I finally got my very own copy.

One of the reactions, Finding Lisp, observes:

Most popular programming languages have the same simple threads+locks paradigm that was popularized with pthreads and Java.

But Java made everything a monitor.

I learned pthreads while working on a big horrible C++/DCE project, Dazel (later bought by HP). One of the guys there (Jim W?) loaned me his copy of Systems Programming with Modula-3, where chapter 4 is a copy of An Introduction to Programming with Threads by Birrell. It's probably a good thing that we never followed thru on our dreams to rewrite the whole project in Modula-3, but it was good to know about partial orders on locks and such while taming the pthread libraries (... and the C++ exception runtimes; what a nightmare!)

And anybody who had studied the Modula-3 Thread and IO design would know better than to make everything a monitor.

A lot of good stuff from Modula-3 lives on in python, and some in Java, but DEC got bought by Compaq which got bought by HP, and a lot of the DEC SRC goodies seem to be disapearing from the net.

The wikipedia article on Modula-3 has a "This article does not cite any references or sources" tag since July 2006. That looks silly, since a number of books and articles are listed. I can see some unsupported claims, though, so I ordered my own copy of SPwM3 so I can separate some of the verifyable claims from the speculation.

I'd also really like to find a host (other than the wayback machine) for the hypertext version of the SRC Modula-3 sources; it's a gold-mine of software engineering theory and practice; for example, from Fingerprint.m3:

The original fingerprint interface offered at SRC did not include the procedure Combine. The Vesta configuration management project built a system that cached intermediate results for large software builds. Abstractly, this is a special case of the common subexpression problem mentioned previously, and the project used fingerprints as keys in the cache. It is instructive to learn what happened.

You might think that a simple way to solve the common subexpression problem without Combine would be to fingerprint the texts that result from printing the expressions represented by the nodes of the DAG. But if the DAG is not a tree, this is a serious error, since the length of the strings produced by printing a DAG can grow geometrically with its size, and therefore the probabilistic guarantee becomes useless even for quite small DAGs.

Avoiding this error, the Vesta group computed the fingerprint of a node by concatenating the node's label with the {\it fingerprints} of its children---treating these fingerprints as 8-byte texts--- and fingerprinted the resulting text. With this strategy, the number of texts fingerprinted is proportional to the number of nodes of the DAG, and the total length of these texts is proportional to the number of edges of the DAG. Thus the method appears efficient and sound.

Alas, the method is not sound. Recall that the probabilistic guarantee is valid only if the strings being fingerprinted are independent of the magic number. But fingerprints themselves are dependent on the magic number, so the probabalistic guarantee is invalid whenever fingerprints are fingerprinted. The Vesta group was soon debugging an unexpected collision.

The moral is simple: the procedure Combine is a convenience, but it is also much more than a convenience. It should be the only way that you ever generate a fingerprint from another fingerprint. In particular, never treat a fingerprint as text to be passed to FromText.

Maybe the python foundation would like to host it? I'm pretty sure Guido has a fondness for Modula-3.

23 Oct 2007

Tue, 23 Oct 2007 20:50:30 GMT

I just discovered SocksiPy - A Python SOCKS client module via a httplib2 - 0.4.0 announcement.

I wonder how SocksiPy relates to the socksForPython module that I hacked together back in 1994. I suppose the clue Guido gave me might have been our first meeting.

27 Jul 2007

Fri, 27 Jul 2007 17:10:34 GMT

Package management is a blessing when there’s integrated QA and support. apt-get is only one half of the equation; the Debian Bug Tracking system (and the people behind it) is the other.

Case in point: fink adds apt-get to the Mac OS, but the result is that updates from Apple and updates from fink compete/collide, and when there’s a problem, the natural result is finger-pointing rather than integrated QA.

Debian/ubuntu have scaled amazingly well, but they rely on quite a bit of trust. The MS Windows marketplace includes lots of mutually distrustful players, as does the Mac OS X marketplace. It’s not at all clear how apt-get and the debian bug tracking system (or ubuntu’s launchpad) can scale to support that sort of marketplace.

This is a comment I left on How package management changed everything by Ian Murdock, spotted in a Tim Bray tab sweep.

tags pending: software installation quality

22 Jun 2007

Fri, 22 Jun 2007 18:32:22 GMT

Music Collections and Office Organization

It worked this time. I wonder why.

I have been trying to rip Paul Simon's Graceland using debian stuff for years. I was sure the CD was mangled beyond recognition, but soundjuicer on Ubuntu on this new HP pavilion just gave me a bunch of .flac files that sound fine.

I listened to this album all summer as I wrote 80,000 lines of HyperTalk code back in 1989. I think my copy was on tape at the time.

I wonder if the CD I ripped today is the same on I failed to rip so many times earlier or if this is a different CD.

This task is in my queue today because the music shelf in my office is poorly organized and overfull, and when I took my worship music binder off the shelf, this CD fell off the shelf. Rather than put it back on the shelf last night, I put it in my "in" box, GTD-style.

This wakes up all sorts of long-swapped-out jobs involving organizing my multimedia collection.

Mostly I don't do back-ups; I just send email and check stuff in using source code control (CVS for ages, but more recently hg... that's another story [@@delicious scm bookmarks]...). And I use google to find it again. But I don't have license to redistribute these flac files, so I have to manage them myself.

Part of me says: don't bother. If I'm going to fill my iPod (shuffle) with things to fill my head, they should be things that glorify God, and while Paul Simon's music is nothing nasty, I'm not sure it glorifies God. Plus, the whole recorded music oligarchy is an unstable marketplace just waiting for a revolution. "If you're whining about iTunes DRM, you're part of the problem," said a slashdot commentor that hit home for me. (@@todo: look it up and link it) much like a conversation with AaronSw about how songs and musicians have been around for a long time, even before you could get rich selling records; if the bottom falls out of the recorded music industry, humanity will go on.

MusicBrains solves the shared metadata problem for CDs pretty well, but Rhythmbox doesn't seem to automatically link to it like it used to. It does show album cover art...

In trying to figure out why, I looked in the help menu which led me to launchpad which links to IRC channels; those irc: links Just Work in Ubuntu. Yay! Except that xchat didn't realize that I was already logged into the Freenode network. Odd.

Better get back to other things... everything is so intertwingled I could go on and on...

tags: digial+media, music, sysadmin, ubuntu, debian, gtd

5 Jun 2007

Tue, 5 Jun 2007 17:11:16 GMT

Not only does the nv driver not do 3D acceleration, it doesn't seem to do 1600x1050 resolution either.

The nvidia module in the debian non-free section blanked the screen and locked up so badly it required a reboot. When investigating options, that gets old real fast.

In #debian, the channel bot said:


<dpkg> from memory, nvidia_post_etch is update-pciids
&&
apt-get install module-assistant nvidia-kernel-source
&& m-a
prepare && m-a a-i nvidia && apt-get install
nvidia-glx &&
depmod -a && modprobe nvidia &&
dpkg-reconfigure xserver-xorg

But that produced the same symptoms.

Besides... debian fonts are ugly, and they seem to be chosen by a zillion different configuration options; it seems to be a full-time job just finding them all.

So I'm running Ubuntu now. I'm not sure how I feel about that.

I replaced the amd64 installation with the stock (quicken-friendly) i386 stuff.

It comes with evolution 2.10, I think. Slightly more polished. I wonder if subscribing to password-protected calendars works.

My code to print MS Word docs to PDF fell over; something about templates. So I'm trying to use OpenOffice instead. I almost got it working. Let's see if ubuntu support will help me out with this...

Bug #118789: trouble using ExportFormFields in uno API; PDF form controls are exported even when set to False

tags: office, debian, ubuntu, linux, hardware,

30 Apr 2007

Mon, 30 Apr 2007 21:58:21 GMT

My desktop PC is a frankenstein, with parts from here and there. It has been crashing and hanging once a week or so since December, and I was getting clues in #hardware on FreeNode about how to diagnose power supply problems.

"Put the multimeter away and get yourself a new PC," the systems guys at W3C told me. Still, it wasn't until about a month of research on PC hardware that I outsourced the decision to Brian at the local Micro Center. "I want a quiet machine," I told him.

It came down to a choice between two HP Pavilion machines; the a6030n with AMD Live! (Athlon 64 X2 etc.) and the a6040n with Intel Viiv (Core 2 Duo etc.). I picked the AMD machine... partly because the cheaper CPU lets them include 2x the RAM for the same price or a little less... but partly because Intel is the market leader and I like to root for the underdog.

I didn't open the box right away when I got it home, because I wanted to research it just a bit more during the 7 day return period without risking a restocking fee. I had some buyer's remorse when I remembered that "at present the nv driver has no 3D acceleration."

I'm still wrestling with so many choices:

stick with debian or switch to Ubuntu for integration and support?
stick with debian sid or switch to a more stable release?
I used to get a few dozen updates when I'd apt-get upgrade after my once-a-month business trip; now it seems that there are hundreds of updates every week; what's going on?
Install x86-64/amd64 packages or stick with i386?
stick with LVM or use the more typical fixed-size partitions?
stay with reiserfs or go back to ext3?

I picked Ubuntu and amd64, at least for starters. I didn't realize I needed the alternate CD image to do LVM until after I had downloaded the desktop image.

The live CD feature is pretty nifty, though it takes longer to come up than a text installer, which starts to add up if you're restarting the install as much as I am. I thought maybe a USB flash disk would be faster than a CD, but it doesn't seem to be. I guess speed varies quite a bit with those things, and the one I was using was a very cheap one.

I picked a goal of getting my Quicken-under-wine setup running as a way to get a feel for the amd64/i386 issue, and then I realized... Why am I still tied to quicken?. I have been noodling about Quicken, RDF, and JSON for a while; why hasn't anyone done an AJAX quicken work-alike yet? Of course, the diff/sync problem is interesting too. The IBM Boca system looks promising. I digress... that's probably a story for my semantic web research blog than this one.

On the other hand, the invited talk by the Mercurial lead developer probably belongs here as well as there.

9 Feb 2007

Fri, 9 Feb 2007 03:10:04 GMT

Social standards and coding fugues

I got Cryptonomicon for Christmas; chalk one up for Amazon wish-lists. It walks the line between geek culture and popular culture in a way that makes me want people close to me to read it so they'll understand me a little better. This bit from p. 819 struck a chord...

"Who framed me, then?" Randy asks, kind of rhetorically. He was just in the middle of doing some pretty cool C++ coding when he got yanked out of his cell to have this surprise encounter with the Dentist, and is surprising himself with just how bored and irritated he is. He has reverted, in other words, back into a pure balls-to-the-wall nerdisim rivaled only by his early game-coding days back in Seattle. The sheer depth and involution of the current nerdism binge would be hard to convey to anyone. Intellectually, he is juggling half a dozen lit torches, Ming vases, live puppies, and running chainsaws. In this frame of mind he cannot bring himself to give a shit about the fact that this incredibly powerful billionaire has gone to a lot of trouble to come and F2F with him. And so he asks the above question as nothing more than a perfunctory guesture, the subtext being I wish you would go away but minimal standards of social decency dictate that I should say something.

In other news, Advogato does foaf, the Semantic Web social networking standard. Rock on, robogato!

5 Jan 2007

Fri, 5 Jan 2007 15:30:30 GMT

18:98992cbff4fd 2007-01-04 work around MS Exchange attachment bug

Exchange 2000 Server requires a file name in the Content-type header. Q12: How does Exchange handle attachments? Microsoft Article 836555

I bet you can guess the rest of the story, but I hope to write it up later today.

#swig notes

8 Dec 2006

Fri, 8 Dec 2006 20:10:16 GMT

appscript and office automation

My wife does office work for a local professional and whenever I see her doing work that I know the computer could do for her, I chip in. The end-of-the-month scramble is a clear case: they take client reports, print them out and then manually sort them by officer and fax them out.

Surely I could do better with faxaway, I thought. The only question was: since the reports are in MS Word and the database is in FileMaker Pro, all on a Mac laptop, how much would I have to sell my soul to Apple and Microsoft in the process?

appscript let me drive the process from python. I did quite a bit of HyperTalk programming, but somehow I'm still a bit mystified by AppleScript: which are the language keywords and which are the application vocabulary? The FileMakerAppscriptingOverview made it trivial to crib bits like:


    fm = app("FileMaker Pro")
    if not fm.databases[db].exists():
        fm.open(FSSpec(path % db))
        return fm.databases[db]

FSSPec is deprecated in the Carbon docs, but I never did figure out a replacement.

appscript's integration of AppleScript references into python with its and con is particularly cute, but I pulled a bit of hair out before I figured out how to use it:


def officerFax(db, oName, cName):
    # hmm... I'm not sure why this str() is necessary...
    officers = db.tables['officers'].records[
            its.fields['name'].cellValue == str(oName)]

If FileMaker has a way to use real SQL, I can't find it. Plus, we're running a PowerPC version on an intel MacBook with only 0.5GB of RAM. Emulating FileMaker and MS Word is using a lot of RAM, I suspect. I looked into open source alternatives and found that OpenOffice's Base looks quite capable, and I'm sure oowriter would do the job as an MS Word replacement. I hope the python-uno bridge works on OS X so that I can switch the whole operation over one of these moths.

I did pay a price for not doing it The Apple Way. Technical Q&A QA1018 Using AppleScript to send an email with an attachment shows exactly how to attach a report to a mail message and send it to faxaway. I was able to create and address a mail message from python/appscript, but making the attachment stumped me. After verifying that the AppleScript example does work as advertised, I gave up and wrote a separate mailfaxes.py program that uses python's email and smtplib modules and skips Mail.app altogether. I had to be a little careful since the laptop runs python2.3 and the email modules have been rearranged a bit in python 2.4 and 2.5, but it was reasonably straightforward.

Driving MS Word was, predicably, even klunkier:


TMP="fax_job.htm"
def asHTML(w, dirpath, fname):
    """save current doc as HTML
    """
    w.do_Visual_Basic('ActiveDocument.SaveAs FileName:="%s",'
        ' FileFormat:= wdFormatHTML,'
        ' HTMLDisplayOnlyOutput:=True' % (TMP,))

Office X has an AppleScript interface, but it's not as rich as the Visual Basic API. I got Word to save as HTML (for processing with BeautifulSoup) but I never did figure out how to tell MS Word which directory to put it in. I wrote a posix2mac() routine to convert /posix/paths to ::mac:paths as used in AppleScript but that didn't help; I ended up with a hard-coded kludge.

Switching syntaxes with do_Visual_Basic is a little bit painful, but when it goes bad the diagnostics are pretty good. "ActivePrinter is read-only on the Macintosh," it said, where w.active_printer = p had just failed silently. The modern VB.NET PrintOut documentation isn't hard to find, but it's a little more tricky to find the 2002 PrintOut docs that are more relevant. I never did get PrintToFile working, nor did I find a way to script the PDF option in Apple's print dialogs. Thank goodness for the Appscript, Word and PDF clue which pointed me to CUPS-PDF for Mac OS X. It worked as advertised, though writing code to wait for a new PDF document in ~/Desktop/cups-pdf/ was tricky; we sent a number of reports to the wrong place due to a timing bug.

The Python Bindings for Quartz 2D rock; composing fax cover pages couldn't be easier than this:


htmltxt = coverHTML(oName, fax, subject, pages)
ctx.drawHTMLTextInRect(
        CG.CGDataProviderCreateWithString(htmltxt),
        pageRect.inset(72, 72))

and concatenating several PDFs into one was similarly straightforward. It doesn't hurt that faxing is Apple's example application.

For reference:


hh-fax2$ hg log --template '#rev#:#node|short#
#date|shortdate# #desc|firstline|strip#\n'
11:7b497e5881d8 2006-12-07 fixed nasty timing bug with PDF
virtual printer
10:5fbd62cf7025 2006-12-07 fixed SMTP details
9:adbc7966d42d 2006-12-07 back to faxaway
8:89f2688b85fc 2006-12-07 smtp host arg
7:eb4eba0ed22c 2006-12-07 mailfaxes.py starting to work
6:c317d0cb9956 2006-12-07 prepares one PDF doc per officer
5:fe8a79a7ed9f 2006-12-06 faxjob.py iterates over reports
and looks up fax numbers
4:52f42112c287 2006-12-06 better diagnostics
3:862514804543 2006-12-04 officer update mostly working
2:61dfc88ab652 2006-12-04 connecting to FM from py works
1:a1f813e53e79 2006-12-02 HTML/CSS page break test
0:aa2d0cc8a7e9 2006-12-02 save as html, doc export working

Tags: python office mac

24 Oct 2006

Tue, 24 Oct 2006 02:31:15 GMT

Worse is better for external disk enclosures too

I got a USB 2.0 disk enclosure this week. It's much quieter than the FireWire enclosure I bought in Oct 2003. I thought surely it was slower, but it's not.

I learned assembly on a 6809E; I thought people were crazy to deal with the 8-bit shackles of the 6502 and the broken segmented architecture of the 8086, though I could see Intel beat Motorola to market with each new generation. I'm a sucker for elegant technology. I paid the SCSI premium for years. I never actually bought a NuBus Mac nor a Newton, but I wanted to. I bought 2 PowerPC Macs, the last one just before even Apple switched to Intel. But slowly, I'm learning: worse is better. As a Web guy, I should know that even a dollar per part is a lot when you're talking about hundreds of millions of parts...

However, Intel was not interested in paying the near-dollar license fee to add an IEEE 1394 subsystem to their board. The fee was reduced to a flat 25 cents, but Intel prefered to push for its own USB 2.0 standard. As a result, they were rarely provided as standard equipment on computers other than Apple Macintosh computers (Apple owns rights to the FireWire standard), and peripheral manufacturers offered many more USB devices.

I divested from SCSI a few years aog; all my disks are commodity IDE now. I'm keeping an eye on SATA but haven't jumped yet.