Older blog entries for chromatic (starting at number 138)

Hiding Behind the Bill of Rights:

brencrypt.pl encrypts any file. It creates a self-extracting archive that requires a very special key in another file -- the text of the Bill of Rights.

"Self-extracting", in this case, means that you must have a recent Perl and my Crypt::CipherSaber module installed. (In fact, you ought to have the most recent, though unreleased, version, to avoid a silly warning.) To encrypt, pass the name of the file to run and the name of the output file. To decrypt a file, pass the name of the key file and, optionally, the name of the decrypted output file. You can also redirect STDOUT, if you're exceptionally daring.

There are a couple of limitations. First, the aforementioned module patch would be nice, but it's not ready. (In debugging, I found some inflexible bits in the module that should be fixed). Second, I'm not positive it writes binary files correctly. The most fatal flaw is that you need the exact text of the Bill of Rights I used, right down to the formatting. I'll fix this flaw first.

I'd point to the text I used, but that would be a circumvention device. Oh wait, did the first ten amendments of the Constitution of the United States of America just become illegal? Oops.

Today's Hacking:

With Test::MockObject on the CPAN, I've released a new version of Text::WikiFormat. The main enhancement is a custom import(), spurred on by a request from Tony Bowden. That's one five week-old project finally off my plate.

At work, I've added a nifty new navigation feature to the product catalogs. In the process, I merged the administrative and end-user lists as far as was reasonable. (More was possible, but there's diminishing returns -- the further I go, the more I'll have to revise.) I'm still in negative lines of code for the week, though.

Seeing that Simon has released new versions of B::Generate and B::Utils, and that multiple people have asked about B::ToXML and B::FromXML (see prior journal entries), it's time to devote more attention to the problem. I can serialize bytecode to XML with no problem. I'm pretty sure it captures the relevant information (at least, everything available through B.pm). I'm pretty sure it's generating the correct opcodes, though some of the optimizations may be getting in my way. (When a LISTOP has been flattened into a NOP, but still has children, I get confused.) I seem to be able to attach the ops in the right order and can splice them into a fresh anonymous sub, but they confuse both the interpreter and B::Deparse... So I get infinite loops when running the spliced sub and strange output from deparsing it.

There's something I'm missing, and it may take a really scary guru to find it.

Perl Mock Objects Make Testing Easier:

I've added documentation to Test::MockObject, added two handy new functions, and bumped up the version number by 0.01. It's ready to be used. Now to write an article that explains the why, as the what is already documented. (Oh yeah, and the buglet in Test::More has been fixed, so the prerequisite is set in my Makefile.PL. Nice.)

Not Having Learned My Lesson Last Time:

I also wrote half of a book outline before giving up for the day.

Up to the Date:

In preparation for an article, a PM talk, a tutorial, and quite possibly a book on software testing with Perl, I've been working on Test::MockObject. The documentation, as usual, comes somewhat after the initial announcement.

I'm not completely thrilled with the interface yet, and testing the Test::Builder components accurately could fall into the bootstrapping category, but the sooner it's out there, the more likely someone with a good idea or spare time could run across it and improve it. My plan is to have something more CPAN-worthy by the weekend. Then again, I planned to make some enhancements to Text::WikiFormat last week, and need to fix a Cygwin test buglet for Sub::Context.

Test #5 fails on the current released version of Test::More (0.43), but I've patched it and 0.44 (or 1.0) is on its way. Schwern is generally good about that.

On work matters, the database stuff has been "Center the date range on the report and make it bold", so that's nearly finished. I've been doing actual Perl programming for a while, and some of it's very clever. I wasn't having a lot of fun in my last entry, but it had been a rough day. Things are looking Sunny.

On Programmer Value:
Overheard:
No, we're not going to use (free software package) for this project. Absolutely not. Our customers don't want us to use code we didn't write. Yes, I know (chromatic) wrote it and he works here. No, we've got similar code, how hard can it be to modify it? Our customers don't want us using some (crap) someone else wrote.

What is the value I add as a programmer? Is it truly reinventing another user authentication system, or another database interface, or another templating system?

Do customers really care if their solution is written in C, Perl, Java, Delphi, Objective-C, Python, Ruby, Smalltask, Haskell, or even Visual Basic by a team of coders around the world, a component company, a "programmer" long since fired, some guy who wrote a book on the subject, or the current development team? Don't they care only that it works to their specifications?

It doesn't amaze me that a software project on which I spent six months of my life may not be the most appropriate tool for the job. It will do what's necessary, and I'd enjoy the opportunity to work on it again, but it may not be appropriate. That's fine. Summarily dismissing it because it wasn't written on company time and it's not what the boss had in mind is another. Insulting it (and me) in my presence is even worse.

If customers really cared that the developers at (current company) hadn't invented every last thing we use, we'd have to throw out Perl, Linux, Apache, mod_perl. PostgreSQL, DBI, CGI, Template TOolkit, CVS, SSH, and Samba, not to mention all of the proprietary software. That's a ridiculous argument. Show me a customer who cares about that enough to make a fuss and I'll show you a customer who'll complain about anything you do.

Were I to advise customers on the real issues, I'd bring up things like scope creep, lack of change plans, longstanding design issues that are only addressed when something breaks, and something I'll bring up in just a moment. My capacity to be amazed is certainly stretching with this job. Who would have thought they'd hire me based on my Perlish contributions of the last year and a half, only to set me on the monkey work of grinding out Crystal Reports reports and making them available with ASP written in VBScript. (To be fair, I have written Perl this week. The problem is, the net is around negative 500 lines, having removed far more detritus than I added. It is now faster, less buggy, and has at least two more features.)

Yeah, I write (crap) code in a language I've never seen before, to a specification that promises things that are physically impossible, with a deadline dropped in my lap on my first day with little explanation, no apparent work from my predecessor, and no ability to test things other than e-mailing files to the client and asking her to install them. The kicker is, I'm good enough at it that I completed the spec to sufficient degree that the client added more work. A monkey could do this, and you wouldn't have to shave him.

I'd say our customers should demand that we don't use our existing software. At least my code works. (It has documentation, too.)

Stealing Pebbles from Perl's Other Hand:

Having fixed several bugs in B::ToXML, I've wobbled back and forth between execution order and tree order. When I realized that I could keep track of op sequence numbers (to reconstruct execution order), I returned to tree form. It works better that way, as children are nested properly.

I'd missed a handful of op types (LOGOP comes to mind), and now support them.

I'm able to splice a generated op tree into the tree of an anonymous subroutine, and it more or less works -- a simple replacement that prints a happy message even deparses (almost) correctly. Splicing is tricky, as it must be careful to lop off 'leavesub' and 'entersub' ops in the right places.

The problem is COPs, or control operations. They're not generated correctly (probably because they're not documented well). A 'nextstate' op somehow turns into a 'lineseq'. I hacked a method into B::Generate to clear the COP label, but that's messy. It also still leaves empty do { } constructs.

On the whole, it's a lot further along than it was a few days ago. I keep learning things, too. It seems just a few tweaks short of success, or at least more reliable segfaults.

The obligatory links are B::ToXML and xmltwigtoperl.pl. (I used XML::Twig because it's easy and Mirod wouldn't like it if I didn't.)

*bonk*:

Yesterday was productive, as Text::WikiFormat received documentation attention. This is always the last step of my release cycle. Strangely enough, though I enjoy commenting on code (think commentary), I dislike writing documentation, example usage, and all of the things that make developers happy. I even prefer writing tests.

It's a good sanity check, though. Things that are difficult to explain clearly usually need polish. Things I'd rather not explain usually need to be fixed. I tend to add a couple of convenient features and fix annoying buglets during the documentation phase.

I've been doing more work on my Perl bytecode->XML->refactoring/linting/improving->XML->Perl code. Since Schwern asked so nicely ("I don't understand what you claimed to be doing, but are you going to make it available?"), you can download it here. It works like any other backend module. Try perl -MO=ToXML proggie.pl to print an XML file representing (what I can decipher to be) the important parts of the compiled bytecode. If you want to output to a file, use perl -MO=ToXML,-ofilename proggie.pl instead. If you just want to dump a function or functions and print to a file, it's perl -MO=ToXML,-ofilename,func1,func2 proggie.pl.

Bypassing the idea of transformations at this point, I'm left with the task of turning XML back into Perl. There are two possibilities. I could use B::Generate to create an optree, then run it through B::Deparse. The other option is to generate just enough data to send to B::Deparse directly. I much prefer the latter, though I haven't tried the former.

The odds either Simon Cozens or Rafael Garcia-Suarez will take pity on me and reveal the secret information I'm not saving in B::ToXML or the magic of sending data to B::Deparse are minimal.

Since I should be writing two articles aimed at novices, packing up everything I own, or any of several other important things, it's a great distraction.

Scary Perl Refactoring:

I first formally encountered the idea of refactoring while studying XP. I'd been doing it already (especially when giving advice to new programmers), but didn't have a vocabulary for it. As my study progressed, I learned that Smalltalk (among other languages) has a Refactoring Browser -- since refactoring can be considered mechanical transformations, why wouldn't a machine be able to do them?

At Schwern's Refactoring talk at TPC 5.0, he demonstrated the beginnings of a Perl parser that warned about dubious constructs. Parsing Perl with anything but perl obviously isn't a simple task -- Damian hasn't released Parse::Perl, though I'm impressed with perltidy.

Looking at Smalltalk in more detail made me think that operating on source code is the hard way. Working on bytecode would be much easier -- except for associating lines of code with opcodes. (I worked up a source filter that would insert a target with comments and code on appropriate occasions. This data can then be extracted as necessary.) Talking to Ned Konz and Simon Cozens, they both thought that bytecode was the right track.

Another piece of the puzzle came in writing an article about the Linux Kernel Janitors. The idea behind the Stanford Checker really stuck out. If you can demonstrate an error pattern, the compiler can look for it in the code. Obviously, I can take a bit of bad Perl, compile it to bytecode, and have a tree that marks a bad pattern. As Simon pointed out, though, searching a tree for a tree is a difficult problem.

I didn't entirely agree. Though it's usually stupid to disagree with someone that smart, sometimes it can lead to a good idea. For some reason, I thought it was doable. Walking near a koi pond one afternoon, it hit me.

The XML guys have, more or less, solved this.

Okay, the LISP guys may be able make a better case, but the important thing is that it's solvable. I thought about sending Matt Sergeant an e-mail, asking him how to take some of the rules of XPath and apply them to a non-XML tree. For a few months, the whole idea was on the back burner.

Nearly all of the pieces are in place already. There's a bytecode decompiler in B::Terse (and I knew a bit about it, having written tests for it). There's a bytecode generator in B::Generate (thanks to Simon). There's a bytecode to Perl converter in B::Deparse (thanks to a lot of people, especially Rafael Garcia-Suarez, lately). I just needed to find some way to apply something like XSLT to Perl bytecode.

This morning, it hit me. Maybe it was the Perl XML fans talking about SAX being important for more than XML, but I realized that if I could write a backend module to turn bytecode into XML, the tree matching and conversions would be solved. The only tricky part that's left is generating XSLT or XPathScript or whatever syntax to refactor an error pattern. The same XML guys who provided the final nudge can probably help out in that respect.

So now I have B::ToXML that can XMLize a code reference, and it works pretty well. If I knew the internals better, I'd be able to tell what kind of information is important. I don't yet have any way to say "go from this to this, keeping this but changing this", but I'm one step closer.

I could still be on the wrong track, but I really think I'm on to something here. Drop me a line if you have a strong opinion either way.

No Grand Plan:

Random thoughts.

This article turned out to be around 6000 words, 4800 of which were written in a single day. Ouch.

Chris Winters wants me to put Text::WikiFormat up on the CPAN soon. First, I have to get the name past people like Jarkko, Tim, and Skud. Yow.

Since nate moved Everything to SourceForge, I'm now a developer on it again. The first step? Comprehensive unit tests. I've got religion, baby.

I have a phone interview with the Dallas company tomorrow afternoon (today, if you go by GMT, which I generally don't). Another possibility has started to consider to begin to present itself, and it's not in Dallas. That has pluses and minuses.

I decided that no matter how much you care, friends who make a habit of stupid and hurtful decisions aren't trustworthy until they start making better choices. Knowing that I'm being wise doesn't make me feel any better about it. Knowing that several other friends kept trying to convince me of it does, but just a little.

The general consensus seems to be that I look better in my normal street clothes (t-shirt, jeans, leather jacket) than in a tuxedo. I'll save the rental fee next time. How many other best men wear a t-shirt with a glow-in-the-dark alien head on it?

The Perl Foundation website runs on software I helped write. I just noticed.

(stream-of-consciousness: I write valid XHTML even in my journal. How... perfect?)

Finally:

After two days of writing documentation (and writing tests to make sure the documentation is right and fixing bugs because it isn't) and adding features (because the documentation makes better sense with the new features) and writing tests (to prove the correctness of the new features), and after several months of inactivity, Regexp::English has escaped. It wings its way to the CPAN, being found in my directory or on my website.

I think it's cool, Damian thought it was worth mentioning, and the variable binding feature is just nice.

With the SlashWiki article scheduled for later this week, I have time to work on articles for Perl Archive and, hopefully, tests for O and CPAN::FirstTime.

It Can't Rain Forever:

In other news, being in the Portland, Oregon area for my brother's wedding, I realized I could live there. Though I have an interview (phone?) with the Texas guy on Thursday, I might consider looking in the Hillsboro/Beaverton/Vancouver area. There are worse things. (And hello, S.B.!)

129 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!