Older blog entries for shapr (starting at number 11)

Well, I've been lazing around and playing a new game I got for $2. I thought it was a nice price.. it also came with spiffy stuff like Autechre and Squarepusher for the sound track, so it can't be all bad. So... I think I'll be preparing for Mr Wizard's visit from Alabama for a few days, don't expect much for a week or so.

Ya know, Open Source seems to be more about kicking something out the door and letting people play with it. I think I'll just hand code each parse tree transformation and save the transformation language till the next version. It seems I don't really understand how to do something until I've already written it anyway. And then it needs doing again.

I think I'll refactor genpy into a subclass of the Python Compiler Visitor, and then hand-code up the tree transformations. I would really like to get a minimal prototype out the door SOON.

Aha, it's called tree pattern matching, and it seems to be a bit of deep algorithmic science. I am unsurprised that it involves algorithms similar to regular expressions. Now that someone told me what it is, and sent me a link to code he had written (Thanks much Thilo :), I can actually do it.

It looks like I'm going to use the AST Visitor that Jeremy Hylton wrote to zip across the tree and fill in a state machine. Should be entertaining.

Much frustration ensues. I'm currently floundering about trying to figure out exactly how to write a search and transform module for ASTs. Everything I've tried so far has turned out flaky. <sigh>

If you have advice, I'd love to hear it.

Okay, so it wasn't almost finished. I found many many bugs to fix. But it seems to be finished now, so I've checked it into the sourceforge cvs tree. If anyone finds bugs, I wish to be notified of them.

I've been unit testing the ast to Python source module by taking original source, turning it into the original ast, then turning that into newsource, and then getting a new ast and comparing original ast to new ast. If they weren't exactly equal, I had a bug. The best file I found to do that with is the x-python file that comes with Python2C, it's full of degenerate cases of legal Python syntax. Boy was it rough to get through.

So now it's time to build a search and transform module for the syntax tree. I don't really know the best way to approach it, but I think I'll just go in a certain direction until it works or the approach turns out to be a dead-end. I wonder how other people do exploratory programming. If you'd like to tell me how you do exploratory programming, send me email!

well, it's 3:30am and genpy.py is almost finished. I think I'm going to abstract out the common code between n_function and n_lambda... but maybe I'll do that tomorrow. I wrote some very strange code to make this module work. If anyone wants a copy to try out, send me email!

I'm looking forward to tomorrow, when I get to start on the parse tree search module. There's some pretty strange syntax in the parse tree matcher from the refactoring browser thesis, it should make for an exciting implementation in Python.

For now, though, it's sleeptime.

Wheee. It's 5:26am and the first rough version of the Python2C AST to Python source module is working! Yeeehaaa!

It's nice that something is working!

The bicyclerepair-general list actually had a message. Wow. I responded with lots of detail, so I probably scared the guy off. *sigh*

Now that I have a rough AST to source, I need to add the rest of the node handlers and get on to something else. I know I should clean up the source and refactor out at least the indentation code, but hey, I want to get other parts working. Hm, XP says I should refactor now while I can still understand the code. Hm.

The next piece I'll be writing is the AST search and transform code. That should be challenging, but I've had my hands deep enough in the AST that I may actually be qualified to do it.

I've also been thinking about IsExclusive. Personally, I never do raw instance variable access, but that's not necessarily true for everyone, therefore, not even self.var = value in the __init__ function is a safe bet for IsExclusive. I'll keep thinking.

Since two people have asked me if they can contribute to BRM, it's about time for me to write up the UserStories and such. When it was just me, YAGNI applied. Now it's time for enough structure to be able to delegate with a minimum of effort.

This is the first Open Source project I've worked on, so I don't know how to proceed. Do I advertise on comp.lang.python? Should I raise public awareness about a wonderful tool on the horizon? Should I wait to mention anything till there's an alpha version?

Somebody tell me, how exactly does the "release early and often" strategy apply?

Oh well, sleeptime, and more coding tomorrow.

Well, crap. I got pretty far into ClassReferences() and discovered that I should really be using the Abstract Syntax Tree regexps module to ask these questions. I wish I had already written the AST regexps. Oh well. So now I'm working on the parse tree searching stuff, once it's done, I'll refactor the analysis methods to use it.

The one thing I don't know how to deal with is the primitive analysis function 'IsExclusive().' The IsExclusive analysis function is supposed to determine if a variable is only used by a single instance of a class at a time. I can safely assume that variables that are set on self in the __init__ function satisfy that criterion, but what about other variables? The original RB assumes true unless it discovers something to the contrary. The rough part is that it instruments the class to detect access by more than one instance and then executes the test suite. That seems like a pretty weak test to me, and a lot of code to write to get the results of that weak test.

I think I may just work on the Python2C AST to Python source problem and let these thoughts simmer in the back of my head. That's the problem with PeterPrincipleProgramming, you don't know what you're doing until you've already done it, and then you get to do it over.

I got a nifty Object Inspector from Anthony Lander (lead architect for TOPlink/Smalltalk) that intregrates nicely with IDLE and looks very much like a Smalltalk Object Inspector. It's cool code, and the first code contribution to the Python Refactoring Browser project.

People who have contributed to this project so far: Erno Kuusela, Taral, and Anthony Lander. (and me) thanks guys! and thanks to demoncrat for coding advice and answer to questions about Smalltalk.

Well, today's task is pulling 'what classes refer to this class' out of a buncha module source files. I've decided to add a list of imports and references to imports to each module, that should handle it.

I noticed that some people put more personal stuff on their development diary. Would that make them open source?
I just don't wish to post personal thoughts and occurences here myself. I don't think I ever want to post them. It's my business, not anyone else's. Hmm.

I'm currently adding the most basic analysis functions to BRM. I have IsClass working fine, and I added IsGlobal last night, but ClassReferences (return all references to this class) was a bit difficult, I think I can do it tonight. I asked on the py2c-dev list whether there was already code to generate python source from an enhanced AST (aka 'transformer nodes') but it doesn't seem to exist. Bill Tutt did respond with "It should be easy." I don't doubt him, but I surely wish I could figure out how to do it.

I found an Emacs commercial refactoring plugin mentioned on the Wiki.
When BRM is done, I plan on integrating it into IDLE first, and Emacs python-mode second. To that end, I wrote up a UnitTest generator called skeletest last night. It uses py2c's enhanced AST to write out a nearly blank test for each class/method combination, but it's not quite finished.