Older blog entries for shapr (starting at number 8)

Much frustration ensues. I'm currently floundering about trying to figure out exactly how to write a search and transform module for ASTs. Everything I've tried so far has turned out flaky. <sigh>

If you have advice, I'd love to hear it.

Okay, so it wasn't almost finished. I found many many bugs to fix. But it seems to be finished now, so I've checked it into the sourceforge cvs tree. If anyone finds bugs, I wish to be notified of them.

I've been unit testing the ast to Python source module by taking original source, turning it into the original ast, then turning that into newsource, and then getting a new ast and comparing original ast to new ast. If they weren't exactly equal, I had a bug. The best file I found to do that with is the x-python file that comes with Python2C, it's full of degenerate cases of legal Python syntax. Boy was it rough to get through.

So now it's time to build a search and transform module for the syntax tree. I don't really know the best way to approach it, but I think I'll just go in a certain direction until it works or the approach turns out to be a dead-end. I wonder how other people do exploratory programming. If you'd like to tell me how you do exploratory programming, send me email!

well, it's 3:30am and genpy.py is almost finished. I think I'm going to abstract out the common code between n_function and n_lambda... but maybe I'll do that tomorrow. I wrote some very strange code to make this module work. If anyone wants a copy to try out, send me email!

I'm looking forward to tomorrow, when I get to start on the parse tree search module. There's some pretty strange syntax in the parse tree matcher from the refactoring browser thesis, it should make for an exciting implementation in Python.

For now, though, it's sleeptime.

Wheee. It's 5:26am and the first rough version of the Python2C AST to Python source module is working! Yeeehaaa!

It's nice that something is working!

The bicyclerepair-general list actually had a message. Wow. I responded with lots of detail, so I probably scared the guy off. *sigh*

Now that I have a rough AST to source, I need to add the rest of the node handlers and get on to something else. I know I should clean up the source and refactor out at least the indentation code, but hey, I want to get other parts working. Hm, XP says I should refactor now while I can still understand the code. Hm.

The next piece I'll be writing is the AST search and transform code. That should be challenging, but I've had my hands deep enough in the AST that I may actually be qualified to do it.

I've also been thinking about IsExclusive. Personally, I never do raw instance variable access, but that's not necessarily true for everyone, therefore, not even self.var = value in the __init__ function is a safe bet for IsExclusive. I'll keep thinking.

Since two people have asked me if they can contribute to BRM, it's about time for me to write up the UserStories and such. When it was just me, YAGNI applied. Now it's time for enough structure to be able to delegate with a minimum of effort.

This is the first Open Source project I've worked on, so I don't know how to proceed. Do I advertise on comp.lang.python? Should I raise public awareness about a wonderful tool on the horizon? Should I wait to mention anything till there's an alpha version?

Somebody tell me, how exactly does the "release early and often" strategy apply?

Oh well, sleeptime, and more coding tomorrow.

Well, crap. I got pretty far into ClassReferences() and discovered that I should really be using the Abstract Syntax Tree regexps module to ask these questions. I wish I had already written the AST regexps. Oh well. So now I'm working on the parse tree searching stuff, once it's done, I'll refactor the analysis methods to use it.

The one thing I don't know how to deal with is the primitive analysis function 'IsExclusive().' The IsExclusive analysis function is supposed to determine if a variable is only used by a single instance of a class at a time. I can safely assume that variables that are set on self in the __init__ function satisfy that criterion, but what about other variables? The original RB assumes true unless it discovers something to the contrary. The rough part is that it instruments the class to detect access by more than one instance and then executes the test suite. That seems like a pretty weak test to me, and a lot of code to write to get the results of that weak test.

I think I may just work on the Python2C AST to Python source problem and let these thoughts simmer in the back of my head. That's the problem with PeterPrincipleProgramming, you don't know what you're doing until you've already done it, and then you get to do it over.

I got a nifty Object Inspector from Anthony Lander (lead architect for TOPlink/Smalltalk) that intregrates nicely with IDLE and looks very much like a Smalltalk Object Inspector. It's cool code, and the first code contribution to the Python Refactoring Browser project.

People who have contributed to this project so far: Erno Kuusela, Taral, and Anthony Lander. (and me) thanks guys! and thanks to demoncrat for coding advice and answer to questions about Smalltalk.

Well, today's task is pulling 'what classes refer to this class' out of a buncha module source files. I've decided to add a list of imports and references to imports to each module, that should handle it.

I noticed that some people put more personal stuff on their development diary. Would that make them open source?
I just don't wish to post personal thoughts and occurences here myself. I don't think I ever want to post them. It's my business, not anyone else's. Hmm.

I'm currently adding the most basic analysis functions to BRM. I have IsClass working fine, and I added IsGlobal last night, but ClassReferences (return all references to this class) was a bit difficult, I think I can do it tonight. I asked on the py2c-dev list whether there was already code to generate python source from an enhanced AST (aka 'transformer nodes') but it doesn't seem to exist. Bill Tutt did respond with "It should be easy." I don't doubt him, but I surely wish I could figure out how to do it.

I found an Emacs commercial refactoring plugin mentioned on the Wiki.
When BRM is done, I plan on integrating it into IDLE first, and Emacs python-mode second. To that end, I wrote up a UnitTest generator called skeletest last night. It uses py2c's enhanced AST to write out a nearly blank test for each class/method combination, but it's not quite finished.

Hmm, nifty, I've gotten the improved AST class from Python2C working somewhat. I can now pull out class, function and randomcode definitions and add them to a module level list. spiffy!

I'm still not sure about generating Python code from the improved AST though. The main thing I'm missing is how to get 'implied indentation' figured out. At least I know it's in there.

It's cool and surprising that people have already certified me, wow!

Ah, the excitement never wanes in the land of the midnight sun. Today I'm puzzling over the difficulty and worth of getting the Python2C improved abstract syntax tree generating Python. It'd make my life lots easier, and it's nearly usable by the non-initiated. I've posted to the py2c-dev list to see if it's already been done, but I doubt it.

Progress is slowly being made on the Python Refactoring Browser.

Advogato.org is interesting, but I don't know if I'll actually use it. We'll see.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!