Older blog entries for osullivj (starting at number 58)

Wow ! IronPython goes open source, and Jim Hugunin joins the CLR team at MS.

Work: starting my new job tomorrow.

Hacking: fun with wxpython.

Hacking: new release of templ. Minor fixes in the Python version driven by levitt development. C#, Java and Tcl code is unchanged.

Hardware acceleration: my first reaction to hearing about XML hardware acceleration was disbelief. Just another symptom of inappropriate application of XML, and code bloat in general, I thought. After all, XML seems to be the default formatting choice for any messaging job these days, irrespective of performance, interoperability or code ownership requirements. And so much enterprise coding reduces to shuffling data between UI layer, database and messaging format.

But then I thought again. Remember the 8087 ? CPUs didn't always have onboard FPUs. If you needed to do a lot of number crunching, you added the floating point capability. Likewise with graphics. CPUs used to do the bit blitting. As the demand for graphics applications increased, dedicated cards came onto the market that relieved the CPU of the graphics processing. Could we be seeing a similar trend for text ? After all, the string processing workload for the typical PC has been ever rising for the last ten years, with ever increasing volumes of email, HTML, XML etc. Maybe CPUs need specialised Text Processing Units alongside their FPUs and GPUs.

SSL & pymqi: I got an email about SSL and Les Smithson's pymqi this morning. I added some code to pymqi a couple of years ago, and it turns out that my code needs further extension to enable SSL support. I don't have time to do this work myself. But I thought I'd better blog my recommmendations so they're in the Google cache for anyone else who runs into this issue.

First off, you need IBM's SSL MQ sample. You can find this by Googling for ssl mq mc6c. It has some example code: amqscnxc.c, which shows the SSL config info being put into a struct that's passed to MQCONNX. The struct is an MQCNO which pymqi 4c doesn't implement. MQCNO holds a ptr to an MQCD, which I added to an earlier version of pymqi.

So to enable SSL configuration in pymqi, you must implement a Python class for the MQCNO struct, and allow it to be passed into QueueManager.connectWithOptions, and then into MQCONNX. The following changes to pymqi will be needed...

• Add class cno(MQOpts) to pymqi.py. Use my cd class as a template, but follow the structure of MQCNO.
• Change QueueManager.connectWithOptions() to add an extra param: an instance of your new cno class.
• Change pymqe_MQCONNX() in pymqe.c to extract the new param you added in 2, and pass it through to MQCONNX.

Insane programming: there's a myth about investment banking software commonly voiced by coders outside the pressure cooker world of trading systems development. It goes something like this: huge amounts of money are at stake, so the software must be super reliable and robust. Very often, the opposite is true. Time to market is the driving factor. Traders want functionality yesterday. If a bank can execute profitable business, it can afford to throw support staff at a flaky system. Bugginess will be tolerated if it means executing business that may have gone elsewhere.

In an earlier entry I mentioned that I don't blog about my day job, since wholesale banks are secretive and paranoid. Most other techies working on Wall St and the City of London keep quiet too. But some are starting to lift the lid. If you're curious about what it's like to code on a trading floor, you should be reading this blog.

Generators: are doing my head in ! Been coding up the generator based async IO approach I mentioned recently this afternoon. I want real, non blocking, multi threaded concurrency. I use generators to yield after sending down a socket. And then I resume when I get a Twisted callback telling me a response has arrived. That way no thread blocks waiting on a response. Less threads, more busyness.

Figuring out where to yield and when to return, and thinking about when I want resumable stack state, and when I don't, has been a head twisting experience. Which is great, because that means I am acquiring a non trivial design idiom. Last time I had this experience was my first multithreaded code in the late 90s. Before that it was OO in the early 90s. All good stuff.

Plan to throw six away: more levitt hacking last night. Really happy with the way the codebase is coming along at the moment. I've been working on this system since 2001. I've restructured and refactored so many times. For server infrastructure I started off with a plain medusa front end, and ZODB at the back. Then I switched to a Zope product approach, to leverage Zope's UI. Then I switched to a browser plug in implementation. Now I'm doing a standalone impl, that will probably use Twisted in a personal server deployment. And there were minor refactorings within those larger implementation decisions. I know Fred Brooks enjoined us to "plan to throw one away", but maybe I'm going too far. However, the upside is the implementation feels smaller and cleaner and far better factored.

clarkbw: my fave Skynyrd tune !

Generators: Python has them, and C# is proposing to add them. What are they ?

def GeneratorFunc( n):
    for i in range( n):
        yield i

 for i in GeneratorFunc( 3):
    print "i=%d" % i

The call to GeneratorFunc does not execute any of the code inside the GeneratorFunc. Instead it returns an iterator object that we can use in the for loop at the bottom. The for loop implicitly calls the iterator's next() method, which in turn executes the code inside GeneratorFunc. Every time this happens, the yield keyword supplies the value to be returned by the iterator's next() method. The important thing to remember is that all the local variables, and stack state, inside the GeneratorFunc body are preserved between invocations of the iterator's next() method. All this may be a little clearer if we invoke the iterator directly rather than relying on for to do it...

def GeneratorFunc( n):
    for i in range( n):
        yield i

 iter = GeneratorFunc( 3)
while 1:
    print "i=%d" % iter.next()

All well and good, you may say. But where can I apply this idiom ? I can see tremendous leverage in two common coding areas: recursive tree walks and blocking IO handler code. More later...

Financial technology blogging: I ply my trade as a software developer in the City of London, working for wholesale banks. So do many of the developers I know. Since wholesale banks tend to be secretive and paranoid, most of us who develop trading systems don't talk about it much in public forums. I don't talk about my professional work at all on line. But a few people are starting to put their heads above the parapet...

A highly talented colleague Quant dev in SF