Older blog entries for dan (starting at number 145)

Testing a monolithic app - how not to

In the process of redesiging the interfaces to thin-prefork, I thought that if it’s going to be a design not a doodle I’d try to do it the TDD way and add some of that rspec goodness.

I’m not so proud of what I ended up with

There are a number of issues with this code that are all kind of overlapped and linked with each other, and this post is, unless it sits as a draft for considerably longer than I intended to spend on it, going to be kind of inchoate because all I really plan to do is list them in the order they occur to me.

  • The first and most obvious hurdle is that once you call #run!, the server process and its kids go off and don’t come back: in real-world use, any interaction you might have with it after that is driven by external events (such as signals). In testing, we have to control the external environment of the server to give it the right stimuli at the right time, then we need some way to look inside it and see how it reacts. So we fork and run it in a child process. (Just to remind you, thin-prefork is a forking server, so we now have a parent and a child and some grandchildren.) This is messy already and leads to heuristics and potential race conditions: for example, there is a sleep 2 after the fork, which we hope is long enough for it to be ready after we fork it, but is sure to fail somewhere and to be annoyingly and unnecessarily long somewhere else especially as the number of tests grows.
  • We make some effort to kill the server off when we’re done, but it’s not robust: if the interpreter dies, for example, we may end up with random grandchild processes lying around and listening to TCP ports, and that means that future runs fail too.
  • Binding a socket to a particular interface is (in Unix-land) pretty portable. Determining what interfaces are available to bind to, less so. I rely on there most likely being a working loopback and hope that there is additionally another interface on which packets to github.com can be routed. I’m sure that’s not always true, but it;‘ll have to do for now. (Once again I am indebted to coderr’s neat trick for getting the local IP address – and no, gethostbyname(gethostname()) doesn’t work on a mobile or a badly-configured system where the hostname may be an alias for in /etc/hosts/)
  • We need the test stanzas (running in the parent code) somehow to call arbitrary methods on the server object (which exists in the child). I know, we’ll make our helper method start accept a block and install another signal handler in the child which yields to it. Ugh
  • We needed a way to determine whether child processes have run the correct code for the commands we’re testing on them. Best idea I came up with was to have the command implementation and hook code set global variables, then do HTTP requests to the children which serve the value of those global variables. I’m sort of pleased with this. In a way.

Overall I think the process has been useful, but the end result feels brittle, it’s taken nearly as long as the code did to write, and it’s still not giving me the confidence to refactor (or indeed to rewrite) blindly that all the TDD/BDD advocates promote as the raison d’embêter

The brighter news is, perhaps, that I’m a lot more comfortable about the hook/event protocol this time round. There are still bits that need filling in, but have a look at Thin::Prefork::Worker::Lifecycle and module TestKidHooks for the worker lifecycle hooks, and then at the modules with names starting Test... for the nucleus of how to add a custom command.

Syndicated 2011-05-11 15:56:25 from diary at Telent Netowrks

HTC Desire S, unlocked, new, £359

No, I’m not selling it at that price. I just bought it at that price. Amazon and Handtec are both advertising it at about £371 and other well-known online retailer (expansys, etc) at even more: what I did was notice that the O2 Store sell it for £349 plus £10 topup, and that Carphone Warehouse have recently introduced a Network Price Promise so even though their advertised price for the same phone and tariff is £399 they will give it to you for the £359 price if you insist. And because it’s CPW and not a tied shop, they will (well, most probably will, and certainly did in my case) give you the unlocked and unbranded handset. In fairness I should say I don’t know whether the handset in the O2 store would have been locked or not.

I’ve only had a few minutes to play with the phone so far so haven’t formed a strong opinion (HTC Sense may have to go …) on it yet, but it’s already very clearly an upgrade from my really rather elderly T-Mobile G1

Syndicated 2011-05-09 14:06:27 from diary at Telent Netowrks

Sinatra and the class/instance distinction

The Sinatra microframework is described as enabling “Classy Web Development”, and it turns out this is more literally true than I previously thought.

The Rack Specification says

A Rack application is an Ruby object (not a class) that responds to call. It takes exactly one argument, the environment and returns an Array of exactly three values: The status, the headers, and the body.

(emphasis mine). When you write a Sinatra app, though, it seems to want a class: whether you call MyApp.run! directly (we assume throughout this post that MyApp is a Sinatra::Base subclass) or use a config.ru or any other way to start the app running, there is a conspicuous lack of MyApp.new anywhere around. Yet the Rack spec says an app is an instance.

At first I thought I was being silly or didn’t understand how Rack works or had in general just misunderstood something, but it turns out not. Some ferretting through Sinatra source code is needed to see how it does this, but the bottom line is that MyApp has a class method MyApp.call which rack invokes, and this delegates to (after first, if necessary, instantiating) a singleton instance of MyApp stored in the prototype field. I am not at all sure why they did this. It may just be a hangover from Sinatra’s heritage and this stuff came along for the ride when Sinatra::Base was factored out of the Sinatra::Application classic app support. Or they may have a perfectly good reason (this is the hypothesis I am leaning towards and I suspect that “Rack middleware pipelines” is that reason). For my purposes currently it’s probably sufficient to know that they do it without needing to know why, and that I should stop trying to write Sinatra::Base subclasses which takes extra parameters to new.

:; irb -r sinatra/base
ruby-1.9.2-p0 > class MyApp  nil 
ruby-1.9.2-p0 > MyApp.respond_to?(:call)
 => true 
ruby-1.9.2-p0 > begin; MyApp.call({}); rescue Exception => e ;nil;end
 => nil 
ruby-1.9.2-p0 > MyApp.prototype.class
 => Sinatra::ShowExceptions 

Ta, and with emphasis, da! (The begin/end around MyApp.call is because for the purpose of this example I am too lazy to craft a legitimate rack environment argument and just want to demonstrate that prototype is created. And we should not be surprised that the prototype’s class is not the same class as we created, because there is middleware chained to it. In summary, this example may be more confusing in its incidentals than it is illuminating in its essentials. Oh well)

Syndicated 2011-05-04 12:39:05 from diary at Telent Netowrks

Preforking multi-process Sinatra serving (with Sequel)

Picture the scene. I have a largish Ruby web application (actually, a combination of several apps, all based on Sinatra, sharing a model layer, and tied together with Rack::URLMap), and I want a better way of reloading it on my development laptop when the files comprising it change.

At the same time, I have a largish Ruby web application (etc etc) and I wanted a better way of running several instances of it on the same machine on different ports, because running a one-request-at-a-time web server in production is not especially prudent if you can’t guarantee that (a) it will always generate a response very very quickly, and (b) there is no way that slow clients can stall it. So, I needed something like the thin command, but with more hooks to do stuff at worker startup time that I need to do but won’t bore you with.

And in the what-the-hell-why-not department I see no good reason that I shouldn’t be using the same code in development as is running in production and plenty of good reasons that I should. And a program that basically fork()s three times (for user-specified values of three) can’t be that hard to write, can it?

Version 0 of “thin-prefork” kind of escaped onto github and contains the germ of a good idea plus two big problems and an exceedingly boring name.

What’s good about it? It consists of a parent process and some workers that are started by fork(). There is a protocol for the master to send control messages to the workers over a socket (start, stop, reload, and basically whatever else you decide), and you subclass the Worker to implement these commands. This was found to be necessary, because version -1 used signals between parent and child, and it was found eventually and empirically that EventMachine (or thin, or something else somewhere in the stack) likes to install signal handlers that overwrote the ones I was depending on. And at that point I had two commands which each needed a signal and in accordance with the Zero-One-Infinity Rule I could easily foresee a future in which I would run out of spare Unix signals.

What’s not so good? Reloading – ironically, the whole reason we set out to write the thing. Reloading is implemented by having the master send a control message to the children, and the children then reload themselves (using Projectr or however else you want to). But when you have 300MB x n children to reload you’d much rather do the reload once in the parent and then kill and respawn the kids than you would have each of the kids go off and do it themselves – that way lies Thrash City, which is a better place for skateboarders than servers. (This would also be a bad thing for sharing pages between parent and child, but I am informed by someone who sounded convincingly knowledgeable that the garbage collector in MRI writes into pretty much every page anyway thus spitting all over COW, so this turns out not to be a concern at present. But someday, maybe – and in the meantime it’s still kinda ugly)

What’s also not so good is that the interaction between “baked in” stuff that needs to happen for some actions – like “quit” – and user-specified customizations is kind of fuzzy and it’s not presently at all obvious if, for example, a worker subclass command should call super: if you want to do somewthing before quitting, then obviously you should then hand off to the superclass to actually exit, but if you want to define a reload handler then you don’t want to call a non-existent superclass method when you’re done. But how do you know it doesn’t exist? Your worker might be based off another customisation that does want to do something important at reload time. So it’s back to the drawing board to work out the protocol there, though rereading what I’ve just written it sounds like I should make a distinction between notifiers and command implementations - “tell me when X is happening because I need to do something” vs “this is the code you should run to implement X”.

And why does the post title namecheck Sequel? Because my experience with other platforms is that holding database handles open across a fork() call can be somewhat fraught and I wanted somewhere to document everything I know about how Sequel handles this

Syndicated 2011-05-03 15:11:12 from diary at Telent Netowrks

Introducing Projectr

Why might you want to know the names of all the files in your project? One might turn the question around and ask why would you possibly would not want to, but maybe that’s not a constructive dialogue. So let’s list some use cases

  • to load them into your application
  • to load them into irb for debugging or for help in constructing test cases
  • to process them through rdoc
  • to put them in a gem
  • to print them (don’t laugh, I did that the other day when I was having trouble deciding how to refactor a bunch of stuff)

As far as I can see from my survey of the Ruby world, the current practices for each of these use cases are pretty ad hoc. Maybe you write a file full of require or require_relative statements (as the RBP blog author likes to do), maybe you use a glob, maybe you write a MANIFEST file, but there seems to be a significant lack of DRYness about it all. This led me to think there is a gap in the market for

  1. a language for describing the files that a project comprises
  2. some tools to interrogate a project description written in this form and find out what’s in it
  3. some code to load them into a running interpreter – and for bonus points, when files have previously been loaded into said image but since changed on disk, to reload them. This could be used in irb sessions, or could form the basis of a development-oriented web server that reloads changed files without needing to be stopped and started all the time

Note that item 3 above gives us something that “file containing list of require statements” doesn’t, because it allows us to reload files that we’ve already seen instead of just saying “meh, seen it already”. If you’re using a comparatively low-powered machine then reloading your entire app in irb every time you change a method definition is unnecessarily and obviously slow. If you’re also using Bundler (which I rather like now i’s settled down a bit, and will write more about in a future entry) then the additional bundle exec is not just slow, it’s SLow with a capital S and a capital L and a pulsating ever-growing O that rules from the centre of the underworld.

Here’s one I made earlier

Projectr::Project.new :test do
  # directories may be named by symbols or strings
  directory :example do
    #as may files
    file "file1"
    file :file2
    directory "subdir" do 
      file :subdir_file

h.load!   # it loads all the files
# and again
h.load!   # nothing happens this time
# touch example/file1.rb
h.load!   # loads only the changed file

At the time of writing this, the github version does about that much, but is quite clearly still version 0. Stuff I am still thinking about:

  • Load-order dependencies. Lisp programmers may recognise that Projectr was inspired by using (and indeed implementing a version of) defsystem (or more recently here) but Projectr is almost minimally featured compared to any of the Lisp-based defsystem facilities. Many of those features I don’t have any strong evidence that the Ruby world would find use for, but load-order dependencies allow us to say for example that if file A defines a DSL and files B and C use that DSL, changing A should make the computer reload B and C as well
  • It seems clear to me that defining a project and loading it are two separate operations – you may wish instead to define it and then generate a Gemspec, for example – but there’s still a lot of verbiage in the common case that you do want to load it, and I haven’t really found file layout and naming conventions that I feel good about
  • likewise, what happens when we redefine the project itself (as would happen if we want to add a file to it, for example) is slightly up for grabs. Should the project definition file be considered a part of the project?

I will doubtless form my own opinions on all of these issues in time and with more experience of using this tool in practice, but feedback on them and on the general approach is warmly welcomed.

Fork, clone, spindle, mutilate

Syndicated 2011-05-02 09:39:18 from diary at Telent Netowrks

I did it my way

short unoriginal observation on ruby blogging engines: quicker to write your own than evaluate all the other poorly documented ones
… this observation only holds if you skimp on the documentation of course. which is where we came in

If you can see this, you can see my blog design all changed again. This time it’s a Ruby Sinatra application (whence the name my-way) running on thin-prefork, which keeps the article texts in git and uses RedCloth plus some ugly regexps to turn them into HTML. The Markdown vs Textile decision is not an especially interesting one in the first place, but gets a lot easier still when you have something like 9 years worth of previous articles in Textile format.

Publishing is achieved by pushing to a git repository on the live machine (a Bytemark vm). A post-update hook in the remote repository is responsible for checking out the updated commit (git doesn’t like pushing to non-bare repositories) and sending SIGHUP to the running instance of my-way which causes it to reindex files.

dan@bytemark:~$ cat /home/git/my-way.git/hooks/post-update        
GIT_WORK_TREE=/home/dan/src/git/my-way git checkout -f
kill -1 `cat /tmp/my-way.pid`

The version of my-way on github lags the actual version slightly, because I need to separate the engine from the articles and from the config data (there are things like adsense subscriber id, flickr api keys, etc) before I push the latter to a public service. Will clean it up in the next few days.

And my apologies to RSS feed subscribers. I’ve finally dropped the /diary prefix on the URL for this blog, and the old RSS feed didn’t use GUIDs and I’m too lazy to make the new one do so either, so the upshot is you just got the ten most recent articles in your feed again. Sorry.

Syndicated 2011-05-01 10:50:35 from diary at Telent Netowrks

What I miss most about Lisp

It’s been three months since I wrote anything longer than one line in Lisp, and over a year since I wrote more than a screenful of the stuff.

What I miss most is not CLOS or the REPL or even macros (per se, anyway). It’s

  • the distinction between READ and EVAL: a sane syntax for constructing complex data structures that look like code, but without actually having the data structures in question interpreted
  • and backtraces with the values of function parameters in them. When you’re doing the same thing 1000 times to different database rows or objects in a collection, and one of them has a nil in it somewhere, it would be really nice to know which one.

And maybe the REPL (although irb does most of that). And kinda sorta Defsystem, but I seem to be in the process of reimplementing that

But I hope soon to get Rubinius installed, just because I still have the irrational opinion that a grown-up programming language ought to be able to implement itself (and I have a thing for native code) so project 1 there is to see if I can hack the backtrace thingy at least into it.

Syndicated 2011-04-30 22:23:33 from diary at Telent Netowrks

TDD, BDD, executable specification

The new system at $WORK finally went live about a week ago, hurrah.

The upgrade itself took a few hours longer than I'd have liked, and (short shameful confession time), some (but probably not all) of this could have been caught by better test coverage. Which set me on a path towards SimpleCov (I'm using Ruby 1.9, rcov doesn't work), which led me to start looking at the uncovered parts, which set me to thinking. Which, as we all know, is dangerous.

TDD advocates (and pro-testing people in general) say "Don't test accessors". There are two reasons to say this that seem to my mind like good reasons: Ron says it because he wants you to write tests that do something else (something useful) that happens to involve calling those accessors. J B Rainsberger says it because "get/set methods just can't break, and if they can't break, then why test them?"

The problem comes when you adopt the mindset typified by BDD that "the test examples are actually your executable specification", because in that case how do you specify that the object has an accessor? This is not an unreasonable demand. Suppose we have objects whose purpose is to store structured data that will be used by client code - for example, User has an age property. Jbrains - which must surely be the Best Nickname Ever for a Java guy - says there's no useful test you can write for this (or not unless you don't trust your platform or something, but that way madness lies). But even if we are going to write one: a test that stores one or a few example values can easily be faked by the bloody-minded implementor ("the setter is called with the argument 42, then the getter is called and should return 42? I know! def age; 42; end") and a test that stores all possible values and tests they can be retrieved will take forever to write/read/run. Really the best notation in which to specify the behaviour of that property is the same notation which, when run by the interpreter, will implement the said behavior -

    attr_accessor :age
It's not just accessors either. Everything on the continuum between declarations of constants (SECONDS_PER_DAY=86400, are you really going to write a test for that?) and simple mathematical formulae
    class Triangle
      def area
        self.base * self.height / 2.0
are most readably expressed to humans as, well, the continuous functions that they are, not the three or four example data points that we might write examples to test for. For any finite number of test cases, you can write a giant case...when statement that passes all of them and still doesn't work in the general case.

Yes, we could and often should write a couple of tests just to make sure we haven't done anything boneheaded in implementing the function, but they're not spec. They're just examples.

But here's the rub: where or how do we put that code to make it obvious that it's specification that happens also to be a valid implementation - and not just implementation that may or may not meet a spec expressed in some other place/form? If we're laying out our app in conventional Ruby style, it can't go in the spec/ directory because that doesn't actually get run as part of the application, and it shouldn't go in lib/ or models/ or wherever else unless we are prepared to make our clients rootle through all that code looking for whatever "this is specification not just an implementation detail" flag we decide to adopt) when they want to use our interface.

I'm going to make a suggestion which is either radical or bone-headed: we should smush the rspec-stuff together with the app code: embed examples (which may in some cases be specification and in other cases be "smoke tests") in the same files as the implementation (which may itself sometimes be specification and other times be the result of our fallible human attempts to derive implementation from spec), and then we can have some kind of annotations to say which is which, and then we can have some kind of rdoc-on-'roids literate programming tool (To Be Implemented) go through the whole lot and produce two separate documents. One for people who want to use our code and want to know what it should do, and the other for the people who have to hack on it and need to know how it does. Or doesn't. And then maybe we can have code coverage metrics that actually mean something.

Syndicated 2011-04-07 14:26:57 from diary at telent netowrks

Corner cases

As you see in the image, right, my notebook recently took a dive onto a laminate floor and ended up a trifle dog-eared. Amazingly, the hardware crash didn't provoke a software crash, but not wishing to take any chances with running it while bending the motherboard, I shut it down myself. Then I dismantled it and dismantled its predecessor , and swapped parts about between them to create one combined functional machine . In the interests of extending battery life I left the fingerprint reader disconnected and removed the original HDD, leaving only the second (and faster) 2.5" disk in the optical media bay.

The new old laptop (henceforth to be known as Igor) worked for a couple of days until I left it running on battery overnight, and when I reapplied power in the morning got a rather nasty-looking "Non-system disk or disk error" message. The usual tedious mucking about with rescue images on USB keys revealed that the disk was still there and fully working, but the BIOS was determined not to see it until I remembered: PATA has two disks on the same channel. Sure enough, after removing the 'slave' jumper on the disk drive I suddenly had hda where previously I had a hole where hdb used to be.

Getting back to a state where it would actually boot, though, that was the trick. Something in Debian's initramfs generation does some kind of magic to detect that the root fs is stored in a logical volume of a volume group of an LVM physical volume on a LUKS device-mapper layer over the physical device, but something in the Debian installer's rescue mode wouldn't let me run anything grub-related (it tried: it failed with unhelpful error messages) after mounting the disk on /target, and nothing in the Debian installer would let me reinstall / without also clobbering /home, which I wasn't really interested in restoring from two-day-old backup. The reason for this rant, though, is not to vent (I did that on Twitter last week) but to document somewhere that if you're booting off a disk that has changed its name or address since this magic stuff happened, you can override it with appropriate kernel command line parameters: in my case it was

where source is the raw disk, target is the name of the mapping that the DM encryption code will set up for the decrypted PV (if you don't understand that clause then take heart because I'm not sure I do either, but if you set things up the debian way it's probably your hostname), and lvm is the appropriate LV name inside that PV.

Then it booted! There were some interesting warnings and it asked me for my passphrase an extra few times for luck, and then I edited /etc/crypttab and /etc/fstab and purged and reinstalled the grub-pc and initramfs-tools packages. Which may or may not have been strictly necessary but by that stage seemed prudent.

Syndicated 2011-03-07 14:25:12 from diary at telent netowrks

The Neighbour-Net Proxy Protocol

Borrowing a riff from Charlie Stross' "books I will not write" meme, I present the first in a series of indeterminate length entitled "Software I will not implement".

This is the result of a couple of days thinking about how to do a distributed Facebook (or at least, the interesting bits thereof) originally inspired by Eben Moglen's Freedom in the Cloud talk last year, and my subsequent disappointment to see that our most publically well-known hope Diaspora were all gungho about their implementation but publically completely silent about the protocols. In my opinion a monoculture is not the way to a robust ecology.

So (per the opening para of this post) why aren't I implementing it? Purely and simply, a severe deficiency of Copious Free Time. I am posting what I've got publically in the hope that it triggers some good ideas in others. In the (perhaps unlikely) event that anyone reading this thinks it's an awesomely good idea and does have the time to drive it forwards, take it or fork it and let me know and I will of course be deliriously happy to flipflop on this position and shelve something else instead if I can contribute.

https://github.com/telent/nnpp # READ ME; READ ME NOT; READ ME; READ ME NOT;

Syndicated 2011-03-02 14:01:46 from diary at telent netowrks

136 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!