Recent blog entries for joey

then and now

It's 2004 and I'm in Oldenburg DE, working on the Debian Installer. Colin and I pair program on partman, its new partitioner, to get it into shape. We've somewhat reluctantly decided to use it. Partman is in some ways a beautful piece of work, a mass of semi-object-oriented, super extensible shell code that sprang fully formed from the brow of Anton. And in many ways, it's mad, full of sector alignment twiddling math implemented in tens of thousands of lines of shell script scattered amoung hundreds of tiny files that are impossible to keep straight. In the tiny Oldenburg Developers Meeting, full of obscure hardware and crazy intensity of ideas like porting Debian to VAXen, we hack late into the night, night after night, and crash on the floor.

sepia toned hackers round a table

It's 2015 and I'm at a Chinese bakery, then at the Berkeley pier, then in a SF food truck lot, catching half an hour here and there in my vacation to add some features to Propellor. Mostly writing down data types for things like filesystem formats, partition layouts, and then some small amount of haskell code to use them in generic ways. Putting these peices together and reusing stuff already in Propellor (like chroot creation).

Before long I have this, which is only 2 undefined functions away from (probably) working:

let chroot d = Chroot.debootstrapped (System (Debian Unstable) "amd64") mempty d
        & Apt.installed ["openssh-server"]
        & ...
    partitions = fitChrootSize MSDOS
        [ (Just "/boot", mkPartiton EXT2)
        , (Just "/", mkPartition EXT4)
        , (Nothing, const (mkPartition LinuxSwap (MegaBytes 256)))
 in Diskimage.built chroot partitions (grubBooted PC)

This is at least a replication of vmdebootstrap, generating a bootable disk image from that config and 400 lines of code, with enormous customizability of the disk image contents, using all the abilities of Propellor. But is also, effectively, a replication of everything partman is used for (aside from UI and RAID/LVM).

sailboat on the SF bay

What a difference a decade and better choices of architecture make! In many ways, this is the loosely coupled, extensible, highly configurable system partman aspired to be. Plus elegance. And I'm writing it on a lark, because I have some spare half hours in my vacation.

Past Debian Installer team lead Tollef stops by for lunch, I show him the code, and we have the conversation old d-i developers always have about partman.

I can't say that partman was a failure, because it's been used by millions to install Debian and Ubuntu and etc for a decade. Anything that deletes that many Windows partitions is a success. But it's been an unhappy success. Nobody has ever had a good time writing partman recipes; the code has grown duplication and unmaintainability.

I can't say that these extensions to Propellor will be a success; there's no plan here to replace Debian Installer (although with a few hundred more lines of code, propellor is d-i 2.0); indeed I'm just adding generic useful stuff and building further stuff out of it without any particular end goal. Perhaps that's the real difference.

Syndicated 2015-08-27 00:01:59 from see shy jo

sweet summer


Having a wonderful summer, full of simple sweet pleasures. Mom visited today, and I made her this blackberry chocolate tart. Picking berries, swimming in the river, perfect summer day.


Earlier this summer, camped at in the dunes on Ocracoke island with many family and friends. Thunderstorms away across the sound flashed and grumbled long in the night, but mostly missed us. Jupiter and Venus in conjunction overhead, and the arch of the milky way completed the show.


Syndicated 2015-07-11 20:23:31 from see shy jo

our beautiful fake histories

Here's an odd thing about the git bisect command: It has only 1 option (--no-checkout). Compare with eg git commit, which has 36 options by my count.

The difference is largely down to git having a pervasive culture of carefully edited history. We need lots of git commit options to carefully produce commits that look Just Right. Staging only some of the files we've edited, perhaps even staging only some of the changes within a file. Amend that commit if we notice we made a mistake. Create a whole series of beautiful commits, and use rebase later to remix them into a more beautiful whole.

Beautiful fake histories. Because coding is actually messy; our actual edit history contains blind alleys and doublings back on itself; contains periods of many days the code isn't building properly. We want to sweep that complexity away, hide it under the rug. This works well except when it doesn't, when some detail airbrushed out of the only remaining history turns out to be important.

Once we have these beautiful fake histories of changes, we can easily bisect them and find the commit that introduced a bug. So bisect doesn't need a lot of options to control how it works.

I'd like to suggest a new option though. At least as a thought experiment. --merges-only would make bisect only check the merge commits in the range of commits being bisected. The bisection would result in not a single commit, but in the set of commits between two merges.

I suspect this would be useful for faster bisecting some histories of the beautiful fake kind. But I know it would be useful when the history is messy and organic and full of false starts and points where the code doesn't build. Merges, in such histories, are often the points where things reach a certian level of beauty, where that messy feature branch got to the point it all built again (please let this happen today) and was merged into master. Bisecting such points in a messy organic history should work about as well as bisecting carefully gardened histories.

I think I'll save the full rant about beautiful fake history vs messy real history for some other day. Or maybe I've already ranted that rant here before, I can't remember.

Let's just say that I personally come down on the side of liking my git history to reflect the actual code I was working on, even if it was broken and even if I threw it away later. I've indeed taken this to extreme lengths with propellor; in its git history you can see every time I've ever run it, and the version of my config file and code at that point. Apologies to anyone who's been put off by that... But oddly, propellor gets by far more contributions from others than any of my other haskell programs.

All in the form of beaufiully constructed commits, naturally.

Syndicated 2015-07-10 15:40:23 from see shy jo

I am ArchiveTeam

This seems as good a day as any to mention that I am a founding member of ArchiveTeam.

ArchiveTeam logo

Way back, when Geocities was closing down, I was one of a small rag-tag group who saved a copy of most of it. That snapshot has since generated more publicity than most other projects I've worked on. I've heard many heartwarning stories of it being the only remaining copy of baby pictures and writings of deceased friends, and so on. It's even been the subject of serious academic study as outlined in this talk, which is pretty awesome.

Jason Scott in full stage regalia

I'm happy to let this guy be the public face of ArchiveTeam in internet meme-land. It's a 0.1% project for me, and has grown into a well-oiled machine, albeit one that shouldn't need to exist. I only get involved these days when there's another crazy internet silo fire drill and/or I'm bored.

(Rumors of me being the hand model for ArchiveTeam are, however, unsubstantiated.)

Syndicated 2015-04-01 17:50:49 from see shy jo

7drl 2015 day 7 scroll success

A frantic last day of work on Scroll.

Until 3 am last night, I was working on adding a new procedurally generated level.

This morning, fixed two major bugs reported by playesters overnight. Also fixed crashes on small screens and got the viewport to scroll. Added a victory animation in time for lunch.

After lunch, more level generation work. Wasted an entire hour tracking down a bug in level gen I introduced last night, when I was bad and didn't create a data type to express an idea. Added a third type of generated level, with its own feel.

Finished up with a level selection screen, which needed just 47 lines of code and features a playable character.

I have six hours until my 7drl is officially over, but I'm done! Success! You can download the code, play, etc, at Scroll's homepage

Syndicated 2015-03-13 22:27:53 from see shy jo

7drl 2015 day 6 must add more

Last night I put up a telnet server and web interface to play a demo of scroll and send me playtester feedback, and I've gotten that almost solid today. Try it!

Today was a scramble to add more features to Scroll and fix bugs. The game still needs some balancing, and generally seems a little too hard, so added a couple more spells, and a powerup feature to make it easier.

Added a way to learn new spells. Added a display of spell inventory on 'i'. For that, I had to write a quick windowing system (20 lines of code).

Added a system for ill effects from eating particular letters. Interestingly, since such a letter is immediately digested, it doesn't prevent the worm from moving forwards. So, the ill effects can be worth it in some situations. Up to the player to decide.

I'm spending a lot of time now looking at letter frequency historgrams to decide which letter to use for a new feature. Since I've several times accidentially used the same letter for two different things (most amusingly, I assigned 'k' to a spell, forgetting it was movement), I refactored all the code to have a single charSet which defines every letter and what it's used for, be that movement, control, spell casting, or ill effects. I'd like to use that to further randomize which letters are used for spell components, out of a set that have around the same frequency. However, I doubt that I'll have time to do that.

In the final push tonight/tomorrow, I hope to add an additional kind of level or two, make the curses viewport scroll when necessary instead of crashing, and hopefully work on game balance/playtester feedback.

I've written ~2800 lines of code so far this week!

Syndicated 2015-03-12 23:02:38 from see shy jo

7drl 2015 day 5 type directed spell system development

I want my 7drl game Scroll to have lots of interesting spells. So, as I'm designing its spell system, I've been looking at the types, and considering the whole universe of possible spells that fit within the constraints of the types.

My first throught was that a spell would be a function from World -> World. That allows any kind of spell that manipulates the game map. Like, for instance a "whiteout" that projects a stream of whitespace from the player's mouth.

Since Scroll has a state monad, I quickly generalized that; making spell actions a state monad M (), which lets spells reuse other monadic actions, and affect the whole game state, including the player. Now I could write a spell like "teleport", or "grow".

But it quickly became apparent this was too limiting: While spells could change the World map, the player, and even change the list of supported spells, they had no way to prompting for input.

I tried a few types of the Event -> M () variety, but they were all too limiting. Finally, I settled on this type for spell actions: M NextStep -> M NextStep.

And then I spent 3 hours exploring the universe of spells that type allows! To understand them, it helps to see what a NextStep is:

type Step = Event -> M NextStep
data NextStep = NextStep View (Maybe Step)

Since NextStep is a continuation, spells take the original continuation, and can not only modify the game state, but can return an altered continuation. Such as one that prompts for input before performing the spell, and then calls the original continuation to get on with the game.

That let me write "new", a most interesting spell, that lets the player add a new way to cast an existing spell. Spells are cast using ingredients, and so this prompts for a new ingredient to cast a spell. (I hope that "new farming" will be one mode of play to possibly win Scroll.)

And, it lets me write spells that fail in game-ending ways. (Ie, "genocide @"). A spell can cause the game to end by returning a continuation that has Nothing as its next step.

Even better, I could write spells that return a continuation that contains a forked background task, using the 66 line contiuation based threading system I built in day 3. This allows writing lots of fun spells that have an effect that lasts for a while. Things like letting the player quickly digest letters they eat, or slow down the speed of events.

And then I thought of "dream". This spell stores the input continuation and game state, and returns a modified continuation that lets the game continue until it ends, and then restores from the point it saved. So, the player dreams they're playing, and wakes back up where they cast the spell. A wild spell, which can have different variants, like precognitive dreams where the same random numbers are used as will be used upon awaking, or dreams where knowledge carries back over to the real world in different ways. (Supports Inception too..)

Look how easy it was to implement dreaming, in this game that didn't have any notion of "save" or "restore"!

runDream :: M NextStep -> M NextStep -> (S -> S) -> M NextStep
runDream sleepcont wakecont wakeupstate = go =<< sleepcont
         go (NextStep v ms) = return $ NextStep v $ Just $
        maybe wake (go <=<) ms
         wake _evt = do
                 modify wakeupstate

I imagine that, if I were not using Haskell, I'd have just made the spell be an action, that can do IO in arbitrary ways. Such a spell system can of course do everything I described above and more. But, I think that using a general IO action is so broad that it hides the interesting possibilities like "dream".

By starting with a limited type for spells, and exploring toward more featureful types, I was able to think about the range of possibilities of spells that each type allowed, be inspired with interesting ideas, and implement them quickly.

Just what I need when writing a roguelike in just 7 days!

Syndicated 2015-03-11 22:51:26 from see shy jo

7drl 2015 day 4 coding through exhaustion

Slow start today; I was pretty exhausted after yesterday and last night's work. Somehow though, I got past the burn and made major progress today.

All the complex movement of both the player and the scroll is finished now, and all that remains is to write interesting spells, and a system for learning spells, and to balance out the game difficulty.

I haven't quite said what Scroll is about yet, let's fix that:

In Scroll, you're a bookworm that's stuck on a scroll. You have to dodge between words and use spells to make your way down the page as the scroll is read. Go too slow and you'll get wound up in the scroll and crushed.

The character is multiple chars in size (size is the worm's only stat), and the worm interacts with the scroll in lots of ways, like swallowing letters, or diving through a hole to the other side of the scroll. While it can swallow some letters, if it gets too full, it can't move forward anymore, so letters are mostly consumed to be used as spell components.

I think that I will manage to get away without adding any kind of monsters to the game; the scroll (and whoever is reading it) is the antagonist.

As I'm writing this very post, I'm imagining the worm wending its way through my paragraphs. This dual experience of text, where you're both reading its content and hyper-aware of its form, is probably the main thing I wanted to explore in writing Scroll.

As to the text that fills the scroll, it's broadly procedurally generated, in what I hope are unusual and repeatedly surprising (and amusing) ways. I'm not showing any screenshots of the real text, because I don't want to give that surprise away. But, the other thing about Scroll is that it's scroll, a completely usable (if rather difficult..) Unix pager!

Syndicated 2015-03-10 23:07:45 from see shy jo

7drl 2015 day 3 movement at last

Got the player moving in the map! And, got the map to be deadly in its own special way.

        HeadCrush -> do
                showMessage "You die."

Even winning the game is implemented. The game has a beginning, a middle, and an end.

I left the player movement mostly unconstrained, today, while I was working on things to do with the end of the game, since that makes it easier to play through and test them. Tomorrow, I will turn on fully constrained movement (an easy change), implement inventory (which is very connected to movement constraints in Scroll), and hope to start on the spell system too.

At this point, Scroll is 622 lines of code, including content. Of which, I notice, fully 119 are types and type classes.

Only 4 days left! Eep! I'm very glad that scroll's central antagonist is already written. I don't plan to add other creatures, which will save some time.

Last night as I was drifting off to sleep, it came to me a way to implement my own threading system for my roguelike. Since time in a roguelike happens in discrete ticks, as the player takes each action, normal OS threads are not suitable. And in my case, I'm doing everything in pure code anyway and certianly cannot fork off a thread for some background job.

But, since I'm using continuation passing style, I can just write my own fork, that takes two continuations and combines them, causing both to be run on each tick, and recursing to handle combining the resulting continuations.

It was really quite simple to implement. Typechecked on the first try even!

fork :: M NextStep -> M NextStep -> M NextStep
fork job rest = do
        jn <- job
        rn <- rest
        runthread jn rn
        runthread (NextStep _ (Just contjob)) (NextStep v (Just contr)) =
                return $ NextStep v $ Just $ \i -> do
                        jn <- contjob i
                        rn <- contr i
                        runthread jn rn
        runthread (NextStep _ Nothing) (NextStep v (Just contr)) =
                return $ NextStep v (Just contr)
        runthread _ (NextStep v Nothing) =
                return $ NextStep v Nothing

endThread :: M NextStep
endThread = nextStep Nothing

background :: M NextStep -> M NextStep
background job = fork job continue

demo :: M NextStep
demo = do
    showMessage "foo"
    background $ next $ const $
        clearMessage >> endThread

That has some warts, but it's good enough for my purposes, and pretty awesome for a threading system in 66 LOC.

Syndicated 2015-03-09 23:33:59 from see shy jo

7drl 2015 day 2 level generation and game concept

Much as I want to get my @ displayed and moving around screen, last night and today have focused on level generation instead.

Scroll has kinda strange level generation method, compared to how I suppose most roguelikes do it. There are only 3 calls to rand in all of Scroll. Just a 3 random parameters, but that's enough to ensure a different level each time.

-- Random level generation function.
level :: Bool -> StdGen -> [String]
level randomize r = concat
        [ final (length tutorial + extra) $ concat $ rand mariner1body
        , concat $ rand mariner1end
        , concatMap rand kubla
    -- here be spoilers

You could say there are two influences in Scroll's level generation method: Nick Montfort and Samuel Taylor Coleridge.

I have thought some about Scroll before starting the 7drl week, but my idea for the game was missing some key concepts. There was nothing to keep the player engaged in moving forward, an unclear victory condition, no clue how to generate appropriate random levels, a large potential for getting stuck, and no way to lose the game. This is all a problem for a roguelike.

But, I had an idea finally last night, about a single unified thing that all of that stuff falls out from. And it's right there in the name of the game!

Now that I understand Scroll better, I wrote the tutorial level. It's a very meta tutorial level that sets the scene well and serves more purposes than are at first apparent. I count a total of 6 things that this "tutorial level" will let the user do.

And interestingly, while the tutorial level is static, it interacts with the rest of the game in a way that will make it be experienced differently every time through.

The strangest line of code I wrote today is:

  import GPL

Somehow, I have never before, in all my time programming, written a line like that one.

Finally, after 7 hours of nonstop coding, I got ncurses to display the generated game world, scrolling around in the display viewport. No @ yet; that will need to wait for tonight or tomorrow!

Syndicated 2015-03-08 23:19:56 from see shy jo

574 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!