joey is currently certified at Master level.

Name: Joey Hess
Member since: 2000-03-06 23:42:41
Last Login: 2011-12-31 20:04:52

FOAF RDF Share This



Recent blog entries by joey

Syndication: RSS 2.0

STM Region contents

concurrent-output released yesterday got a lot of fun features. It now does full curses-style minimization of the output, to redraw updated lines with optimal efficiency. And supports multiline regions/wrapping too long lines. And allows the user to embed ANSI colors in a region. 3 features that are in some tension and were fun to implement all together.

But I have a more interesting feature to blog about... I've added the ability for the content of a Region to be determined by a (STM transaction).

Here, for example, is a region that's a clock:

timeDisplay :: TVar UTCTime -> STM Text
timeDisplay tv = T.pack . show <$> readTVar tv

clockRegion :: IO ConsoleRegionHandle
clockRegion = do
    tv <- atomically . newTVar =<< getCurrentTime
    r <- openConsoleRegion Linear
    setConsoleRegion r (timeDisplay tv)
    async $ forever $ do
        threadDelay 1000000 -- 1 sec
        atomically . (writeTVar tv) =<< getCurrentTime
    return r

There's something magical about this. Whenever a new value is written into the TVar, concurrent-output automatically knows that this region needs to be updated. How does it know how to do that?

Magic of STM. Basically, concurrent-output composes all the STM transactions of Regions, and asks STM to wait until there's something new to display. STM keeps track of whatever TVars might be looked at, and so can put the display thread to sleep until there's a change to display.

Using STM I've gotten extensability for free, due to the nice ways that STM transactions compose.

A few other obvious things to do with this: Compose 2 regions with padding so they display on the same line, left and right aligned. Trim a region's content to the display width. (Handily exported by concurrent-output in a TVar for this kind of thing.)

I'm tempted to write a console spreadsheet using this. Each visible cell of the spreadsheet would have its own region, that uses a STM transaction to display. Plain data Cells would just display their current value. Cells that contain a function would read the current values of other Cells, and use that to calculate what to display. Which means that a Cell containing a function would automatically update whenever any of the Cells that it depends on were updated!

Do you think that a simple interactive spreadsheet built this way would be more than 100 lines of code?

Syndicated 2015-11-03 20:03:53 from see shy jo

a tiling region manager for the console

Building on top of concurrent-output, and some related work Joachim Breitner did earlier, I now have a kind of equivilant to a tiling window manager, except it's managing regions of the console for different parts of a single program.

Here's a really silly demo, in an animated gif:


Not bad for 23 lines of code, is that? Seems much less tedious to do things this way than using ncurses. Even with its panels, ncurses requires you to think about layout of various things on the screen, and many low-level details. This, by contrast, is compositional, just add another region and a thread to update it, and away it goes.

So, here's an apt-like download progress display, in 30 lines of code.


Not only does it have regions which are individual lines of the screen, but those can have sub-regions within them as seen here (and so on).

And, log-type messages automatically scroll up above the regions. External programs run by createProcessConcurrent will automatically get their output/errors displayed there, too.

What I'm working on now is support for multiline regions, which automatically grow/shrink to fit what's placed in them. The hard part, which I'm putting the finishing touches on, is to accurately work out how large a region is before displaying it, in order to lay it out. Requires parsing ANSI codes amoung other things.

STM rules

There's so much concurrency, with complicated interrelated data being updated by different threads, that I couldn't have possibly built this without Software Transactional Memory.

Rather than a nightmare of locks behind locks behind locks, the result is so well behaved that I'm confident that anyone who needs more control over the region layout, or wants to do funky things can dive into to the STM interface and update the data structures, and nothing will ever deadlock or be inconsistent, and as soon as an update completes, it'll display on-screen.

An example of how powerful and beuatiful STM is, here's how the main display thread determines when it needs to refresh the display:

data DisplayChange
        = BufferChange [(StdHandle, OutputBuffer)]
        | RegionChange RegionSnapshot
        | TerminalResize (Maybe Width)
        | EndSignal ()

                change <- atomically $
                        (RegionChange <$> regionWaiter origsnapshot)
                        (RegionChange <$> regionListWaiter origsnapshot)
                        (BufferChange <$> outputBufferWaiterSTM waitCompleteLines)
                        (TerminalResize <$> waitwidthchange)
                        (EndSignal <$> waitTSem endsignal)
                case change of
                        RegionChange snapshot -> do
                        BufferChange buffers -> do
                        TerminalResize width -> do

So, it composes all these STM actions that can wait on various kinds of changes, to get one big action, that waits for all of the above, and builds up a nice sum type to represent what's changed.

Another example is that the whole support for sub-regions only involved adding 30 lines of code, all of it using STM, and it worked 100% the first time.

Available in concurrent-output 1.1.0.

Syndicated 2015-10-31 01:44:47 from see shy jo

concurrent output library

concurrent-output is a Haskell library I've developed this week, to make it easier to write console programs that do a lot of different things concurrently, and want to serialize concurrent outputs sanely.

It's increasingly easy to write concurrent programs, but all their status reporting has to feed back through the good old console, which is still obstinately serial.

Haskell illustrates problem this well with this "Linus's first kernel" equivilant interleaving the output of 2 threads:

  > import System.IO
> import Control.Concurrent.Async
> putStrLn (repeat 'A') `concurrently` putStrLn (repeat 'B')

That's fun, but also horrible if you wanted to display some messages to the user:

  > putStrLn "washed the car" `concurrently` putStrLn "walked the dog"
walwkaesdh etdh et hdeo gc

To add to the problem, we often want to run separate programs concurrently, which have output of their own to display. And, just to keep things interesting, sometimes a unix program will behave differently when stdout is not connected to a terminal (eg, ls | cat).

To tame simple concurrent programs like these so they generate readable output involves a lot of plumbing. Something like, run the actions concurrently, taking care to capture the output of any commands, and then feed the output that the user should see though some sort of serializing channel to the display. Dealing with that when you just wanted a simple concurrent program risks ending up with a not-so-simple program.

So, I wanted an library with basically 2 functions:

outputConcurrent :: String -> IO ()
createProcessConcurrent :: CreateProcess -> IO whatever

The idea is, you make your program use outputConcurrent to display all its output, and each String you pass to that will be displayed serially, without getting mixed up with any other concurrent output.

And, you make your program use createProcessConcurrent everywhere it starts a process that might output to stdout or stderr, and it'll likewise make sure its output is displayed serially.

Oh, and createProcessConcurrent should avoid redirecting stdout and stderr away from the console, when no other concurrent output is happening. So, if programs are mostly run sequentially, they behave as they normally would at the console; any behavior changes should only occur when there is concurrency. (It might also be nice for it to allocate ttys and run programs there to avoid any behavior changes at all, although I have not tried to do that.)

And that should be pretty much the whole API, although it's ok if it needs some function called by main to set it up:

import Control.Concurrent.Async
import Control.Concurrent.Output

main = withConcurrentOutput $
    outputConcurrent "washed the car\n"
    createProcessConcurrent (proc "ls" [])
    outputConcurrent "walked the dog\n"
  $ ./demo
washed the car
walked the dog
Maildir/  bin/  doc/  html/  lib/  mail/  mnt/  src/  tmp/

I think that's a pretty good API to deal with this concurrent output problem. Anyone know of any other attempts at this I could learn from?

I implemented this over the past 3 days and 320 lines of code. It got rather hairy:

  • It has to do buffering of the output.
  • There can be any quantity of output, but program memory use should be reasonably small. Solved by buffering up to 1 mb of output in RAM, and writing excess buffer to temp files.
  • Falling off the end of the program is complicated; there can be buffered output to flush and it may have to wait for some processes to finish running etc.
  • The locking was tough to get right! I could not have managed to write it correctly without STM.

It seems to work pretty great though. I got Propellor using it, and Propellor can now run actions concurrently!

Syndicated 2015-10-29 02:07:34 from see shy jo

propelling disk images

Following up on Then and Now ...

In quiet moments at ICFP last August, I finished teaching Propellor to generate disk images. With an emphasis on doing a whole lot with very little new code and extreme amount of code reuse.

For example, let's make a disk image with nethack on it. First, we need to define a chroot. Disk image creation reuses propellor's chroot support, described back in propelling containers. Any propellor properties can be assigned to the chroot, so it's easy to describe the system we want.

 nethackChroot :: FilePath -> Chroot
    nethackChroot d = Chroot.debootstrapped (System (Debian Stable) "amd64") mempty d
        & Apt.installed ["linux-image-amd64"]
        & Apt.installed ["nethack-console"]
        & accountFor gamer
        & gamer `hasInsecurePassword` "hello"
        & gamer `hasLoginShell` "/usr/games/nethack"
      where gamer = User "gamer"

Now to make an image from that chroot, we just have to tell propellor where to put the image file, some partitioning information, and to make it boot using grub.

 nethackImage :: RevertableProperty
    nethackImage = imageBuilt "/srv/images/nethack.img" nethackChroot
        MSDOS (grubBooted PC)
        [ partition EXT2 `mountedAt` "/boot"
            `setFlag` BootFlag
        , partition EXT4 `mountedAt` "/"
            `addFreeSpace` MegaBytes 100
        , swapPartition (MegaBytes 256)

The disk image partitions default to being sized to fit exactly the files from the chroot that go into each partition, so, the disk image is as small as possible by default. There's a little DSL to configure the partitions. To give control over the partition size, it has some functions, like addFreeSpace and setSize. Other functions like setFlag and extended can further adjust the partitions. I think that worked out rather well; the partition specification is compact and avoids unecessary hardcoded sizes, while providing plenty of control.

By the end of ICFP, I had Propellor building complete disk images, but no boot loader installed on them.

Fast forward to today. After stuggling with some strange grub behavior, I found a working method to install grub onto a disk image.

The whole disk image feature weighs in at:

203 lines to interface with parted
88 lines to format and mount partitions
90 lines for the partition table specification DSL and partition sizing
196 lines to generate disk images
75 lines to install grub on a disk image
652 lines of code total

Which is about half the size of vmdebootstrap 1/4th the size of partman-base (probably 1/100th the size of total partman), and 1/13th the size of live-build. All of which do similar things, in ways that seem to me to be much less flexible than Propellor.

One thing I'm considering doing is extending this so Propellor can use qemu-user-static to create disk images for eg, arm. Add some u-boot setup, and this could create bootable images for arm boards. A library of configs for various arm boards could then be included in Propellor. This would be a lot easier than running the Debian Installer on an arm board.

Oh! I only just now realized that if you have a propellor host configured, like this example for my dialup gateway, leech --

 leech = host ""
        & os (System (Debian (Stable "jessie")) "armel")
        & Apt.installed ["linux-image-kirkwood", "ppp", "screen", "iftop"]
        & privContent "/etc/ppp/peers/provider"
        & privContent "/etc/ppp/pap-secrets"
        & Ppp.onBoot
        & hasPassword (User "root")
        & Ssh.installed

-- The host's properties can be extracted from it, using eg hostProperties leech and reused to create a disk image with the same properties as the host!

So, when my dialup gateway gets struck by lightning again, I could use this to build a disk image for its replacement:

 import qualified Propellor.Property.Hardware.SheevaPlug as SheevaPlug

    laptop = host ""
        & SheevaPlug.diskImage "/srv/images/leech.img" (MegaBytes 2000)
            (& propertyList "has all of leech's properties"
                (hostProperties leech))

This also means you can start with a manually built system, write down the properties it has, and iteratively run Propellor against it until you think you have a full specification of it, and then use that to generate a new, clean disk image. Nice way to transition from sysadmin days of yore to a clean declaratively specified system.

Syndicated 2015-10-23 02:09:17 from see shy jo

propellor orchestration

With the disclamer that I don't really know much about orchestration, I have added support for something resembling it to Propellor.

Until now, when using propellor to manage a bunch of hosts, you updated them one at a time by running propellor --spin $somehost, or maybe you set up a central git repository, and a cron job to run propellor on each host, pulling changes from git.

I like both of these ways to use propellor, but they only go so far...

  • Perhaps you have a lot of hosts, and would like to run propellor on them all concurrently.

    master = host "" & concurrently conducts alotofhosts

  • Perhaps you want to run propellor on your dns server last, so when you add a new webserver host, it gets set up and working before the dns is updated to point to it.

    master = host "" & conducts webservers before conducts dnsserver

  • Perhaps you have something more complex, with multiple subnets that propellor can run in concurrently, finishing up by updating that dnsserver.

    master = host "" & concurrently conducts [sub1, sub2] before conducts dnsserver

    sub1 = "" & concurrently conducts webservers & conducts loadbalancers

    sub2 = "" & conducts dockerservers

  • Perhaps you need to first run some command that creates a VPS host, and then want to run propellor on that host to set it up.

    vpscreate h = cmdProperty "vpscreate" [hostName h] before conducts h

All those scenarios are supported by propellor now!

Well, I haven't actually implemented concurrently yet, but the point is that the conducts property can be used with any of propellor's property combinators, like before etc, to express all kinds of scenarios.

The conducts property works in combination with an orchestrate function to set up all the necessary stuff to let one host ssh into another and run propellor there.

  main = defaultMain (orchestrate hosts)

hosts = 
    [ master
    , webservers 
    , ...

The orchestrate function does a bunch of stuff:

  • Builds up a graph of what conducts what.
  • Removes any cycles that might have snuck in by accident, before they cause foot shooting.
  • Arranges for the ssh keys to be accepted as necessary.
    Note that you you need to add ssh key properties to all relevant hosts so it knows what keys to trust.
  • Arranges for the private data of a host to be provided to the hosts that conduct it, so they can pass it along.

I've very pleased that I was able to add the Propellor.Property.Conductor module implementing this with only a tiny change to the rest of propellor. Almost everything needed to implement it was there in propellor's infrastructure already.

Also kind of cool that it only needed 13 lines of imperative code, the other several hundred lines of the implementation being all pure code.

Syndicated 2015-10-22 01:02:58 from see shy jo

585 older entries...


joey certified others as follows:

  • joey certified joey as Journeyer
  • joey certified davidw as Journeyer
  • joey certified bombadil as Journeyer
  • joey certified dhd as Journeyer
  • joey certified ajt as Journeyer
  • joey certified chrisd as Journeyer
  • joey certified scandal as Journeyer
  • joey certified lewing as Journeyer
  • joey certified jwz as Master
  • joey certified graydon as Journeyer
  • joey certified cas as Journeyer
  • joey certified garrett as Journeyer
  • joey certified lupus as Journeyer
  • joey certified octobrx as Journeyer
  • joey certified pudge as Journeyer
  • joey certified marcel as Journeyer
  • joey certified ljlane as Journeyer
  • joey certified uzi as Journeyer
  • joey certified quinlan as Journeyer
  • joey certified bribass as Journeyer
  • joey certified jonas as Journeyer
  • joey certified dsifry as Journeyer
  • joey certified plundis as Journeyer
  • joey certified deirdre as Journeyer
  • joey certified crackmonkey as Journeyer
  • joey certified jim as Journeyer
  • joey certified vincent as Journeyer
  • joey certified apenwarr as Journeyer
  • joey certified schoen as Journeyer
  • joey certified CentralScrutinizer as Apprentice
  • joey certified wichert as Master
  • joey certified doogie as Journeyer
  • joey certified espy as Journeyer
  • joey certified omnic as Journeyer
  • joey certified hands as Journeyer
  • joey certified stig as Journeyer
  • joey certified nick as Journeyer
  • joey certified tausq as Journeyer
  • joey certified broonie as Journeyer
  • joey certified dunham as Journeyer
  • joey certified austin as Journeyer
  • joey certified lordsutch as Journeyer
  • joey certified Gimptek as Apprentice
  • joey certified jimd as Journeyer
  • joey certified chip as Master
  • joey certified jgg as Master
  • joey certified branden as Journeyer
  • joey certified z as Journeyer
  • joey certified srivasta as Journeyer
  • joey certified danpat as Journeyer
  • joey certified lilo as Journeyer
  • joey certified seeS as Journeyer
  • joey certified netgod as Journeyer
  • joey certified dres as Journeyer
  • joey certified cech as Journeyer
  • joey certified knghtbrd as Journeyer
  • joey certified calc as Journeyer
  • joey certified ruud as Journeyer
  • joey certified edlang as Journeyer
  • joey certified gorgo as Journeyer
  • joey certified jwalther as Journeyer
  • joey certified bma as Journeyer
  • joey certified claw as Apprentice
  • joey certified hp as Journeyer
  • joey certified esr as Master
  • joey certified tobi as Journeyer
  • joey certified ajk as Journeyer
  • joey certified Joy as Journeyer
  • joey certified ejb as Journeyer
  • joey certified corbet as Journeyer
  • joey certified rcw as Journeyer
  • joey certified woot as Journeyer
  • joey certified bcollins as Journeyer
  • joey certified neuro as Journeyer
  • joey certified biffhero as Journeyer
  • joey certified Trakker as Journeyer
  • joey certified bdale as Journeyer
  • joey certified foka as Journeyer
  • joey certified davem as Master
  • joey certified logic as Journeyer
  • joey certified mstone as Journeyer
  • joey certified drow as Journeyer
  • joey certified clameter as Journeyer
  • joey certified mdorman as Journeyer
  • joey certified bwoodard as Journeyer
  • joey certified JHM as Journeyer
  • joey certified lalo as Journeyer
  • joey certified edb as Journeyer
  • joey certified shaleh as Journeyer
  • joey certified x as Apprentice
  • joey certified stephenc as Journeyer
  • joey certified bodo as Journeyer
  • joey certified jpick as Journeyer
  • joey certified ncm as Journeyer
  • joey certified gord as Journeyer
  • joey certified mpav as Journeyer
  • joey certified lazarus as Apprentice
  • joey certified starshine as Journeyer
  • joey certified che as Journeyer
  • joey certified brother as Journeyer
  • joey certified joeysmith as Journeyer
  • joey certified bod as Journeyer
  • joey certified decklin as Journeyer
  • joey certified gibreel as Journeyer
  • joey certified torsten as Journeyer
  • joey certified alfie as Apprentice
  • joey certified aclark as Journeyer
  • joey certified kju as Journeyer
  • joey certified psg as Journeyer
  • joey certified zed as Journeyer
  • joey certified evo as Journeyer
  • joey certified mbaker as Journeyer
  • joey certified cmr as Journeyer
  • joey certified Tv as Journeyer
  • joey certified xtifr as Journeyer
  • joey certified sstrickl as Journeyer
  • joey certified etbe as Journeyer

Others have certified joey as follows:

  • joey certified joey as Journeyer
  • dhd certified joey as Journeyer
  • ajt certified joey as Master
  • davidw certified joey as Journeyer
  • alan certified joey as Journeyer
  • uzi certified joey as Journeyer
  • caolan certified joey as Journeyer
  • tron certified joey as Master
  • bombadil certified joey as Journeyer
  • cas certified joey as Journeyer
  • garrett certified joey as Master
  • lupus certified joey as Journeyer
  • graydon certified joey as Journeyer
  • marcel certified joey as Journeyer
  • mblevin certified joey as Journeyer
  • bribass certified joey as Master
  • plundis certified joey as Journeyer
  • matias certified joey as Journeyer
  • ajv certified joey as Journeyer
  • crackmonkey certified joey as Master
  • jim certified joey as Master
  • CentralScrutinizer certified joey as Master
  • schoen certified joey as Master
  • pedro certified joey as Master
  • omnic certified joey as Master
  • hands certified joey as Master
  • tausq certified joey as Journeyer
  • suzi certified joey as Master
  • broonie certified joey as Master
  • nick certified joey as Journeyer
  • lordsutch certified joey as Master
  • jimd certified joey as Master
  • chip certified joey as Master
  • jgg certified joey as Master
  • branden certified joey as Master
  • srivasta certified joey as Master
  • danpat certified joey as Master
  • darkewolf certified joey as Master
  • z certified joey as Journeyer
  • cech certified joey as Master
  • dres certified joey as Master
  • gorgo certified joey as Master
  • ruud certified joey as Master
  • kaig certified joey as Master
  • wichert certified joey as Master
  • ajk certified joey as Master
  • ljlane certified joey as Master
  • Joy certified joey as Journeyer
  • andrei certified joey as Master
  • rcw certified joey as Master
  • Trakker certified joey as Master
  • neuro certified joey as Master
  • starshine certified joey as Master
  • seeS certified joey as Master
  • foka certified joey as Master
  • pretzelgod certified joey as Master
  • mstone certified joey as Master
  • bcollins certified joey as Master
  • doviende certified joey as Master
  • dmarti certified joey as Master
  • splork certified joey as Master
  • bdale certified joey as Master
  • drow certified joey as Master
  • edward certified joey as Master
  • ljb certified joey as Journeyer
  • claw certified joey as Master
  • edb certified joey as Master
  • shaleh certified joey as Master
  • jpick certified joey as Master
  • zacs certified joey as Journeyer
  • jae certified joey as Master
  • benson certified joey as Journeyer
  • wardv certified joey as Master
  • jeroen certified joey as Master
  • lazarus certified joey as Journeyer
  • mpav certified joey as Master
  • walken certified joey as Master
  • ncm certified joey as Master
  • Barbwired certified joey as Master
  • kraai certified joey as Master
  • che certified joey as Master
  • lstep certified joey as Master
  • brother certified joey as Master
  • nas certified joey as Journeyer
  • acme certified joey as Master
  • moshez certified joey as Master
  • tca certified joey as Journeyer
  • cord certified joey as Master
  • sethcohn certified joey as Master
  • bod certified joey as Journeyer
  • tripix certified joey as Journeyer
  • jLoki certified joey as Master
  • sh certified joey as Master
  • lerdsuwa certified joey as Master
  • torsten certified joey as Master
  • alfie certified joey as Master
  • mhatta certified joey as Master
  • aclark certified joey as Master
  • kju certified joey as Master
  • psg certified joey as Master
  • zed certified joey as Master
  • karlheg certified joey as Master
  • evo certified joey as Master
  • ole certified joey as Master
  • jfs certified joey as Master
  • bma certified joey as Master
  • jtc certified joey as Master
  • gibreel certified joey as Master
  • Jordi certified joey as Master
  • jhasler certified joey as Master
  • cpbs certified joey as Journeyer
  • ths certified joey as Master
  • decklin certified joey as Master
  • Tv certified joey as Master
  • xtifr certified joey as Master
  • joeysmith certified joey as Master
  • mishan certified joey as Master
  • keverets certified joey as Master
  • pa certified joey as Master
  • Slimer certified joey as Master
  • weasel certified joey as Master
  • technik certified joey as Master
  • baretta certified joey as Master
  • robster certified joey as Master
  • juhtolv certified joey as Master
  • rcyeske certified joey as Master
  • kmself certified joey as Master
  • andersee certified joey as Master
  • asuffield certified joey as Master
  • charon certified joey as Master
  • claviola certified joey as Master
  • chrisd certified joey as Master
  • mdz certified joey as Master
  • buckley certified joey as Master
  • moray certified joey as Master
  • jtjm certified joey as Master
  • mwk certified joey as Master
  • proski certified joey as Master
  • cmiller certified joey as Master
  • pau certified joey as Master
  • rkrishnan certified joey as Master
  • dieman certified joey as Master
  • eckes certified joey as Master
  • fxn certified joey as Master
  • etbe certified joey as Master
  • Sam certified joey as Master
  • fallenlord certified joey as Master
  • hanna certified joey as Master
  • maxx certified joey as Master
  • dopey certified joey as Master
  • tfheen certified joey as Master
  • ttroxell certified joey as Master
  • Netsnipe certified joey as Master
  • quarl certified joey as Journeyer
  • amck certified joey as Master
  • riverwind certified joey as Master
  • pere certified joey as Journeyer
  • NoWhereMan certified joey as Master
  • jochen certified joey as Master
  • faw certified joey as Master
  • mako certified joey as Master
  • Pizza certified joey as Master
  • sysdebug certified joey as Master
  • vern certified joey as Master
  • ctrlsoft certified joey as Master
  • lkcl certified joey as Master
  • hasienda certified joey as Master
  • gesslein certified joey as Master
  • ean certified joey as Master
  • dangermaus certified joey as Master

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Share this page