Older blog entries for joey (starting at number 505)

git push over XMPP

Imagine if you could send git pushes to any of your friends on Google Talk or other Jabber (XMPP) servers. Even though you're in different places and your computers probably cannot talk to one-another directly, you can share a git repository, without relying on a git hosting provider such as GitHub.

  To xmpp::joey@kitenet.net
  * [new branch]      master -> xmpp/master

That's what I've built over the past week. So far, it's only supported by the git-annex assistant, but it would not be very hard to make this into a separate git-remote-xmpp program, and a push listening daemon, that could be used with non-git-annex repisitories too.

You can read about the details in my git-annex assistant dev blog:

  1. day 124 git push over xmpp groundwork
  2. day 125 xmpp push continues
  3. day 126 mr watson come here

And yes, I update that blog every single work day. If you're not getting enough blog updates from me, and are not scared by rather technical posts, you might want to subscribe to it. :)

Syndicated 2012-11-09 22:42:19 from see shy jo

UPS farce

This is not the kind of place I live, though I don't mind a visit. Errol looking out over Salsalito and the Bay I'm not bothered by a more rural life, I thrive on peace and quiet, and am not bothered by needing to haul my water, split firewood, or spend an hour just mowing my driveway.

I bought a GPS the other day, and it doesn't know where my driveway is..
or where the road leading to my driveway is..
or even about the highway leading to that road!


not many roads on this map..

The only thing that really bothers me about living here off the map is that I'm in the middle of a food desert too. The nearest semi-decent grocery store and produce market are 40 minutes away. There's one restaurant worth eating at, and while a very congenial place to eat rustic food while watching the river flow by, it serves almost no vegetables. Adding to the problem, my fridge is a rather expensive propane model, which is about half a normal size, and has no freezer. Managing a healthy meal plan is a constant struggle, though I do have time to cook, at least, thanks to not having the kind of commute that is typical around here.

Well, the food desert was my only real problem. Now I can add to that the UPS emergency. A week ago, a package left Shenzen China for here. It quickly passed through Anchorage Alaska, and with a few days had arrived an hour and a half from here. Getting it here has taken the whole rest of the week.


I only knew UPS attempted delivery because a neighbor told me. Seems they took one look at my driveway and turned around. They never tried again, and they didn't bother to leave a note on the mailbox, or anything. Now, the US postal service is happy to drive up this driveway; they drive a sensible 4x4 around here. I think DHL and Fedex have been here too. (Contrary to John Goerzen's experiences) My compact car is not bothered by my driveway either...

But UPS's brand involves large brown trucks, even where inappropriate, and damage to them from such things as low-hanging branches apparently comes out of the driver's paycheck. And while normally UPS leaves small things in my mailbox, or with a neighbor, this was an important package from China, and it needed a signature. But really, a post card? "emergency"?!

So, I bought the GPS to find UPS, because my package was in an industrial park in some town I've never been to. And on Tuesday I was within an hour of there and swung by. And it was closed. UPS's website had not mentioned this; other businesses are open the day after Memorial day. That turned out to be a really annoying day all around. When I got home, after a night away, I found my fridge's fire had blown out, and nearly all my food was only fit for the dog.

Today I went out to try again (and buy groceries). When I reached the UPS depot.. it was closed! Until 3 pm. At this point I felt this story was reaching full-on farce.

After cooling my heels in a library for three hours, I was finally able to open the most over-packed package I've probably ever seen.

My kickstarter reward sample looks good.. I'll be ordering a lot more, but not to this address.

As a coda to the UPS farce, arriving home today I checked my mailbox, and found the UPS postcard had finally arrived. It included mention of the 3 pm opening time, and noted that I had until tomorrow before they'd return my package to China.

The coda to the coda is that on my way home from this second annoying day delivered courtesy of UPS, I stumbled on a restaurant overlooking a waterfall, just outside the town where I get my groceries. It serves food that satisfies both my Cajun and healthy cravings. Now I just need a larger fridge..

Syndicated 2012-09-06 23:56:11 from see shy jo

California postcard 2012

San Francisco skyline, Errol & Dani, Dani on cliffs at Mendocino, Sango on Hendy woods redwood

Syndicated 2012-08-16 16:25:25 from see shy jo

build a dynamic webapp in Yesod in just three days

This screenscast shows what I built. Scroll down for my Yesod braindump.

I've been astonished how quickly this went together. This is my first time using any sort of web framework, and I used a still unusual one, Yesod. It's my first time using Bootstrap too. It's also the first time I've done any AJAX programming!

Bootstrap was something I'd heard of and seen some suspiciously similar looking sites built with, but it was really a pleasant surprise. Being able to make things that work and look good on the web without struggling with CSS is such a nice change. For the first time, it makes the web feel like a UI toolkit to me.

As I've learned Yesod, I kept seeing problems the web framework solves that I either solved in an ad-hoc basis in Ikiwiki, or didn't realize were problems. In the former group are things like having a consistent way to store a URL to redirect to after an action (such as logging in) finishes, and generating urls programmatically in ways that avoid broken likes. In the latter group are things like exploiting types to guarantee mistakes can't be made. And also the abstraction of widgets, which combine html, css, and javascript, in a way that can be composed with other widgets.

Overall, I'm really enjoying Yesod, and it's making me productive in new ways.

I also see a lot of potential in Yesod to improve from where it is.

  • Probably the biggest improvement would be generating javascript from haskell code, so the type safety extends to the javascript side.

    There have been many tries at doing this, the first one I'm really tempted to use is Fay. Rather than build a heavy haskell runtime on top of javascript, it essentially uses javascript as its runtime, and offloads type checking to GHC. This lets it generate javascript code that is short and often comprehensible enough to give insights into what the original haskell code does.

    I'm betting this will be integrated into Yesod eventually. They have an active wiki page about it.

  • There's a WAI library for building local webapps with Yesod, but it was not suitable for my needs (for one thing, it lacks security; for another it kills the haskell program when the web page is closed); so I built my own webapp library. A problem with my current pace of development is that I'm building lots of reusable libraries, but I don't have the time to stabalize them and make them generally available. That one goes in the pile of 2k+ lines of such code.

  • Yesod needs a version of the Hamlet markup that can be edited by people who only understand html. That means it should allow closing tags, and tabs, and not have nasty whitespace gotchas. I think this would be easy to build, starting from Hamlet. It could be called "Hecate".. I don't have time right now.

  • The compile time error messages are often beyond atrocious. Seriously, I'm tempted to write a filter to filter out common patterns where there's one line about a syntax error in a Hamlet file sandwitched in between 150 lines of type error gobbly-gook and generated code.

  • Some really nice things could be done integrating Yesod with Bootstrap. Like the ability to build entire sites by just composing together Bootstrap components, with no need to write a single line of html or css. I'm very tempted to try to build this library.

    webpage = bootstrap Dynamic $ do setTitle "FooCorp" login <- getLogin navbar [FixedTop] $ do brand "FooCorp" link AboutR link BlogR nav [PullRight] $ link . maybe LoginR ProfileR login div [ContainerFluid] $ content login where content Nothing = heroUnit $ do para $ "We're the FooCorp for you." button "Register Today" [Primary, Large] SignUpR carousel [ amazingFeatures , aboutFooCorp , pricing ] content (Just user) = do para $ "Welcome back " ++ name user ++ "!" showProfile user

Syndicated 2012-07-29 20:00:27 from see shy jo

ghc threaded runtime gotchas

That's a seemingly simple question I started asking various people two weeks ago. I didn't get many useful answers, but now I have experience doing it myself, and so here's a blog post brain dump.

I have been trying to convert git-annex to use GHC's threaded runtime, for a variety of reasons. Naively adding the -threaded option resulted in a git-annex command that seemed to randomly freeze, but only sometimes (and, infuriatingly, never when I straced it), and a test suite that froze at a random point almost every time I ran it. Not a good result, and lacking any knowledge about gotchas with using the threaded runtime, I was at a loss for a long time (most of DebConf) about how to fix it.

I now know of at least three classes of problems that enabling the threaded runtime can turn up in programs that have been developed using the non-threaded runtime.

accessing a MVar after forkprocess can hang

MissingH has some code similar to this, which works ok with the non-threaded runtime:

  forkProcess $ do
    debugM "about to run some command"
    executeFile ...

In the above example, debugM accesses a MVar. Doing that after forkProcess can result in a MVar deadlock, as it tries to access a MVar value, that is, apparently, not accessible to the forked process. (Bug report with test case)

So, using System.Cmd.Utils from MissingH is asking for trouble. I switched all my code to the newer and, apparently, threaded runtime safe System.Process.

forkProcess is a massively bad idea

Even when not accessing a MVar after forkProcess, it's very unsafe to use. It's easy to miss the warning attached to forkProcess, when the code seems to work. But with the threaded runtime, I've found that most any call to forkProcess will sometimes succeed, and sometimes freeze the whole program. This might only happen around 1 time in 1000. Then you'll find this warning and do a lot of head-scratching about what it really means:

forkProcess comes with a giant warning: since any other running threads are not copied into the child process, it's easy to go wrong: e.g. by accessing some shared resource that was held by another thread in the parent.

The hangs I saw could be due to laziness issues deferring code to run after the forkProcess that you'd expect to have run before it ... or who knows what else.

It's not clear to me that it's possible to use forkProcess safely in Haskell code. I think it's notable that System.Process runs the whole fork/exec in C code instead.

unsafe FFI calls block

According to most of the documentation you'll find in eg, the Haskell wiki, Real World Haskell, etc, the only difference between the safe and unsafe imports in the FFI is that unsafe is faster, and shouldn't be used for C code that calls back into Haskell code.

But the documentation is out of date. Actually, if you're using the FFI, and the foreign function can block, you need to use safe. When using unsafe, a blocking foreign function can block all threads of the program.

In my case, I was using kqueue to wait for changes to files, and this indeed blocked my whole program when linked with -threaded. Marking it safe fixed this.

The details are well described in this paper: http://community.haskell.org/~simonmar/papers/conc-ffi.pdf

Somewhat surprisingly, this blocking only happens when using the threaded runtime. If you're using the non-threaded runtime with unsafe blocking FFI functions, your other pseudo-threads won't be blocked. This is because the non-threaded runtime has an SIGALARM timer that interrupts (most) blocking system calls. This leads to other troubles of its own (like needing to restart interrupted FFI functions, or blocking the other pseudo-threads from running if the C code ignores SIGALARM), but that's offtopic for this post.

summary

Converting a large Haskell code base from the default, non-threaded runtime to the threaded runtime can be quite tricky. None of the problems are the sort of thing that Haskell helps to manage either. I will be developing new programs using the threaded runtime from the beginning from now on.

By the way, don't take this post to say that threading in Haskell sucks. I've really been enjoying writing threaded Haskell code. The control Haskell gives over isolation of state to threads, and the excellent and wide-ranging suite of thread communications data types (MVar, Chan, QSemN, TMVar, SampleVar, etc) have made developing a complex threaded program something I feel comfortable doing for the first time, in any language.

Syndicated 2012-07-20 19:31:37 from see shy jo

I am become Joey, destroyer of drives

drive #1

To add to the fun of being in Nicaragua, my laptop's solid state drive died the second day here. Seems this is the failure mode for a SSD, get a little slow, and then a switch flips and it stops accepting any writes, at all, becoming a read-only media. Never was there an indication of a problem in SMART etc.

Did you know that Nicaragua has neither names for most roads, nor addresses? Rather than trying to shoehorn the directions to my hotel into the address fields of Amazon.com, I shipped the replacement SSD to home, and thought I'd limp along.

drive #2

I destroyed my second drive one hour after getting it working. A borrowed USB flash drive, which seems to have melted under the unusual load while I was away at lunch. Perhaps putting an ext3 filesystem on it was a very bad idea, although I have successfully run other USB flash drives that way for years. Perhaps it was a cheap drive that only pretended to hold 32 gb.

drive #3

For several days I happily used the third drive, a USB hard drive, as my temporary root filesystem. Until I destroyed it by knocking it off my bed.

drive #4, #5

Now my netbook is running from a Debian Live USB key, with a second USB key for my customisations. So far I have not managed to destroy these, but there's still a day left in my trip..

Syndicated 2012-07-13 22:43:36 from see shy jo

debian-cd work at DebCamp

I've been working the past two days on debian-cd, which was the main thing (besides git-annex assistant) I planned to work on at DebCamp.

Yesterday, Steve McIntyre and I cleaned up some cruft in debian-cd's package lists. This freed somewhere in the area of 30 mb. I also took a grep through d-i and made sure the CDs include packages that d-i can optionally install in some situations.

Today, I investigated how Recommends are ordered on the CD, and concluded it's as close to optimal as can be easily achieved. So was not able to save space there, but I did find a way to reorganize the desktop task that avoids needing to include a lot of printing stuff and some other stuff on the first CD. While that helped some, it still didn't get either Gnome or Kde to entirely fit, so getting there will probably involve rebuilding 100 or so packages with xz, if someone decides to do that.

So it's stil TBD whether Gnome or Kde will fit on a single CD in Debian wheezy. At this point I think most of us are getting tired of fighting this increasingly losing battle every release, and so other options like only having a desktop DVD are looking more appealing, as they solve the problem long-term. This would also free up the first CD for other interesting use cases, perhaps xfce, or perhaps a CD targeted at server users, and/or containing high-popularity non-desktop packages.

Syndicated 2012-07-09 02:52:58 from see shy jo

git-annex-assistant milestone

I reached a nice milestone on my git-annex assistant in my first day's work at DebCamp in Nicaragua. Here's a screencast demoing it.

git-annex-assistant.ogg (12 MB)

By the way, the weather, food, and people here are all excellent.

Syndicated 2012-07-05 22:40:10 from see shy jo

notes for a caretaker

I recently had the sort of weird experience of a recent blog post being on the top of Hacker News and there being a fair amount of interest in the details of my wacky living situation.

I could write a detailed post explaining everything, but that'd be boring .. instead I'll tease the stalkers with more oblique references to it. So here's my notes for a caretaker who will be here while I'm away at DebConf.


Please make yourself at home!

  • The back bedroom bed is made
  • That weird thing on a stand next to it that looks like a juice container with a wire coming out of it is a lamp. Plug it in to use it.
  • Check battery voltage (press the green button on the Xantrex controller) and avoid using much power if it gets below 12.0 volts. (10.0 is absolute minimum)
  • Orange inverter on stand in front bedroom has a switch to turn on, regular power cord on front porch.
  • Plug in server computer in kitchen (under my portrait), wait 10 minutes, and you can get online with wireless. (Wireless network is "hollow", and is an ad-hoc network, with Cell id AA:AA:AA:AA:AA:AA. You may have to enter this information manually.) Slow is normal.
  • Cook stove: Pull red handle behind stove to turn on gas. Light w/lighter. Turn off cook stove gas with red handle when not in use!
  • Eat any staple foods in kitchen.
  • Eat everything in fridge! (It won't keep; food lasts only half as long in it as you're used to.)
  • Check once a day (especially on windy or stormy days) that the fridge is lit, by touching the black pipe sticking above it on the back. That's its chimney. It should be warm to the touch. Temperature can be adjusted in rear also. If fridge is blown out, follow instructions printed in back to re-light.
  • Human drinking water is in large jugs in kitchen + red water cooler. Sorry it's so turbid right now, I'm using a neighbor's well.. but it's safe. I have not tested the water from the large cistern yet this year.
  • Dishwashing water: Use water from any of these (non-potable) sources:
    • rain barrels
    • outdoor shower (on way to outhouse, not at parking area; that other one is fed from a spring that has dried up)
    • large pail on coffee table
  • Please don't drink the distilled water; it's used in battery maintenance.
  • Feed dog 3/4 of metal bowl each day (on porch, next to table)
  • Cat: 1 dish full each day.
  • Consider brushing the cat, with the folding saw, on the left-side porch steps, every other day. Hairballs..
  • Chickens: 3/4 yoghurt container of feed per day (feeding station is across yard, down steps, on right; some feed is in container there)
  • Eggs can be found in nest under bush in woods. Trail starts to left of shower, goes about 10 feet in. Eat eggs, or give to dog with food.
  • Blueberries are currently available on the small hill near the greenhouse.
  • Dog and chicken feed is in front bedroom.
  • Keep solar powered fountian to the left of the sliding door full; it's used by the dog and chickens.
  • Cat has large water pail in bathroom, good for about 2 weeks.
  • Take animal water from rain barrels or from shower outside. (Can detach hose for easier access and watering.)
  • Also water lemon and lime trees, in pots, if it's dry. (Feel free to eat lemons.)
  • The house does best in hot weather if all windows are closed during the day, and opened during the night.
  • If it gets too humid inside, drain water from trays of salt underneath the pantry shelves, and run the fan.

Syndicated 2012-07-02 02:07:35 from see shy jo

496 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!