idcmp is currently certified at Journeyer level.

Name: JAmes Atwill
Member since: N/A
Last Login: 2008-09-25 19:46:06

FOAF RDF Share This

Homepage: http://idcmp.linuxstuff.org/

Notes:

Hi There.
Ages ago, before modern search engines, when you searched for IDCMP you got the original meaning of the acronym, which stemmed from the good ole days of the Amiga.

These days, when you search for IDCMP, you find this page; and me. (Sometimes you'll find non-me related things too).
You can find my blog elsewhere. This is just a syndication.

Articles Posted by idcmp

Recent blog entries by idcmp

Syndication: RSS 2.0
28 Aug 2014 (updated 29 Aug 2014 at 18:09 UTC) »

Backing up OS X onto Linux. Finally.

I've tried all sorts of black voodoo magic to make this work, but finally I have something repeatable and reliable. Also, please excuse the horrible formatting of this post.

History

I started with Time Machine talking to my Linux box with netatalk. This worked fine until one day Time Machine told me it needed to back up everything all over again.

Then it started to tell me this almost weekly.  Apparently when this happens, the existing backup is corrupt and all that precious data is at worst irretrievable or at best tedious to retrieve. These are not attributes I want associated with my backup solution.

Then I did an rsync over ssh to my Linux box.  This is fine exception that it lacks all the special permissions, resource forks, acls, etc, etc that are hidden in a Mac filesystem.

Then I tried SuperDuper! backing up to a directory served up via netatalk mounted via afp:// on OS X.  This worked, but was mind numbingly slow. Also, it would mean I'd have to pay for a tool if I wanted to do incremental backups. This gets expensive as I also back up a few friends OS X laptops on my Linux file server.

I tried SuperDuper! backing up over Samba, but hdiutil create apparently doesn't work over Samba. Workarounds all needed the purchased version of SuperDuper!.

There's *another* work around for SuperDuper! where I can use MacFUSE and sshfs, but the MacFUSE author has abandoned the project and recommends people not to use it.

Sheesh.

The Solution

Ultimately, the goal is to make a sparsebundle HFS+ disk image, put it on a Samba mounted share and rsync my data over to it. You'd be surprised how many niggly bits there are for this.

Install Rsync


First, I grabbed the 3.1.x version of rsync from Homebrew - install Homebrew as per the directions there, then run:

brew install https://raw.github.com/Homebrew/homebrew-dupes/master/rsync.rb

If you've been digging through voodoo magic, then you'll be happy to hear this version of rsync has the all the rsync patches you'll read about (like --protect-decmpfs).

Samba

Nobody needs another out of date blog entry explaining how to setup Samba. Follow some other guide, make sure Samba starts automatically and use smbpasswd to create an account.  

I recommend using the name of the machine being backed up as the account name. I'm calling that machinename for the rest of this post.

Make sure you can mount this share on OS X via smb:// ( Finder > Go > Connect to Server... ).  Make sure you can 1) create a file, 2) edit and save the file, 3) delete the file.  I'm going to assume you've mounted this share at /Volumes/Machinename

Backup Into

Now lets make something for us to backup into.  Figure out how big the disk is on the source machine (we'll assume 100g) then run:

hdiutil create /tmp/backup.sparsebundle -size 100g -type SPARSEBUNDLE -nospotlight -volname 'Machinename-Backup' -verbose -fs 'Case-sensitive Journaled HFS+'

Yes, you're creating it in /tmp, this is to work around hdiutil create not liking Samba.

Next you'll want to copy this sparse bundle onto your Samba share:

 cp -rvp /tmp/backup.sparsebundle /Volumes/machinename


This will copy a bunch of files and should be successful without any warnings.  Now lets mount this sparse bundle:

hdiutil attach /Volumes/machinename/backup.sparsebundle -mount required -verbose

You should now have /Volumes/Machinename-Backup mounted on your system. Fun story, OS X recognizes that this disk image is hosted off the machine, so it mounts this disk image with "noowners" (see mount man page). That's going to be a problem for our backup, so we need to tell OS X it's okay to use userids normally:

sudo diskutil enableOwnership /Volumes/Machinename-Backup

Preparing Rsync

There are a handful of files which recommend to be excluded:


.DocumentRevisions-*/
.Spotlight-*/
/.fseventsd
/.hotfiles.btree
.Trashes
/afs/*
/automount/*
/cores/*
/private/var/db/dyld/dyld_*
/System/Library/Caches/com.apple.bootstamps/*
/System/Library/Caches/com.apple.corestorage/*
/System/Library/Caches/com.apple.kext.caches/*
/dev/*
/automount
/.vol/*
/net
/private/tmp/*
/private/var/run/*
/private/var/spool/postfix/*
/private/var/vm/*
/private/var/folders/*
/Previous Systems.localized
/tmp/*
/Volumes/*
*/.Trash
/Backups.backupdb
/.MobileBackups
/.bzvol
/PGPWDE01
/PGPWDE02

Store this in a file somewhere. I stored mine as exclude-from.txt in /Volumes/Machinename

Okay, now we're ready to run rsync.  I think the correct arguments to rsync are: -aNHAXx --nfs-compression --protect-decmpfs --itemize-changes --super --fileflags --force-change --crtimes

So, we run:

rsync all-those-args-above --exclude-from=/Volumes/Machinename/exclude-from.txt / /Volumes/Machinename-Backup/

When The Backup Is Done

This will take a little while. When it's done, you can then bless your backup so it could be booted:

sudo bless -folder /Volumes/Machinename-Backup/System/Library/CoreServices

Then you can umount your backup:

hdiutil detach /Volumes/Machinename-Backup

Periodically, and after your first run, you should compact down your sparsebundle disk image:

hdiutil compact /Volumes/Machinename/backup.sparsebundle -batteryallowed

You can now log into your Linux server and tar up the backup (apparently XZ > bzip2 for compression size).

 tar Jcvf machnename-backup.tar.xz backup.sparsebundle

Depending on the size of that tarball, you could upload it to Google Drive, Drop Box, etc. Before you do, you'll probably want to encrypt it. I used OpenSSL:

openssl aes-256-cbc -a -salt -in machinename-backup.tar.xz -out machinename-backup.tar.xz.aes-256-cbc

Many of these steps will take *hours* if you have a lot of data, so you may consider just backing up parts of your system more frequently and doing your whole system once every-so-often.

Why This Is Nice

One thing I really love about this setup is that each piece of the puzzle does one thing and does it well. If Samba is too slow, I could go back to netatalk. The sparsebundle disk image hosts HFS+ properly with all its OS X-specific voodoo and rsync's job is to copy files. If there's a better file copier, I could drop that in.

Conclusion

I left a lot out. I know. I'm kind of expecting you to have a rough idea of how to get around OS X and Linux, figure out how to put most of the above in a shell script, decide when to do backups, how to store those tarballs, etc. Hopefully though this will help someone that just needs some of the key ingredients to make it work.  



Syndicated 2014-08-27 23:49:00 (Updated 2014-08-29 17:52:35) from Idcmp

Navigation 2.0: The Missing Features

I recently found myself driving a car in an unfamiliar city relying heavily on Google Navigation on my phone to get me from point A to point B (and back to point A again). From this two week experience I found some things in the Navigation experience I'd really love to have. Maybe other Navigation apps have this and I've just been googleblind.

Not Looking At The Phone

Most of the time I'm driving, I'm relying on Navigation to tell me what's going on. You're probably the same. Some simple options would make me a lot more comfortable. These also apply to when I'm on a motorbike and my phone is locked away in a pocket.
  1. Don't tell me compass directions: If I'm in a new city, in a new area telling me "Head East" is useless. Especially at night where I can't use the Sun to figure out which way East is.
  2. Give me a hint which lane I should eventually be in, "Travel three kilometers, eventually turning left."
  3. If I'm driving for thirty minutes on a highway, periodically remind me that you haven't crashed or suddenly run out of battery power. Even a little beep every 10 minutes (or % of a leg) would be nice. Maybe even tell me what % of battery is left.
  4. Along those lines, since Navigation knows how fast I'm going and how far I have to go, telling me "Continue for about 15 minutes." is more handy than "continue for 23 kilometers".
  5. On navigation start, tell me something useful; even "Route calculated" or "Estimated twenty minutes of travel time."
  6. If I'll have less than 10% battery life by the calculated arrival time, tell me this up front and give me the option to just read the list without actual navigation.
  7. It may annoy you, but I like to know when Navigation has changed its mind about where we're going. Telling me that a route has been recalculated is re-assuring.

Tell Me Less

There are some things where being quieter makes more sense:
  1. Cut me some slack in a parking lot. Unless you're going to tell me exactly how to get out of the parking lot, keep quiet until I'm close to an exit. Calculate some routes of the closest few exits and get ready to tell me "Turn left here." or "Turn right here.". Telling me to "Head North to 11th Ave South West going to 23rd Street East going East." is just distracting while I'm trying to navigate the maze of grocery carts and SUVs.
  2. Some cities have Main St. North, West, East, South or North East / North West / etc.  If I'm on "Main St North West" turning onto "1st Street North West", don't say "Northwest" both times.  In fact, unless I'm close to a boundary where it's important to know which cardinal direction of the street I'm on, simply saying "Turn left onto First Street." is fine.

Nicer Touches

And there are a few nice things I bet wouldn't be too hard.
  1. Traffic Lights. I'm sure most cities have a database of which intersections have traffic lights. Telling me "At the lights, turn left onto Third street." trumps "In six hundred and fifty meters turn left onto Third street.".
  2. Let me mark certain areas as "well known" areas and silence turn by turn directions if my destination is in that area unless I ask for them to be re-enabled.
  3. All major highways in Canada have numbered exits. They're numbered based on kilometer distance. If I'm at exit 35, and I'm going to exit 25, I know there's ten kilometers to go. When I get on a highway, let me know the exit number I'm going to be exiting on.
  4. If I'm going to a destination that has a Google Place (or whatever they're called now), and it seems that they'll be closed or closing within an hour of the calculated arrival, give me an option to call the place.
  5. Let me tell Navigation how comfortable I am with a given city or area. If I don't know it very well, give me notifications earlier and take me on possibly longer but easier to navigate routes.


Syndicated 2012-05-24 23:28:00 (Updated 2012-05-24 23:30:54) from Idcmp

New Plugin: licensing-maven-plugin

Ever wanted to know what licenses your dependencies (and their dependencies) are using? Maybe you work for a company that wants to sell their source code so you're wanting to avoid the GPL (and AGPL)? I've got the plugin for you!

I mentioned a while back that the build process at my day job had been declared bankrupt. Well, it's doing much better now; what used to be a multi-day process where you were never quite sure if it was working 100% now takes less than an hour (including data population). We're quite happy about that part.

Along the way we started looking at some of the other bits that fit more into release management; one of them was a "licensing report". This report listed most of our dependencies and which open source license they were in. Instead of hacking at the old scripts, we decided to let Maven take over and handle providing licensing and dependency information.

So with a rough idea of what we wanted to do, I put together the Licensing Maven Plugin. It has a few handy features:

  1. Transitively aggregate licensing information of dependencies of child modules in multi-module reactors (or in English, it works the way you would expect it to on multi-module builds).
  2. Coalesce license names (so "Apache License, Version 2.0", "Apache 2.0" and "ASLv2.0" can all be reported as "The Apache Software License, Version 2.0").
  3. Fail builds if a dependency is only available under a disliked license.
  4. Exempt artifacts under disliked licenses from failing the build.
  5. Manually declare licenses for dependencies that fail to provide their own.


It's hosted on central, so give it a whirl:

mvn org.linuxstuff.maven:licensing-maven-plugin:1.5:aggregate

You'll see a truckload of warnings go by, and when it stops, you'll have a target/aggregated-third-party-licensing.xml file (yes, I know it's not nicely formatted yet).

If you'd like some more details, checkout the licensing-maven-plugin README.


Syndicated 2012-01-09 07:54:00 (Updated 2012-01-09 08:41:16) from Idcmp

New Plugin: licensing-maven-plugin

Ever wanted to know what licenses your dependencies (and their dependencies) are using? Maybe you work for a company that wants to sell their source code so you're wanting to avoid the GPL (and AGPL)? I've got the plugin for you!

I mentioned a while back that the build process at my day job had been declared bankrupt. Well, it's doing much better now; what used to be a multi-day process where you were never quite sure if it was working 100% now takes less than an hour (including data population). We're quite happy about that part.

Along the way we started looking at some of the other bits that fit more into release management; one of them was a "licensing report". This report listed most of our dependencies and which open source license they were in. Instead of hacking at the old scripts, we decided to let Maven take over and handle providing licensing and dependency information.

So with a rough idea of what we wanted to do, I put together the Licensing Maven Plugin. It has a few handy features:

  1. Transitively aggregate licensing information of dependencies of child modules in multi-module reactors (or in English, it works the way you would expect it to on multi-module builds).
  2. Coalesce license names (so "Apache License, Version 2.0", "Apache 2.0" and "ASLv2.0" can all be reported as "The Apache Software License, Version 2.0").
  3. Fail builds if a dependency is only available under a disliked license.
  4. Exempt artifacts under disliked licenses from failing the build.
  5. Manually declare licenses for dependencies that fail to provide their own.


It's hosted on central, so give it a whirl:

## NOTE: If you're reading this early Saturday morning (07-Jan-2012), central may not have synced 1.5 yet.
mvn org.linuxstuff.maven:licensing-maven-plugin:1.5:aggregate


You'll see a truckload of warnings go by, and when it stops, you'll have a target/aggregated-third-party-licensing.xml file (yes, I know it's not nicely formatted yet).

If you'd like some more details, checkout the licensing-maven-plugin README.


Syndicated 2012-01-08 07:54:00 (Updated 2012-01-08 07:56:46) from Idcmp

14 Dec 2011 (updated 17 Dec 2011 at 07:28 UTC) »

Hello Developers, look at your code.

Hello developers, look at your code, now back to me, now back at your code, now back to me. Sadly, your code isn't me, but if it stopped collecting tech debt and you started making time for technical quality it could smell like me. Page down, back up. Where are you? You're at an interface with the code that your code could smell like. What's in your hand, back at me. I have it, it's an oyster with two different implementations to that thing you have an API for. Look again, the implementation is now secure and scalable. Anything is possible when your code has no smells. I'm on a horse.


Syndicated 2011-12-14 09:03:00 (Updated 2011-12-17 07:26:00) from Idcmp

72 older entries...

 

idcmp certified others as follows:

  • idcmp certified miguel as Master
  • idcmp certified idcmp as Journeyer
  • idcmp certified tigert as Journeyer
  • idcmp certified raph as Master
  • idcmp certified rasmus as Master
  • idcmp certified zeev as Master
  • idcmp certified hp as Journeyer
  • idcmp certified eskil as Journeyer
  • idcmp certified Yashy as Apprentice
  • idcmp certified ajh as Journeyer
  • idcmp certified stevex as Apprentice
  • idcmp certified pjones as Journeyer
  • idcmp certified julian as Journeyer

Others have certified idcmp as follows:

  • idcmp certified idcmp as Journeyer
  • rasmus certified idcmp as Apprentice
  • Damian certified idcmp as Apprentice
  • raph certified idcmp as Apprentice
  • stevex certified idcmp as Apprentice
  • beppu certified idcmp as Apprentice
  • Iain certified idcmp as Apprentice
  • duncanm certified idcmp as Journeyer
  • scottj certified idcmp as Journeyer
  • pjones certified idcmp as Apprentice
  • dej certified idcmp as Journeyer
  • nixnut certified idcmp as Journeyer
  • realblades certified idcmp as Journeyer
  • nourantofort certified idcmp as Master
  • gmoore certified idcmp as Journeyer

[ Certification disabled because you're not logged in. ]

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page