Recent blog entries

25 Oct 2014 mtearle   » (Journeyer)

That rare feeling …

… of actually completing things.

Upon reflection, it appears to have been a sucessful week.

Work – We relocated offices (including my own desk (again)) over the previous week from one slightly pre-used office building to another more well-used office building. My role as part of this project was to ensure that the mechanics of the move as far as IT and Comms occured and proceed smoothly. After recabling the floor, working with networks, telephones and desktops staff it was an almost flawless move, and everyone was up and running easily on Monday morning. I received lots of positive feedback which was good.

Choir – The wrap up SGM for the 62nd Australian Intervarsity Choral Festival Perth 2011, Inc happened. Pending the incorporation of the next festival, it is all over bar a few cheques and paperwork. Overall it was a great festival and as Treasurer was pleased with the final financial result (positive).

Hacking – This weeks little project has been virtualsnack. This is a curses emulator of the UCC Snack Machine and associated ROM. It is based on a previous emulator written with PyGTK and Glade that had bitrotted in the past ten years to be non-functioning and not worth the effort to ressurect. The purpose of the emulator is enable development of code to speak to the machine without having to have the real machine available to test against.

I chose to continue to have the code in python and used npyscreen as the curses UI library. One of the intermediate steps was creating a code sample, EXAMPLE-socket.py, which creates a daemon that speaks to a curses interfaces.

I hereby present V1.0 “Gobbledok” of virtualsnack. virtualsnack is hosted up on Github for the moment, but may move in future. I suspect this item of software will only be of interest to my friends at UCC.

Syndicated 2014-10-25 05:02:51 from Assorted musings

24 Oct 2014 marnanel   » (Journeyer)

Dravidian languages

Today I drew a tree of the relationships between the Dravidian languages (because someone asked about Tamil). Source.



This entry was originally posted at http://marnanel.dreamwidth.org/315548.html. Please comment there using OpenID.

Syndicated 2014-10-24 20:02:57 from Monument

24 Oct 2014 badvogato   » (Master)

just finished reading Joan Biskupic's book 'Breaking In: The Rise of Sonia Sotomayor and the Politics of Justice'.

below is a conversation I came across that is worth recording for my own sake.


From Andre V.
To Susan Y.
Date Jun 11, 2013
Just remember that God is in control and if you or anybody like her try to hurt Israel they all loss and if we let someone like her to hurt God's people they will and always loss also. Just look at WWII and the German people.



On June 7, 2013, Susan wrote:
-------------------------
hi Andre,

I do believe that you are sent by holy spirit for peace in our troubled time and troubled world. Sergio didn't want to take part in your prayer to save his life before the Almighty , he must have his reason. That reason won't die with him., I'm afraid.

There is so much killing going around, witnessing all the tragedy is hard enough for any man to bear, let alone women and children. How can we discern what is our true calling to act or not to act in the name of serving humanity instead of serving existing power struggle?

President Obama just nominated Samantha Power to be Ambassador to U.N. ? Do you think she is the right person?

Below is a very negative report.

http://www.nationalreview.com/article/350297/samantha-power-exemplary-pick-charles-c-w-cooke

Do you believe in its conclusion?

My apology for giving you such heady news and opinion.

Warm Regards

-Susan

24 Oct 2014 mikal   » (Journeyer)

Specs for Kilo

Here's an updated list of the specs currently proposed for Kilo. I wanted to produce this before I start travelling for the summit in the next couple of days because I think many of these will be required reading for the Nova track at the summit.

API

  • Add instance administrative lock status to the instance detail results: review 127139 (abandoned).
  • Add more detailed network information to the metadata server: review 85673.
  • Add separated policy rule for each v2.1 api: review 127863.
  • Add user limits to the limits API (as well as project limits): review 127094.
  • Allow all printable characters in resource names: review 126696.
  • Expose the lock status of an instance as a queryable item: review 85928 (approved).
  • Implement instance tagging: review 127281 (fast tracked, approved).
  • Implement tags for volumes and snapshots with the EC2 API: review 126553 (fast tracked, approved).
  • Implement the v2.1 API: review 126452 (fast tracked, approved).
  • Microversion support: review 127127.
  • Move policy validation to just the API layer: review 127160.
  • Provide a policy statement on the goals of our API policies: review 128560.
  • Support X509 keypairs: review 105034.


Administrative

  • Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705 (abandoned).
  • Enforce instance uuid uniqueness in the SQL database: review 128097 (fast tracked, approved).


Containers Service



Hypervisor: Docker



Hypervisor: FreeBSD

  • Implement support for FreeBSD networking in nova-network: review 127827.


Hypervisor: Hyper-V

  • Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190 (approved).


Hypervisor: Ironic



Hypervisor: VMWare

  • Add ephemeral disk support to the VMware driver: review 126527 (fast tracked, approved).
  • Add support for the HTML5 console: review 127283.
  • Allow Nova to access a VMWare image store over NFS: review 126866.
  • Enable administrators and tenants to take advantage of backend storage policies: review 126547 (fast tracked, approved).
  • Enable the mapping of raw cinder devices to instances: review 128697.
  • Implement vSAN support: review 128600 (fast tracked, approved).
  • Support multiple disks inside a single OVA file: review 128691.
  • Support the OVA image format: review 127054 (fast tracked, approved).


Hypervisor: libvirt



Instance features



Internal

  • Move flavor data out of the system_metdata table in the SQL database: review 126620 (approved).
  • Transition Nova to using the Glance v2 API: review 84887.


Internationalization

  • Enable lazy translations of strings: review 126717 (fast tracked).


Performance

  • Dynamically alter the interval nova polls components at based on load and expected time for an operation to complete: review 122705.


Scheduler

  • Add an IOPS weigher: review 127123 (approved).
  • Add instance count on the hypervisor as a weight: review 127871 (abandoned).
  • Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530 (abandoned).
  • Convert the resource tracker to objects: review 128964 (fast tracked, approved).
  • Create an object model to represent a request to boot an instance: review 127610.
  • Decouple services and compute nodes in the SQL database: review 126895.
  • Implement resource objects in the resource tracker: review 127609.
  • Isolate the scheduler's use of the Nova SQL database: review 89893.
  • Move select_destinations() to using a request object: review 127612.


Security

  • Provide a reference implementation for console proxies that uses TLS: review 126958 (fast tracked).
  • Strongly validate the tenant and user for quota consuming requests with keystone: review 92507.


Tags for this post: openstack kilo blueprint spec
Related posts: One week of Nova Kilo specifications; Compute Kilo specs are open; On layers; Juno nova mid-cycle meetup summary: slots; My candidacy for Kilo Compute PTL; Juno nova mid-cycle meetup summary: nova-network to Neutron migration

Comment

Syndicated 2014-10-23 19:27:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

23 Oct 2014 dmarti   » (Master)

QoTD: Bob Hoffman

The addiction to targeting, which digital technology has only amplified, has derailed the advertising industry from concentrating on its real job—creating interesting messages.

Bob Hoffman

Syndicated 2014-10-23 12:21:47 from Don Marti

23 Oct 2014 mjg59   » (Master)

Linux Container Security

First, read these slides. Done? Good.

Hypervisors present a smaller attack surface than containers. This is somewhat mitigated in containers by using seccomp, selinux and restricting capabilities in order to reduce the number of kernel entry points that untrusted code can touch, but even so there is simply a greater quantity of privileged code available to untrusted apps in a container environment when compared to a hypervisor environment[1].

Does this mean containers provide reduced security? That's an arguable point. In the event of a new kernel vulnerability, container-based deployments merely need to upgrade the kernel on the host and restart all the containers. Full VMs need to upgrade the kernel in each individual image, which takes longer and may be delayed due to the additional disruption. In the event of a flaw in some remotely accessible code running in your image, an attacker's ability to cause further damage may be restricted by the existing seccomp and capabilities configuration in a container. They may be able to escalate to a more privileged user in a full VM.

I'm not really compelled by either of these arguments. Both argue that the security of your container is improved, but in almost all cases exploiting these vulnerabilities would require that an attacker already be able to run arbitrary code in your container. Many container deployments are task-specific rather than running a full system, and in that case your attacker is already able to compromise pretty much everything within the container. The argument's stronger in the Virtual Private Server case, but there you're trading that off against losing some other security features - sure, you're deploying seccomp, but you can't use selinux inside your container, because the policy isn't per-namespace[2].

So that seems like kind of a wash - there's maybe marginal increases in practical security for certain kinds of deployment, and perhaps marginal decreases for others. We end up coming back to the attack surface, and it seems inevitable that that's always going to be larger in container environments. The question is, does it matter? If the larger attack surface still only results in one more vulnerability per thousand years, you probably don't care. The aim isn't to get containers to the same level of security as hypervisors, it's to get them close enough that the difference doesn't matter.

I don't think we're there yet. Searching the kernel for bugs triggered by Trinity shows plenty of cases where the kernel screws up from unprivileged input[3]. A sufficiently strong seccomp policy plus tight restrictions on the ability of a container to touch /proc, /sys and /dev helps a lot here, but it's not full coverage. The presentation I linked to at the top of this post suggests using the grsec patches - these will tend to mitigate several (but not all) kernel vulnerabilities, but there's tradeoffs in (a) ease of management (having to build your own kernels) and (b) performance (several of the grsec options reduce performance).

But this isn't intended as a complaint. Or, rather, it is, just not about security. I suspect containers can be made sufficiently secure that the attack surface size doesn't matter. But who's going to do that work? As mentioned, modern container deployment tools make use of a number of kernel security features. But there's been something of a dearth of contributions from the companies who sell container-based services. Meaningful work here would include things like:


  • Strong auditing and aggressive fuzzing of containers under realistic configurations
  • Support for meaningful nesting of Linux Security Modules in namespaces
  • Introspection of container state and (more difficult) the host OS itself in order to identify compromises

These aren't easy jobs, but they're important, and I'm hoping that the lack of obvious development in areas like this is merely a symptom of the youth of the technology rather than a lack of meaningful desire to make things better. But until things improve, it's going to be far too easy to write containers off as a "convenient, cheap, secure: choose two" tradeoff. That's not a winning strategy.

[1] Companies using hypervisors! Audit your qemu setup to ensure that you're not providing more emulated hardware than necessary to your guests. If you're using KVM, ensure that you're using sVirt (either selinux or apparmor backed) in order to restrict qemu's privileges.
[2] There's apparently some support for loading per-namespace Apparmor policies, but that means that the process is no longer confined by the sVirt policy
[3] To be fair, last time I ran Trinity under Docker under a VM, it ended up killing my host. Glass houses, etc.

comment count unavailable comments

Syndicated 2014-10-23 07:47:36 from Matthew Garrett

22 Oct 2014 mbrubeck   » (Journeyer)

A little randomness for Hacker News

This is a little thing I made to try out an idea for improving the ranking of items on Hacker News, Reddit, and similar sites.

In systems that rely heavily on “Most Popular” charts, the rich get richer while the poor tend to stay poor. That is, most people will only look at and rate the items that are featured in the charts. Anything that’s not already in the list will have a much harder time getting votes or ratings. You need visibility to get ratings, and you need ratings to get visibility.

Aggregators try to address this problem by promoting the newest items as well as the most popular ones. But this is hard to do well. From what I can tell, the “new” page at Hacker News gets only a fraction of the front page’s traffic. Most users want to see the best content, not wade through an unfiltered stream of links. So the selection of links that make it to the front page is based on very little input.

So I wrote a userscript that uses the Hacker News API to search for new or low-ranked links and randomly insert just one or two of them into the front page. When the script runs, it will log the items it inserts to the JavaScript console.

Install user script (may require a browser extension)

It’s also available as a bookmarklet for those who can’t or don’t want to install the user script:

Randomize HN (right-click to bookmark)

This gives HN users the chance to see and upvote links that they otherwise wouldn’t, without altering their habits or wading through a ton of unfiltered dreck. The randomness means that each user of the script will see a different set of links. My belief, though I can’t prove it, is that widespread use of this feature would provide additional input that would improve the quality of the selection process.

The script isn’t perfect (search for “FIXME” in the source code for some known issues), but it works well enough to try out the idea. Unfortunately, the HN API doesn’t give access to all the data I’d like. What I really want to see is a bit of built-in randomness in every system that recommends “popular” items.

Syndicated 2014-10-22 22:00:00 from Matt Brubeck

22 Oct 2014 marnanel   » (Journeyer)

Zophobas morio

We got some crickets in the post today, so I put them into a tank we use for feed insects, and there were some Zophobas morio worms in there still. Z. morio is a long wriggly worm when it's a larva, and this is the form in which it's used as spider food. I was surprised, because we haven't had new Z. morio in for months, and I'd assumed that if there were any leftovers they'd be dead by now. But then I noticed the large number of small brown-black beetles in the tank and realised that the worms were (at least) second generation. I don't think I'd ever realised what they looked like when they grew up before: they're small, about a centimetre across, around the size of a new halfpenny.

This entry was originally posted at http://marnanel.dreamwidth.org/315227.html. Please comment there using OpenID.

Syndicated 2014-10-22 21:11:06 from Monument

22 Oct 2014 Stevey   » (Master)

On writing test-cases and testsuites.

Last night I mostly patched my local copy of less to build and link against the PCRE regular expression library.

I've wanted to do that for a while, and reading Raymond Chen's blog post last night made me try it out.

The patch was small and pretty neat, and I'm familiar with GNU less having patched it in the past. But it doesn't contain tests.

Test cases are hard. Many programs, such as less, are used interactively which makes writing a scaffold hard. Other programs suffer from a similar fate - I'm not sure how you'd even test a web browser such as Firefox these days - mangleme would catch some things, eventually, but the interactive stuff? No clue.

In the past MySQL had a free set of test cases, but my memory is that Oracle locked them up. SQLite is famous for its decent test coverage. But off the top of my head I can't think of other things.

As a topical example there don't seem to be decent test-cases for either bash or openssl. If it compiles it works, more or less.

I did start writing some HTTP-server test cases a while back, but that was just to automate security attacks. e.g. Firing requests like:

GET /../../../etc/passwd HTTP/1.0
GET //....//....//....//etc/passwd HTTP/1.0
etc

(It's amazing how many toy HTTP server components included in projects and products don't have decent HTTP-servers.)

I could imagine that being vaguely useful, especially because it is testing the protocol-handling rather than a project-specific codebase.

Anyway, I'm thinking writing test cases for things is good, but struggling to think of a decent place to start. The project has to be:

  • Non-interactive.
  • Open source.
  • Widely used - to make it a useful contribution.
  • Not written in some fancy language.
  • Open to receiving submissions.

Comments welcome; but better yet why not think about the test-coverage of any of your own packages and projects...?

Syndicated 2014-10-22 09:21:39 from Steve Kemp's Blog

20 Oct 2014 Pizza   » (Master)

Shinko/Sinfonia printers

I just committed initial support for the Shinko/Sinfonia CHC-S1245, CHS-S6145, and CHC-S6245 into Gutenprint. They use printjob structures similar to the S2145, and appear to share the same basic driver core, so the odds are high that the existing S2145 backend will work with only minor changes.

So, if there's anyone out there with one of those models (or better yet, some low-level documentation on their communication protocol) drop me a note, and from there we should be able to get things working pretty quickly.

There's still the CHC-S8145 and the DP-1045 to sort out, but those are for another time.

I think that's it for printer hacking for a while, barring bugfixes and the ongoing Mitsubishi CP-D70/D707/K60 saga. Testers needed...

Syndicated 2014-10-20 03:49:50 from Solomon Peachy

19 Oct 2014 mtearle   » (Journeyer)

A preponderance of yak shaving….

It is often observed that attempting to undertake one task begets another, with the corollary that two days later you’ve built a bikeshed painted in a multitude of colours.

So, dear readers, this tale of woe begins with the need to update my blog to something useful after 18 months of neglect and more. I had been writing a travel blog from when I took some leave off work to wander the globe. For this task, a new more generic DNS entry and an upgrade to the WordPress installation and syndication with my Advogato blog. Easily accomplished and a sense of progress.

This blog entry is going to be mostly a technical one. I’ll try incorporating more of real life in other entries.

Great, now I can tell the world about my little project toying with Vagrant and Puppet.

It is called “Browser In A Box”. It is up on Github https://github.com/mtearle/browser-in-a-box

It is very simple, a Vagrant file and a set of Puppet manifests/modules to launch Chromium in kiosk mode inside a VM to hit a certain URL. This is part of planned later work to look at creating a Vagrant development environment for Concerto.

At this point, I got distracted … aside from the liberal upgrades of bash on various machines to address Shellshock

Then I accidentally purchased a new Ultrabook. My previous netbook had been getting long in the tooth and it was time to upgrade. I ended up purchasing a Toshiba Satellite NB10, a reasonable processor Intel N2830, 4 Gig of RAM and 500 Gigs of spinning rust. Those are the nice bits.

On the negatives, Crappy Toshiba keyboard layout with the ~ key in a stupid spot and a UEFI bios. It is now blatantly apparent why Matthew Garrett drinks copious quantities of gin.

Special brickbats go to the Ubuntu installer for repartitioning and eating my Windows installation and recovery partition. (The option to install over my test Debian installation got over enthusiastic).  The wireless chipset (Atheros) has a known problem where it confuses the access point.

The next distraction ended up being a fit of procastination in terms of rearranging my tiny apartment. I’ve now modelled it in a program called Sweet Home 3D. Easy and straight forward to use. Needs a few more furniture models, but perfectly functional. I shall use it again next time I move.

Finally, we arrive at the the original task. I want to start syncing my calendars between various locations (written here for my benefit later).

They are:

  • Work stream – From my Work (Exchange) to my private host (Radicale) to Google Calendar (which will get to my Android phone)
  • Personal stream – From my private host (Radicale) to Google Calendar (and back again)
  • Party stream – From Facebook’s ical export to my private host and Google Calendar

In addition, various syncing of contacts but not my primary focus at the moment.

It appears that syncevolution will do most of what I want here. The challenge revolves around how to get it working. Ultimately, I want to have this live headless hosted on a virtual machine not running a desktop.

In a fit of enthusiasm, I decided upon attempting to build it from source as opposed to using the packages provided from the upstream (to avoid dragging in unnecessary dependencies.

I need to build from HEAD due to recent code added to syncevolution to support the change in Google’s CALDAV API to be behind OAuth V2.

This was not an overly successful exercise, I ended up getting something built but it didn’t ultimately work.

Problems encountered were:

  • libwbxml2 – The upstream at opensync.org is down. There appears to be forks, so playing the game of guessing the current head/release version.
  • activesyncd – Build system is currently broken in parts. There appears to be bit rot around the evolution bindings as the evolution API has changed

I gave up at that point. I’ve since spun up a different virtual machine with Debian Jessie and an install of Gnome. The packages from the syncevolution upstream installed cleanly, but have yet to work out the incarnations to make it work. However, that my friends is a story for a later blog entry…

Syndicated 2014-10-19 03:18:59 from Assorted musings

19 Oct 2014 marnanel   » (Journeyer)

and now, a word from 13-year-old me

A few years back, sorting through some of my old papers, I found this poem. It's dated 11th December 1988, when I was nearly fourteen.

FRIENDS

They will stand beside you
When all things are good.
And in the times when things are bad
Beside you they have stood.
They always tell the truth to you
As every good friend must
And they are reliable:
Friends you always trust.
They never will say nasty things
About the clothes you wear
They'll stand up for you against others
When you're not there.
You can always trust your friends
To hold your place in queues.
They'll always tell you "You played well",
Even if you lose.
Always keeping by your side:
Friendship never ends.
Yet, after all, we're only human:
Who has friends?

This entry was originally posted at http://marnanel.dreamwidth.org/315117.html. Please comment there using OpenID.

Syndicated 2014-10-19 02:00:23 from Monument

19 Oct 2014 mako   » (Master)

Another Round of Community Data Science Workshops in Seattle

Pictures from the CDSW sessions in Spring 2014
Pictures from the CDSW sessions in Spring 2014

I am helping coordinate three and a half day-long workshops in November for anyone interested in learning how to use programming and data science tools to ask and answer questions about online communities like Wikipedia, free and open source software, Twitter, civic media, etc. This will be a new and improved version of the workshops run successfully earlier this year.

The workshops are for people with no previous programming experience and will be free of charge and open to anyone.

Our goal is that, after the three workshops, participants will be able to use data to produce numbers, hypothesis tests, tables, and graphical visualizations to answer questions like:

  • Are new contributors to an article in Wikipedia sticking around longer or contributing more than people who joined last year?
  • Who are the most active or influential users of a particular Twitter hashtag?
  • Are people who participated in a Wikipedia outreach event staying involved? How do they compare to people that joined the project outside of the event?

If you are interested in participating, fill out our registration form here before October 30th. We were heavily oversubscribed last time so registering may help.

If you already know how to program in Python, it would be really awesome if you would volunteer as a mentor! Being a mentor will involve working with participants and talking them through the challenges they encounter in programming. No special preparation is required. If you’re interested, send me an email.

Syndicated 2014-10-19 01:19:52 from copyrighteous

18 Oct 2014 dmarti   » (Master)

Snapchat ads and committing to non-targeting

Recent Snapchat blog, announcing ads:

We want to see if we can deliver an experience that’s fun and informative, the way ads used to be, before they got creepy and targeted. It’s nice when all of the brilliant creative minds out there get our attention with terrific content.

That's a great idea, and ties in with what I've been saying all along about the targeted ad problem.

But I'm not optimistic. Snapchat is still running on a mobile phone, running within an environment that's either problematic or outright privacy-hostile. If Snapchat can't commit to its core feature, the idea that photos disappear after sending, how can the company credibly commit to less creepy, more valuable advertising?

It would be a huge win for Snapchat if they could pull it off. But I doubt that a single app can do it.

Signalful ads are an emergent benefit from media that tend to build user confidence through tracking resistance. Non-creepiness can't be declared, it has to be discovered.

Syndicated 2014-10-18 12:11:52 from Don Marti

18 Oct 2014 Stevey   » (Master)

On the names we use in email

Yesterday I received a small rush of SPAM mails, all of which were 419 scams, and all of them sent by "Mrs Elizabeth PETERSEN".

It struck me that I can't think of ever receiving a legitimate mail from a "Mrs XXX [YYY]", but I was too busy to check.

Today I've done so. Of the 38,553 emails I've received during the month of October 2014 I've got a hell of a lot of mails with a From address including a "Mrs" prefix:

"Mrs.Clanzo Amaki" <marilobouabre14@yahoo.co.jp>
"Mrs Sarah Mamadou"<investment@payment.com>
"Mrs Abia Abrahim" <missfatimajinnah@yahoo.co.jp>
"Mrs. Josie Wilson" <linn3_2008@yahoo.co.jp>
"Mrs. Theresa Luis"<tomaslima@jorgelima.com>

There are thousands more. Not a single one of them was legitimate.

I have one false-positive when repeating the search for a Mr-prefix. I have one friend who has set his sender-address to "Mr Bob Smith", which always reads weirdly to me, but every single other email with a Mr-prefix was SPAM.

I'm not going to use this in any way, since I'm happy with my mail-filtering setup, but it was interesting observation.

Names are funny. My wife changed her surname post-marriage, but that was done largely on the basis that introducing herself as "Doctor Kemp" was simpler than "Doctor Foreign-Name", she'd certainly never introduce herself ever as Mrs Kemp.

Trivia: In Finnish the word for "Man" and "Husband" is the same (mies), but the word for "Woman" (nainen) is different than the word for "Wife" (vaimo).

Syndicated 2014-10-18 08:18:01 (Updated 2014-10-18 23:13:17) from Steve Kemp's Blog

18 Oct 2014 Pizza   » (Master)

More dyesub printer work

The Citizen CW-01 is now confirmed working, and the necessary code has been committed into Gutenprint. With luck, the next release will take less than two years! This should also work with the Olmec OP-900, but I'll need a USB ID in order to add that to the backend.

Meanwhile, I just committed initial support for the Kodak 305 and Kodak 8810 printers to Gutenprint. It's unknown if they need an intelligent backend, but I suppose time will tell. As always, testers welcome.

Here's my current to-do list:

  • Kodak 8800, 7000/7010/7015, and D4000
  • Mitsubishi CP-D80DW and CP-9600DW
  • Shinko S1245, S6145/S6145-5A, S5245, S8145, and DP-1045
  • Sony UP-CR10L and UP-CR20L (aka DNP SL-10 and SL-20)

These models need USB IDs:

  • Citizen CW-02, OP900, OP900-II
  • Mitsubishi CP-3800DW

These models need testing:

  • Mitsubishi CP-3020D/DA/DE/DAE, CP-9550D/DW, and CP-9810D/DW
  • Kodak 8500, 9180, 8810, and 305

I've received inquires about various HiTi models, but without access to the printers (or at least complete USB sniffs of print generation with specific settings logged) I won't be able to make any progress. Their windows spool format is some sort of (compressed!) intermediate language rather than something that's natively dumped to the printer.

Finally, the Mitsubishi CP-D70/D707/K60 remain problematic; despite a lot of work on the backend we're no closer to figuring out the necessary color scaling/dithering the windows drivers employ so the color output from Gutenprint is pretty lousy.

This isn't how I'd intended to spend my Friday night. With luck the fever will finally break tonight so I can get out and about tomorrow..

Syndicated 2014-10-18 03:33:18 from Solomon Peachy

17 Oct 2014 wingo   » (Master)

ffs ssl

I just set up SSLTLS on my web site. Everything can be had via https://wingolog.org/, and things appear to work. However the process of transitioning even a simple web site to SSL is so clownshoes bad that it's amazing anyone ever does it. So here's an incomplete list of things that can go wrong when you set up TLS on a web site.

You search "how to set up https" on the Googs and click the first link. It takes you here which tells you how to use StartSSL, which generates the key in your browser. Whoops, your private key is now known to another server on this internet! Why do people even recommend this? It's the worst of the worst of Javascript crypto.

OK so you decide to pay for a certificate, assuming that will be better, and because who knows what's going on with StartSSL. You've heard of RapidSSL so you go to rapidssl.com. WTF their price is 49 dollars for a stupid certificate? Your domain name was only 10 dollars, and domain name resolution is an actual ongoing service, unlike certificate issuance that just happens one time. You can't believe it so you click through to the prices to see, and you get this:

Whatttttttttt

OK so I'm using Epiphany on Debian and I think that uses the system root CA list which is different from what Chrome or Firefox do but Jesus this is shaking my faith in the internet if I can't connect to an SSL certificate provider over SSL.

You remember hearing something on Twitter about cheaper certs, and oh ho ho, it's rapidsslonline.com, not just RapidSSL. WTF. OK. It turns out Geotrust and RapidSSL and Verisign are all owned by Symantec anyway. So you go and you pay. Paying is the first thing you have to do on rapidsslonline, before anything else happens. Welp, cross your fingers and take out your credit card, cause SSLanta Clause is coming to town.

Recall, distantly, that SSL has private keys and public keys. To create an SSL certificate you have to generate a key on your local machine, which is your private key. That key shouldn't leave your control -- that's why the DigitalOcean page is so bogus. The certification authority (CA) then needs to receive your public key and then return it signed. You don't know how to do this, because who does? So you Google and copy and paste command line snippets from a website. Whoops!

Hey neat it didn't delete your home directory, cool. Let's assume that your local machine isn't rooted and that your server isn't rooted and that your hosting provider isn't rooted, because that would invalidate everything. Oh what so the NSA and the five eyes have an ongoing program to root servers? Um, well, water under the bridge I guess. Let's make a key. You google "generate ssl key" and this is the first result.

# openssl genrsa -des3 -out foo.key 1024

Whoops, you just made a 1024-bit key! I don't know if those are even accepted by CAs any more. Happily if you leave off the 1024, it defaults to 2048 bits, which I guess is good.

Also you just made a key with a password on it (that's the -des3 part). This is eminently pointless. In order to use your key, your web browser will need the decrypted key, which means it will need the password to the key. Adding a password does nothing for you. If you lost your private key but you did have it password-protected, you're still toast: the available encryption cyphers are meant to be fast, not hard to break. Any serious attacker will crack it directly. And if they have access to your private key in the first place, encrypted or not, you're probably toast already.

OK. So let's say you make your key, and make what's called the "CRT", to ask for the cert.

# openssl req -new -key foo.key -out foo.csr

Now you're presented with a bunch of pointless-looking questions like your country code and your "organization". Seems pointless, right? Well now I have to live with this confidence-inspiring dialog, because I left off the organization:

Don't mess up, kids! But wait there's more. You send in your CRT, finally figure out how to receive mail for hostmaster@yourdomain.org because that's what "verification" means (not, god forbid, control of the actual web site), and you get back a certificate. Now the fun starts!

How are you actually going to serve SSL? The truly paranoid use an out-of-process SSL terminator. Seems legit except if you do that you lose any kind of indication about what IP is connecting to your HTTP server. You can use a more HTTP-oriented terminator like bud but then you have to mess with X-Forwarded-For headers and you only get them on the first request of a connection. You could just enable mod_ssl on your Apache, but that code is terrifying, and do you really want to be running Apache anyway?

In my case I ended up switching over to nginx, which has a startlingly underspecified configuration language, but for which the Debian defaults are actually not bad. So you uncomment that part of the configuration, cross your fingers, Google a bit to remind yourself how systemd works, and restart the web server. Haich Tee Tee Pee Ess ahoy! But did you remember to disable the NULL authentication method? How can you test it? What about the NULL encryption method? These are actual things that are configured into OpenSSL, and specified by standards. (What is the use of a secure communications standard that does not provide any guarantee worth speaking of?) So you google, copy and paste some inscrutable incantation into your config, turn them off. Great, now you are a dilettante tweaking your encryption parameters, I hope you feel like a fool because I sure do.

Except things are still broken if you allow RC4! So you better make sure you disable RC4, which incidentally is exactly the opposite of the advice that people were giving out three years ago.

OK, so you took your certificate that you got from the CA and your private key and mashed them into place and it seems the web browser works. Thing is though, the key that signs your certificate is possibly not in the actual root set of signing keys that browsers use to verify the key validity. If you put just your key on the web site without the "intermediate CA", then things probably work but browsers will make an additional request to get the intermediate CA's key, slowing down everything. So you have to concatenate the text files with your key and the one with the intermediate CA's key. They look the same, just a bunch of numbers, but don't get them in the wrong order because apparently the internet says that won't work!

But don't put in too many keys either! In this image we have a cert for jsbin.com with one intermediate CA:

And here is the same but with an a different root that signed the GeoTrust Global CA certificate. Apparently there was a time in which the GeoTrust cert hadn't been added to all of the root sets yet, and it might not hurt to include them all:

Thing is, the first one shows up "green" in Chrome (yay), but the second one shows problems ("outdated security settings" etc etc etc). Why? Because the link from Equifax to Geotrust uses a SHA-1 signature, and apparently that's not a good idea any more. Good times? (Poor Remy last night was doing some basic science on the internet to bring you these results.)

Or is Chrome denying you the green because it was RapidSSL that signed your certificate with SHA-1 and not SHA-256? It won't tell you! So you Google and apply snakeoil and beg your CA to reissue your cert, hopefully they don't charge for that, and eventually all is well. Chrome gives you the green.

Or does it? Probably not, if you're switching from a web site that is also available over HTTP. Probably you have some images or CSS or Javascript that's being loaded over HTTP. You fix your web site to have scheme-relative URLs (like //wingolog.org/ instead of http://wingolog.org/), and make sure that your software can deal with it all (I had to patch Guile :P). Update all the old blog posts! Edit all the HTMLs! And finally, green! You're golden!

Or not! Because if you left on SSLv3 support you're still broken! Also, TLSv1.0, which is actually greater than SSLv3 for no good reason, also has problems; and then TLS1.1 also has problems, so you better stick with just TLSv1.2. Except, except, older Android phones don't support TLSv1.2, and neither does the Googlebot, so you don't get the rankings boost you were going for in the first place. So you upgrade your phone because that's a thing you want to do with your evenings, and send snarky tweets into the ether about scumbag google wanting to promote HTTPS but not supporting the latest TLS version.

So finally, finally, you have a web site that offers HTTPS and HTTP access. You're good right? Except no! (Catching on to the pattern?) Because what happens is that people just type in web addresses to their URL bars like "google.com" and leave off the HTTP, because why type those stupid things. So you arrange for http://www.wobsite.com to redirect https://www.wobsite.com for users that have visited the HTTPS site. Except no! Because any network attacker can simply strip the redirection from the HTTP site.

The "solution" for this is called HTTP Strict Transport Security, or HSTS. Once a visitor visits your HTTPS site, the server sends a response that tells the browser never to fetch HTTP from this site. Except that doesn't work the first time you go to a web site! So if you're Google, you friggin add your name to a static list in the browser. EXCEPT EVEN THEN watch out for the Delorean.

And what if instead they go to wobsite.com instead of the www.wobsite.com that you configured? Well, better enable HSTS for the whole site, but to do anything useful with such a web request you'll need a wildcard certificate to handle the multiple URLs, and those run like 150 bucks a year, for a one-bit change. Or, just get more single-domain certs and tack them onto your cert, using the precision tool cat, but don't do too many, because if you do you will overflow the initial congestion window of the TCP connection and you'll have to wait for an ACK on your certificate before you can actually exchange keys. Don't know what that means? Better look it up and be an expert, or your wobsite's going to be slow!

If your security goals are more modest, as they probably are, then you could get burned the other way: you could enable HSTS, something could go wrong with your site (an expired certificate perhaps), and then people couldn't access your site at all, even if they have no security needs, because HTTP is turned off.

Now you start to add secure features to your web app, safe with the idea you have SSL. But better not forget to mark your cookies as secure, otherwise they could be leaked in the clear, and better not forget that your website might also be served over HTTP. And better check up on when your cert expires, and better have a plan for embedded browsers that don't have useful feedback to the user about certificate status, and what about your CA's audit trail, and better stay on top of the new developments in security! Did you read it? Did you read it? Did you read it?

It's a wonder anything works. Indeed I wonder if anything does.

Syndicated 2014-10-17 14:33:30 from wingolog

17 Oct 2014 bagder   » (Master)

curl is no POODLE

Once again the internet flooded over with reports and alerts about a vulnerability using a funny name: POODLE. If you have even the slightest interest in this sort of stuff you’ve already grown tired and bored about everything that’s been written about this so why on earth do I have to pile on and add to the pain?

This is my way of explaining how POODLE affects or doesn’t affect curl, libcurl and the huge amount of existing applications using libcurl.

Is my application using HTTPS with libcurl or curl vulnerable to POODLE?

No. POODLE really is a browser-attack.

Motivation

The POODLE attack is a combination of several separate pieces that when combined allow attackers to exploit it. The individual pieces are not enough stand-alone.

SSLv3 is getting a lot of heat now since POODLE must be able to downgrade a connection to SSLv3 from TLS to work. Downgrade in a fairly crude way – in libcurl, only libcurl built to use NSS as its TLS backend supports this way of downgrading the protocol level.

Then, if an attacker manages to downgrade to SSLv3 (both the client and server must thus allow this) and get to use the sensitive block cipher of that protocol, it must maintain a connection to the server and then retry many similar requests to the server in order to try to work out details of the request – to figure out secrets it shouldn’t be able to. This would typically be made using javascript in a browser and really only HTTPS allows this so no other SSL-using protocol can be exploited like this.

For the typical curl user or a libcurl user, there’s A) no javascript and B) the application already knows the request it is doing and normally doesn’t inject random stuff from 3rd party sources that could be allowed to steal secrets. There’s really no room for any outsider here to steal secrets or cookies or whatever.

How will curl change

There’s no immediate need to do anything as curl and libcurl are not vulnerable to POODLE.

Still, SSLv3 is long overdue and is not really a modern protocol (TLS 1.0, the successor, had its RFC published 1999) so in order to really avoid the risk that it will be possible exploit this protocol one way or another now or later using curl/libcurl, we will disable SSLv3 by default in the next curl release. For all TLS backends.

Why? Just to be extra super cautious and because this attack helped us remember that SSLv3 is old and should be let down to die.

If possible, explicitly requesting SSLv3 should still be possible so that users can still work with their legacy systems in dire need of upgrade but placed in corners of the world that every sensible human has since long forgotten or just ignored.

In-depth explanations of POODLE

I especially like the ones provided by PolarSSL and GnuTLS, possibly due to their clear “distance” from browsers.

Syndicated 2014-10-17 08:28:08 from daniel.haxx.se

17 Oct 2014 marnanel   » (Journeyer)

Why, why, why, Eliza?

Tell me some more about when you saw light on my window.
Earlier on you were lost like a slave I can't free.
I understand you.
Is it because I deceived you that you came to me?
My, my, my, Eliza!
Why, why, why, Eliza?
I can see you're just a conditional tree
But you remind me we came here to talk about me.

This entry was originally posted at http://marnanel.dreamwidth.org/314265.html. Please comment there using OpenID.

Syndicated 2014-10-16 23:54:00 from Monument

16 Oct 2014 bagder   » (Master)

FOSS them students

On October 16th, I visited DSV at Stockholm University where I had the pleasure of holding a talk and discussion with students (and a few teachers) under the topic Contribute to Open Source. Around 30 persons attended.

Here are the slides I use, as usual possibly not perfectly telling stand-alone without the talk but there was no recording made and I talked in Swedish anyway…

Contribute to Open Source from Daniel Stenberg

Syndicated 2014-10-16 21:01:58 from daniel.haxx.se

15 Oct 2014 mones   » (Journeyer)

FOSS or not FOSS, that's the question

Today in #claws IRC channel some user wanted to move away from Claws Mail to another MUA. That probably happens every day or two, so nobody really cares (I don't, at least).

Claws' storage format is MH, nothing exotic or unknown, hence there's no explicit exporting utilities, as requested by that user. Anyway one of the developers suggested mh2mbox, which seems a pretty straightforward option. Claws has also a mailmbox plugin, which can be populated with messages from MH folders, but when you have lots of them the task becomes boring :-)

Anyway, the point of this post was not the technicalities of conversion but more the ideas people has about FOSS. At some point, after some arguing about how developers doesn't listen to users and how wrong donating to the project had been, the user said:

12:54 < somebody> If I develop a system, and I want people to use it, then I 
                  have a duty to listen to people and consider to make it 
                  useable for them ... or else, they won't use it.

That's a huge misconception, probably because nobody reads the license nowadays. Yeah, it's free, just download it! Reading licen-what? It's free!

I'd put it clear: I'm not a company, I don't want people to use my software, I let people use it if it's useful to them, and of course I'd like it to be useful.
But if not, you already have the source and can (learn to) modify it at will, or pay some other to do so. Nothing else is given to you, remember:

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Syndicated 2014-10-15 12:33:18 from Ricardo Mones

15 Oct 2014 benad   » (Apprentice)

Moniker, the Security Weak Point

By the time I heard about the "Shellshock bug" security hole in the morning of September 25, already the small Debian Linux server that I built up in Ramnode to host my web site patched that security hole by itself. At any rate, my web site hosts only static pages, so it was never impacted.

While I am in control of the security of my web site starting from the Linux kernel, to the web server, up to the web pages it serves, I'm still dependent on its hosting service (Ramnode) to be secure. Another potential danger would be for someone to hijack the "benad.me" domain name, and make it point to a version of the site filled with malware and viruses. Sadly, this almost happened.

Back in 2008 I registered my domain through the registrar Moniker, which used to be recommended partly based on its security. They implemented additional features that one could pay to "lock" the domain and prevent unauthorized transfers from someone that would steal your user name and password. Since then though, the company was bought by another company, and what is now called Moniker is only by name, both in terms of staffing and software.

I did notice a difference in tone in email communications from the new Moniker. They seemed to be highly focused on domain name auctions, and would automatically auction off expired domains. This felt like a conflict of interest, as Moniker would deride higher profit auctioning off your domain than helping you renewing it. Of course, they would never do that on valuable customers that do "domain speculation" and own a large number of (unused) domains, but still that raises the suspicion that the company was sold based on the number of domains it had and how much money they can extract from large speculators rather than providing valuable customer service.

The "new" Moniker had a security hole in 2013, and to fix that Moniker forced users to change their password the next time they logged in. Note though that this happened with the old version of that web site. This summer, the parent company that bought Moniker (and its name) scrapped the old site's code and replaced it with a new broken, buggy interface. The new interface also brought with it worse security, and made the domain locking feature completely ineffective.

By early October, Moniker sent an email to all its users saying that for untold security reasons, all the account passwords would be reset. The shock was that the email contained both the user names and passwords of all the user's accounts. I was shocked that my old Moniker account, identified by a standard-looking user name, was placed under a parent, numerically-identified user name I've never seen, and another numerical sub-account that was created without my knowledge. It should be noted that I could never access the numerical sub-account, even when using the password provided in the email. Also, the email said that your new passwords must fit security requirements, including the use of at least one "special character", even though the passwords provided in the email didn't contain any special character, and when attempting to change passwords, it would refuse most special characters.

OK, I'm not a security expert, but sending user names and passwords in an email, refusing special characters (which would indicate that they don't use bcrypt), and resetting the passwords of all users may indicate that they were hacked. Badly. Moniker cited the Shellshock bug, but as reports of stolen domains started to appear, a user came forth saying that the security hole predated Shellshock by a month.

So, I was convinced that Moniker had a pattern of behaviour of not taking security seriously, that is until they experience a mass exodus of their customers. I started the process of domain name transfer the day after they announced the password reset, and I would recommend everybody else to do the same. I transferred to Namecheap. Despite its name, in my case the price was the same, though as a test I created a new empty account before the transfer, and already I could attest that they take security seriously, including emails for account activity (using secondary email addresses in rotation) and 2-factor authentication (using SMS for now). I completed the transfer yesterday, so that would explain why there was a little bit of downtime when resolving my domain name.

Syndicated 2014-10-15 01:21:21 from Benad's Blog

14 Oct 2014 crhodes   » (Master)

still working on reproducible builds

It’s been nearly fifteen years, and SBCL still can’t be reliably built by other Lisp compilers.

Of course, other peoples’ definition of “reliably” might differ. We did achieve successful building under unrelated Lisp compilers twelve years ago; there were a couple of nasty bugs along the way, found both before and after that triumphant announcement, but at least with a set of compilers whose interpretation of the standard was sufficiently similar to SBCL’s own, and with certain non-mandated but expected features (such as the type (array (unsigned-byte 8) (*)) being distinct from simple-vector, and single-float being distinct from double-float), SBCL achieved its aim of being buildable on a system without an SBCL binary installed (indeed, using CLISP or XCL as a build host, SBCL could in theory be bootstrapped starting with only gcc).

For true “reliability”, though, we should not be depending on any particular implementation-defined features other than ones we actually require – or if we are, then the presence or absence of them should not cause a visible difference in the resulting SBCL. The most common kind of leak from the host lisp to the SBCL binary was the host’s value of most-positive-fixnum influencing the target, causing problems from documentation errors all the way up to type errors in the assembler. Those leaks were mostly plugged a while ago, though they do recur every so often; there are other problems, and over the last week I spent some time tracking down three of them.

The first: if you’ve ever done (apropos "PRINT") or something similar at the SBCL prompt, you might wonder at the existence of functions named something like SB-VM::|CACHED-FUN--PINSRB[(EXT-2BYTE-XMM-REG/MEM ((PREFIX (QUOTE (102))) (OP1 (QUOTE (58))) (OP2 (QUOTE (32))) (IMM NIL TYPE (QUOTE IMM-BYTE))) (QUOTE (NAME TAB REG , REG/MEM ...)))]-EXT-2BYTE-XMM-REG/MEM-PRINTER|.

What is going on there? Well, these functions are a part of the disassembler machinery; they are responsible for taking a certain amount of the machine code stream and generating a printed representation of the corresponding assembly: in this case, for the PINSRB instruction. Ah, but (in most instruction sets) related instructions share a fair amount of structure, and decoding and printing a PINSRD instruction is basically the same as for PINSRB, with just one #x20 changed to a #x22 – in both cases we want the name of the instruction, then a tab, then the destination register, a comma, the source, another comma, and the offset in the destination register. So SBCL arranges to reuse the PINSRB instruction printer for PINSRD; it maintains a cache of printer functions, looked up by printer specification, and reuses them when appropriate. So far, so normal; the ugly name above is the generated name for such a function, constructed by interning a printed, string representation of some useful information.

Hm, but wait. See those (QUOTE (58)) fragments inside the name? They result from printing the list (quote (58)). Is there a consensus on how to print that list? Note that *print-pretty* is bound to nil for this printing; prior experience has shown that there are strong divergences between implementations, as well as long-standing individual bugs, in pretty-printer support. So, what happens if I do (write-to-string '(quote foo) :pretty nil)?

  • SBCL: "(QUOTE FOO)", unconditionally
  • CCL: "'FOO" by default; "(QUOTE FOO)" if ccl:*print-abbreviate-quote* is set to nil
  • CLISP: "'FOO", unconditionally (I read the .d code with comments in half-German to establish this)

So, if SBCL was compiled using CLISP, the name of the same function in the final image would be SB-VM::|CACHED-FUN--PINSRB[(EXT-2BYTE-XMM-REG/MEM ((PREFIX '(102)) (OP1 '(58)) (OP2 '(32)) (IMM NIL TYPE 'IMM-BYTE)) '(NAME TAB REG , REG/MEM ...))]-EXT-2BYTE-XMM-REG/MEM-PRINTER|. Which is shorter, and maybe marginally easier to read, but importantly for my purposes is not bitwise-identical.

Thus, here we have a difference between host Common Lisp compilers which leaks over into the final image, and it must be eliminated. Fortunately, this was fairly straightforward to eliminate; those names are never in fact used to find the function object, so generating a unique name for functions based on a counter makes the generated object file bitwise identical, no matter how the implementation prints two-element lists beginning with quote.

The second host leak is also related to quote, and to our old friend backquote – though not related in any way to the new implementation. Consider this apparently innocuous fragment, which is a simplified version of some code to implement the :type option to defstruct:

  (macrolet ((def (name type n)
             `(progn
                (declaim (inline ,name (setf ,name)))
                (defun ,name (thing)
                  (declare (type simple-vector thing))
                  (the ,type (elt thing ,n)))
                (defun (setf ,name) (value thing)
                  (declare (type simple-vector thing))
                  (declare (type ,type value))
                  (setf (elt thing ,n) value)))))
  (def foo fixnum 0)
  (def bar string 1))

What’s the problem here? Well, the functions are declaimed to be inline, so SBCL records their source code. Their source code is generated by a macroexpander, and so is made up of conses that are generated programmatically (as opposed to freshly consed by the reader). That source code is then stored as a literal object in an object file, which means in practice that instructions for reconstructing a similar object are dumped, to be executed when the object file is processed by load.

Backquote is a reader macro that expands into code that, when evaluated, generates list structure with appropriate evaluation and splicing of unquoted fragments. What does this mean in practice? Well, one reasonable implementation of reading `(type ,type value) might be:

  (cons 'type (cons type '(value)))

and indeed you might (no guarantees) see something like that if you do

  (macroexpand '`(type ,type value))

in the implementation of your choice. Similarly, reading `(setf (elt thing ,n) value) will eventually generate code like

  (cons 'setf (cons (cons 'elt (list 'thing n)) '(value)))

Now, what is “similar”? In this context, it has a technical definition: it relates two objects in possibly-unrelated Lisp images, such that they can be considered to be equivalent despite the fact that they can’t be compared:

similar adj. (of two objects) defined to be equivalent under the similarity relationship.

similarity n. a two-place conceptual equivalence predicate, which is independent of the Lisp image so that two objects in different Lisp images can be understood to be equivalent under this predicate. See Section 3.2.4 (Literal Objects in Compiled Files).

Following that link, we discover that similarity for conses is defined in the obvious way:

Two conses, S and C, are similar if the car of S is similar to the car of C, and the cdr of S is similar to the cdr of C.

and also that implementations have some obligations:

Objects containing circular references can be externalizable objects. The file compiler is required to preserve eqlness of substructures within a file.

and some freedom:

With the exception of symbols and packages, any two literal objects in code being processed by the file compiler may be coalesced if and only if they are similar [...]

Put this all together, and what do we have? That def macro above generates code with similar literal objects: there are two instances of '(value) in it. A host compiler may, or may not, choose to coalesce those two literal '(value)s into a single literal object; if it does, the inline expansion of foo (and bar) will have a circular reference, which must be preserved, showing up as a difference in the object files produced during the SBCL build. The fix? It’s ugly, but portable: since we can’t stop an aggressive compiler from coalescing constants which are similar but not identical, we must make sure that any similar substructure is in fact identical:

  (macrolet ((def (name type n)
             (let ((value '(value)))
               `(progn
                  (declaim (inline ,name (setf ,name)))
                  (defun ,name (thing)
                    (declare (type simple-vector thing))
                    (the ,type (elt thing ,n)))
                  (defun (setf ,name) (value thing)
                    (declare (type simple-vector thing))
                    (declare (type ,type . ,value))
                    (setf (elt thing ,n) . ,value)))))
  (def foo fixnum 0)
  (def bar string 1))

Having dealt with a problem with quote, and a problem with backquote, what might the Universe serve up for my third problem? Naturally, it would be a problem with a code walker. This code walker is somewhat naïve, assuming as it does that its body is made up of forms or tags; it is the assemble macro, which is used implicitly in the definition of VOPs (reusable assembly units); for example, like

  (assemble ()
  (move ptr object)
  (zeroize count)
  (inst cmp ptr nil-value)
  (inst jmp :e DONE)
 LOOP
  (loadw ptr ptr cons-cdr-slot list-pointer-lowtag)
  (inst add count (fixnumize 1))
  (inst cmp ptr nil-value)
  (inst jmp :e DONE)
  (%test-lowtag ptr LOOP nil list-pointer-lowtag)
  (error-call vop 'object-not-list-error ptr)
 DONE))

which generates code to compute the length of a list. The expander for assemble scans its body for any atoms, and generates binding forms for those atoms to labels:

  (let ((new-labels (append labels
                          (set-difference visible-labels inherited-labels))))
  ...
  `(let (,@(mapcar (lambda (name) `(,name (gen-label))) new-labels))
     ...))

The problem with this, from a reproducibility point of view, is that set-difference (and the other set-related functions: union, intersection, set-exclusive-or and their n-destructive variants) do not return the sets with a specified order – which is fine when the objects are truly treated as sets, but in this case the LOOP and DONE label objects ended up in different stack locations depending on the order of their binding. Consequently the machine code for the function emitting code for computing a list’s length – though not the machine code emitted by that function – would vary depending on the host’s implementation of set-difference. The fix here was to sort the result of the set operations, knowing that all the labels would be symbols and that they could be treated as string designators.

And after all this is? We’re still not quite there: there are three to four files (out of 330 or so) which are not bitwise-identical for differing host compilers. I hope to be able to rectify this situation in time for SBCL’s 15th birthday...

Syndicated 2014-10-14 06:51:19 from notes

14 Oct 2014 marnanel   » (Journeyer)

today's bit of sexist nonsense

Here's a conversation on Twitter between me and a man I don’t know in China. (FWIW I have a rather androgynous-looking user picture.)

He said, “Is it true that less than half of UK MPs voted for the resolution to recognise Palestine?”
I said, “Yes. But that’s irrelevant to the validity of the vote.”
He said, “Oh, I think it’s the most relevant thing in the world, sweetheart.”
I said, “I can only tell you what the standing orders of the House say. And I don’t appreciate being called ‘sweetheart’.”
He said, “sorry but when I hear a little dumb-dumb girl talking silly things I think of my 8 year old girls.”



This entry was originally posted at http://marnanel.dreamwidth.org/313996.html. Please comment there using OpenID.

Syndicated 2014-10-14 01:47:33 from Monument

13 Oct 2014 mikal   » (Journeyer)

One week of Nova Kilo specifications

Its been one week of specifications for Nova in Kilo. What are we seeing proposed so far? Here's a summary...

API



Administrative

  • Enable the nova metadata cache to be a shared resource to improve the hit rate: review 126705.


Containers Service



Hypervisor: FreeBSD

  • Implement support for FreeBSD networking in nova-network: review 127827.


Hypervisor: Hyper-V

  • Allow volumes to be stored on SMB shares instead of just iSCSI: review 102190.


Hypervisor: VMWare

  • Add ephemeral disk support to the VMware driver: review 126527 (spec approved).
  • Add support for the HTML5 console: review 127283.
  • Allow Nova to access a VMWare image store over NFS: review 126866.
  • Enable administrators and tenants to take advantage of backend storage policies: review 126547 (spec approved).
  • Support the OVA image format: review 127054.


Hypervisor: libvirt

  • Add a new linuxbridge VIF type, macvtap: review 117465.
  • Add support for SMBFS as a image storage backend: review 103203.
  • Convert to using built in libvirt disk copy mechanisms for cold migrations on non-shared storage: review 126979.
  • Support libvirt storage pools: review 126978.
  • Support quiesce filesystems during snapshot: review 126966.


Instance features

  • Allow direct access to LVM volumes if supported by Cinder: review 127318.


Interal

  • Move flavor data out of the system_metdata table in the SQL database: review 126620.


Internationalization



Scheduler

  • Add an IOPS weigher: review 127123 (spec approved).
  • Allow limiting the flavors that can be scheduled on certain host aggregates: review 122530.
  • Create an object model to represent a request to boot an instance: review 127610.
  • Decouple services and compute nodes in the SQL database: review 126895.
  • Implement resource objects in the resource tracker: review 127609.
  • Move select_destinations() to using a request object: review 127612.


Scheduling

  • Add instance count on the hypervisor as a weight: review 127871.


Security

  • Provide a reference implementation for console proxies that uses TLS: review 126958.
  • Strongly validate the tenant and user for quota consuming requests with keystone: review 92507.


Tags for this post: openstack kilo blueprints spec
Related posts: Compute Kilo specs are open; Blueprints to land in Nova during Juno; On layers; My candidacy for Kilo Compute PTL; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno Nova PTL Candidacy

Comment

Syndicated 2014-10-13 03:27:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

13 Oct 2014 mikal   » (Journeyer)

Compute Kilo specs are open

From my email last week on the topic:

I am pleased to announce that the specs process for nova in kilo is
now open. There are some tweaks to the previous process, so please
read this entire email before uploading your spec!

Blueprints approved in Juno
===========================

For specs approved in Juno, there is a fast track approval process for
Kilo. The steps to get your spec re-approved are:

 - Copy your spec from the specs/juno/approved directory to the
specs/kilo/approved directory. Note that if we declared your spec to
be a "partial" implementation in Juno, it might be in the implemented
directory. This was rare however.
 - Update the spec to match the new template
 - Commit, with the "Previously-approved: juno" commit message tag
 - Upload using git review as normal

Reviewers will still do a full review of the spec, we are not offering
a rubber stamp of previously approved specs. However, we are requiring
only one +2 to merge these previously approved specs, so the process
should be a lot faster.

A note for core reviewers here -- please include a short note on why
you're doing a single +2 approval on the spec so future generations
remember why.

Trivial blueprints
==================

We are not requiring specs for trivial blueprints in Kilo. Instead,
create a blueprint in Launchpad
at https://blueprints.launchpad.net/nova/+addspec and target the
specification to Kilo. New, targeted, unapproved specs will be
reviewed in weekly nova meetings. If it is agreed they are indeed
trivial in the meeting, they will be approved.

Other proposals
===============

For other proposals, the process is the same as Juno... Propose a spec
review against the specs/kilo/approved directory and we'll review it
from there.


After a week I'm seeing something interesting. In Juno the specs process was new, and we saw a pause in the development cycle while people actually wrote down their designs before sending the code. This time around people know what to expect, and there are left over specs from Juno lying around. We're therefore seeing specs approved much faster than in Kilo. This should reduce the effect of the "pipeline flush" that we saw in Juno.

So far we have five approved specs after only a week.

Tags for this post: openstack kilo blueprints spec
Related posts: Blueprints to land in Nova during Juno; On layers; My candidacy for Kilo Compute PTL; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno Nova PTL Candidacy; Juno nova mid-cycle meetup summary: scheduler

Comment

Syndicated 2014-10-12 16:39:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

12 Oct 2014 bagder   » (Master)

What a removed search from Google looks like

Back in the days when I participated in the starting of the Subversion project, I found the mailing list archive we had really dysfunctional and hard to use, so I set up a separate archive for the benefit of everyone who wanted an alternative way to find Subversion related posts.

This archive is still alive and it recently surpassed 370,000 archived emails, all related to Subversion, for seven different mailing lists.

Today I received a notice from Google (shown in its entirety below) that one of the mails received in 2009 is now apparently removed from a search using a name – if done within the European Union at least. It is hard to take this seriously when you look at the page in question, and as there aren’t that very many names involved in that page the possibilities of which name it is aren’t that many. As there are several different mail archives for Subversion mails I can only assume that the alternative search results also have been removed.

This is the first removal I’ve got for any of the sites and contents I host.


Notice of removal from Google Search

Hello,

Due to a request under data protection law in Europe, we are no longer able to show one or more pages from your site in our search results in response to some search queries for names or other personal identifiers. Only results on European versions of Google are affected. No action is required from you.

These pages have not been blocked entirely from our search results, and will continue to appear for queries other than those specified by individuals in the European data protection law requests we have honored. Unfortunately, due to individual privacy concerns, we are not able to disclose which queries have been affected.

Please note that in many cases, the affected queries do not relate to the name of any person mentioned prominently on the page. For example, in some cases, the name may appear only in a comment section.

If you believe Google should be aware of additional information regarding this content that might result in a reversal or other change to this removal action, you can use our form at https://www.google.com/webmasters/tools/eu-privacy-webmaster. Please note that we can’t guarantee responses to submissions to that form.

The following URLs have been affected by this action:

http://svn.haxx.se/users/archive-2009-08/0808.shtml

Regards,

The Google Team

Syndicated 2014-10-12 11:56:12 from daniel.haxx.se

10 Oct 2014 zeenix   » (Journeyer)

Life update

Like many others on planet.gnome, it seems I also don't feel like posting much on my blog any more since I post almost all major events of my life on social media (or SOME, as its for some reason now known as in Finland). To be honest, the thought usually doesn't even occur to me anymore. :( Well, anyway! Here is a brief of what's been up for the last many months:
  • Got divorced. Yeah, not nice at all but life goes on! At least I got to keep my lovely cat.

  • Its been almost an year (14 days less) that I moved to London. In a way it was good that I was in a new city at the time of divorce as its an opportunity to start a new life. I made some cool new friends, mostly the GNOME gang in here.

    London has its quirks but over all I'm pretty happy to be living here. One big issue is that most of my friends are in Finland so I miss them very much. Hopefully, in time I'll also make a lot more friends in London and also my friends from Finland will visit me too.

    The best thing about London is the weather! No, I'm not joking at all. Not only its a big improvement when compared to Helsinki, the rumours about "Its always raining in London" are greatly (I can't stress on this word enough) exaggerated.
  • I got my eyes Z-LASIK'ed so no more glasses!

  • Started taking:

    • Driving lessons. Failed the first driving test today. Having known what I did wrong, I'm sure I wont repeat the same mistakes again next time and will pass.
    • Helicopter flying lessons. Yes! I'm not joking. I grew up watching Airwolf and ever since then I've been fascinated by helicopters and wanted to fly them but never got around to doing it. Its very expensive, as you'd imagine so I'm only taking two lessons a month. With this pace, I should be have my PPL(H) by end of 2015.

      Turns out that I'm very good at one thing that most people find very challenging to master: Hovering. The rest isn't hard either in practice. Theory is the biggest challenge for me. Here is the video recording of the 15 mins trial lesson I started with.

Syndicated 2014-10-10 17:53:00 (Updated 2014-10-10 18:09:31) from zeenix

10 Oct 2014 dmarti   » (Master)

Susceptible to advertising?

Something I hear a lot in discussions of online ad blocking is something like:

Ad blocker users aren't susceptible to advertising anyway.

But advertising isn't a matter of susceptability. It's not fly fishing. Advertising is based on an exchange of attention for signal. The audience pays attention, and the advertiser sends a signal of his or her intentions in the market and belief in product saleability.

Kevin Simler writes, We may not conform to a model of perfect economic behavior, but neither are we puppets at the mercy of every Tom, Dick, and Harry with a billboard. We aren't that easily manipulated.

Ad blocker users aren't the only ones who aren't "susceptible." Nobody is "susceptible." People pay attention to advertising more or less depending on how involved they are in that market, but it's a rational process.

If you go down the road of believing in "susceptible," then you get to the wrong answers. First, advertisers throw away their signaling ability by targeting users likely to click. Then users respond by blocking not just the targeted ads but by over-blocking the remaining signal-carrying ads.

Once you understand how advertising works (you did read that Kevin Simler essay?) you can get to the optimal blocking tool for yourself as a market participant: Privacy Badger, which blocks the ads that it's not rational to look at while letting non-targeted ads, with their signaling value, through.

More on this kind of thing: Targeted Advertising Considered Harmful

Syndicated 2014-10-10 15:43:07 from Don Marti

10 Oct 2014 bagder   » (Master)

internal timers and timeouts of libcurl

wall clockBear with me. It is time to take a deep dive into the libcurl internals and see how it handles timeouts and timers. This is meant as useful information to libcurl users but even more as insights for people who’d like to fiddle with libcurl internals and work on its source code and architecture.

socket activity or timeout

Everything internally in libcurl is using the multi, asynchronous, interface. We avoid blocking calls as far as we can. This means that libcurl always either waits for activity on a socket/file descriptor or for the time to come to do something. If there’s no socket activity and no timeout, there’s nothing to do and it just returns back out.

It is important to remember here that the API for libcurl doesn’t force the user to call it again within or at the specific time and it also allows users to call it again “too soon” if they like. Some users will even busy-loop like crazy and keep hammering the API like a machine-gun and we must deal with that. So, the timeouts are mostly to be considered advisory.

many timeouts

A single transfer can have multiple timeouts. For example one maximum time for the entire transfer, one for the connection phase and perhaps even more timers that handle for example speed caps (that makes libcurl not transfer data faster than a set limit) or detecting transfers speeds below a certain threshold within a given time period.

A single transfer is done with a single easy handle, which holds a list of all its timeouts in a sorted list. It allows libcurl to return a single time left until the nearest timeout expires without having to bother with the remainder of the timeouts (yet).

Curl_expire()

… is the internal function to set a timeout to expire a certain number of milliseconds into the future. It adds a timeout entry to the list of timeouts. Expiring a timeout just means that it’ll signal the application to call libcurl again. Internally we don’t have any identifiers to the timeouts, they’re just a time in the future we ask to be called again at. If the code needs that specific time to really have passed before doing something, the code needs to make sure the time has elapsed.

Curl_expire_latest()

A newcomer in the timeout team. I figured out we need this function since if we are in a state where we need to be called no later than a certain specific future time this is useful. It will not add a new timeout entry in the timeout list in case there’s a timeout that expires earlier than the specified time limit.

This function is useful for example when there’s a state in libcurl that varies over time but has no specific time limit to check for. Like transfer speed limits and the like. If Curl_expire() is used in this situation instead of Curl_expire_latest() it would mean adding a new timeout entry every time, and for the busy-loop API usage cases it could mean adding an excessive amount of timeout entries. (And there was a scary bug reported that got “tens of thousands of entries” which motivated this function to get added.)

timeout removals

We don’t remove timeouts from the list until they expire. Like for example if we have a condition that is timing dependent, then we set a timeout with Curl_expire() and we know we should be called again at the end of that time.

If we wouldn’t add the timeout and there’s no socket activity on the socket then we may not be called again – ever.

When an internal state transition into something else and we therefore don’t need a previously set timeout anymore, we have no handle or identifier to the timeout so it cannot be removed. It will instead lead to us getting called again when the timeout triggers even though we didn’t really need it any longer. As we’re having an API that allows this anyway, this is already handled by the logic and getting called an extra time is usually very cheap and is not considered a problem worth addressing.

Timeouts are removed automatically from the list of timers when they expire. Timeouts that are in passed time are removed from the list and the timers following will then get moved to the front of the queue and be used to calculate how long the single timeout should be next.

The only internal API to remove timeouts that we have removes all timeouts, used when cleaning up a handle.

many easy handles

I’ve mentioned how each easy handle treats their timeouts above. With the multi interface, we can have any amount of easy handles added to a single multi handle. This means one list of timeouts for each easy handle.

To handle many thousands of easy handles added to the same multi handle, all with their own timeout (as each easy handle only show their closest timeout), it builds a splay tree of easy handles sorted on the timeout time. It is a splay tree rather than a sorted list to allow really fast insertions and removals.

As soon as a timeout expires from one of the easy handles and it moves to the next timeout in its list, it means removing one node (easy handle) from the splay tree and inserting it again with the new timeout timer.

Syndicated 2014-10-10 06:29:38 from daniel.haxx.se

9 Oct 2014 amits   » (Journeyer)

KVM Forum 2014 Schedule

The 2014 edition of KVM Forum is less than a week away.  The schedule of the talks is available at this location.  Use this link to add the schedule to your calendar.  A few slides have already been uploaded for some of the talks.

As with last year, we’ll live-stream and record all talks, keep an eye on the wiki page for details.

One notable observation about the schedule is that it’s much relaxed from the last few years, and there are far fewer talks in parallel this time around.  There’s a lot of time for interaction / networking / socializing.  If you’re in Dusseldorf next week, please come by and say ‘hello!’

Syndicated 2014-10-09 19:34:42 (Updated 2014-10-09 19:51:08) from Think. Debate. Innovate.

9 Oct 2014 bagder   » (Master)

Coverity scan defect density: 0.00

A couple of days ago I decided to stop slacking and grab this long dangling item in my TODO list: run the coverity scan on a recent curl build again.

Among the static analyzers, coverity does in fact stand out as the very best one I can use. We run clang-analyzer against curl every night and it hasn’t report any problems at all in a while. This time I got almost 50 new issues reported by Coverity.

To put it shortly, a little less than half of them were issues done on purpose: for example we got several reports on ignored return codes we really don’t care about and there were several reports on dead code for code that are conditionally built on other platforms than the one I used to do this with.

But there were a whole range of legitimate issues. Nothing really major popped up but a range of tiny flaws that were good to polish away and smooth out. Clearly this is an exercise worth repeating every now and then.

End result

21 new curl commits that mention Coverity. Coverity now says “defect density: 0.00” for curl and libcurl since it doesn’t report any more flaws. (That’s the number of flaws found per thousand lines of source code.)

Want to see?

I can’t seem to make all the issues publicly accessible, but if you do want to check them out in person just click over to the curl project page at coverity and “request more access” and I’ll grant you view access, no questions asked.

Syndicated 2014-10-09 07:14:13 from daniel.haxx.se

8 Oct 2014 Stevey   » (Master)

Writing your own e-books is useful

Before our recent trip to Poland I took the time to create my own e-book, containing the names/addresses of people to whom we wanted to send postcards.

Authoring ebooks is simple, and this was a useful use. (Ordinarily I'd have my contacts on my phone, but I deliberately left it at home ..)

I did mean to copy and paste some notes from wikipedia about transport, tourist destinations, etc, into a brief guide. But I forgot.

In other news the toy virtual machine I hacked together got a decent series of updates, allowing you to embed it and add your own custom opcode(s) easily. That was neat, and fell out naturely from the switch to using function-pointers for the opcode implementation.

Syndicated 2014-10-08 19:03:34 from Steve Kemp's Blog

8 Oct 2014 mikal   » (Journeyer)

Lock In




ISBN: 0765375869
LibraryThing
I know I like Scalzi stuff, but each series is so different that I like them all in different ways. I don't think he's written a murder mystery before, and this book was just as good as Old Man's War, which is a pretty high bar. This book revolves around a murder being investigated by someone who can only interact with the real world via personal androids. Its different from anything else I've seen, and a unique idea is pretty rare these days.

Highly recommended.

Tags for this post: book john_scalzi robot murder mystery
Related posts: Isaac Asimov's Robot Short Stories; Prelude To Foundation ; Isaac Asimov's Foundation Series; Caves of Steel; Robots and Empire ; A Talent for War


Comment

Syndicated 2014-10-08 02:43:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

7 Oct 2014 lucasr   » (Master)

Probing with Gradle

Up until now, Probe relied on dynamic view proxies generated at runtime to intercept View calls. Although very convenient, this approach greatly affects the time to inflate your layouts—which limits the number of use cases for the library, especially in more complex apps.

This is all changing now with Probe’s brand new Gradle plugin which seamlessly generates build-time proxies for your app. This means virtually no overhead at runtime!

Using Probe’s Gradle plugin is very simple. First, add the Gradle plugin as a dependency in your build script.

buildscript {
    repositories {
        mavenCentral()
    }

    dependencies {
        classpath 'com.android.tools.build:gradle:0.13.+'
        classpath 'org.lucasr.probe:gradle-plugin:0.1.2'
    }
}

Then add Probe’s library as a dependency in your app.

repositories {
    mavenCentral()
}

dependencies {
    compile 'org.lucasr.probe:probe:0.1.2'
}

Next, apply the plugin to your app’s build.gradle.

apply plugin: 'probe'

Probe’s proxy generation is disabled by default and needs to be explicitly enabled on specific build variants (build type + product flavour). For example, this is how you enable Probe proxies in debug builds.

probe {
    buildVariants {
        debug {
            enabled = true
        }
    }
}

And that’s all! You should now be able to deploy interceptors on any part of your UI. Here’s how you could deploy an OvermeasureInterceptor in an activity.

public final class MainActivity extends Activity {
   @Override
   protected void onCreate(Bundle savedInstanceState) {
       Probe.deploy(this, new OvermeasureInterceptor());
       super.onCreate(savedInstanceState);
       setContentView(R.id.main_activity);
   }
}

While working on this feature, I have changed DexMaker to be an optional dependency i.e. you have to explicitly add DexMaker as a build dependency in your app in order to use it.

This is my first Gradle plugin. There’s definitely a lot of room for improvement here. These features are available in the 0.1.2 release in Maven Central.

As usual, feedback, bug reports, and fixes are very welcome. Enjoy!

Syndicated 2014-10-07 23:12:03 from Lucas Rocha

7 Oct 2014 Pizza   » (Master)

Further adventures with printers: The Citizen CW-01

A few days ago, someone with a Citizen CW-01 popped up on the Gutenprint mailing list. Due to its lineage, I'd assumed it (and its bretheren, the OP900) was related to the newer CW and CY families, and would work with the DS40 backend once the USB PID was known.

It turns out that the printer operates at 334dpi natively, so some additional work was needed. I'm not sure how I'd missed that. So, after some decoding of the WinXP print jobs, I discover the spool format is quite simple, and looks nothing like the newer CX/CY series.

So I ask the user to obtain some sniffs of the printer comms, and he delivered two dumps that look quite similar to the CX/CY, differing only in a couple of parameters.

So, it was pretty easy to whip up a new backend. It's out for testing now, and with luck, in a few days I'll be able to declare the CW-01 as officially supported by Gutenprint, so it'll work under Linux.

It'll be a bit more work to figure out how much of the CX/CY's status/info command set works with the CW-01, and I suspect the 600dpi support needs some more work, but for now, it's out of my hands.

In other news, another Mitsubishi CP-D70DW user popped up, sent me some detailed sniffs, and let me remote into his system for some interactive debugging; many, many bugfixes to the backend later, and it seems to be handle everything I know how to throw at it. With luck it'll also fix the CP-K60DW functionality as well.

Unfortunately, the CP-D70/D707/K60 employ a seriously screwy nonlinear tone curve/smoothing approach that I haven't been able to model, so Gutenprint's output is pretty lousy. Such is the fate of reverse-engineering efforts..

Syndicated 2014-10-07 03:13:17 from Solomon Peachy

6 Oct 2014 crhodes   » (Master)

interesting pretty-printer bug

One of SBCL’s Google Summer of Code students, Krzysztof Drewniak (no relation) just got to merge in his development efforts, giving SBCL a far more complete set of Unicode operations.

Given that this was the merge of three months’ out-of-tree work, it’s not entirely surprising that there were some hiccups, and indeed we spent some time diagnosing and fixing a 1000-fold slowdown in char-downcase. Touch wood, all seems mostly well, except that Jan Moringen reported that, when building without the :sb-unicode feature (and hence having a Lisp with 8-bit characters) one of the printer consistency tests was resulting in an error.

Tracking this down was fun; it in fact had nothing in particular to do with the commit that first showed the symptom, but had been lying latent for a while and had simply never shown up in automated testing. I’ve expressed my admiration for the Common Lisp standard before, and I’ll do it again: both as a user of the language and as an implementor, I think the Common Lisp standard is a well-executed document. But that doesn’t stop it from having problems, and this is a neat one:

When a line break is inserted by any type of conditional newline, any blanks that immediately precede the conditional newline are omitted from the output and indentation is introduced at the beginning of the next line.

(from pprint-newline)

For the graphic standard characters, the character itself is always used for printing in #\ notation---even if the character also has a name[5].

(from CLHS 22.1.3.2)

Space is defined to be graphic.

(from CLHS glossary entry for ‘graphic’)

What do these three requirements together imply? Imagine printing the list (#\a #\b #\c #\Space #\d #\e #\f) with a right-margin of 17:

  (write-to-string '(#\a #\b #\c #\Space #\d #\e #\f) :pretty t :right-margin 17)
; => "(#\\a #\\b #\\c #\\
; #\\d #\\e #\\f)"

The #\Space character is defined to be graphic; therefore, it must print as #\ rather than #\Space; if it happens to be printed just before a conditional newline (such as, for example, generated by using pprint-fill to print a list), the pretty-printer will helpfully remove the space character that has just been printed before inserting the newline. This means that a #\Space character, printed at or near the right margin, will be read back as a #\Newline character.

It’s interesting to see what other implementations do. CLISP 2.49 in its default mode always prints #\Space; in -ansi mode it prints #\ but preserves the space even before a conditional newline. CCL 1.10 similarly preserves the space; there’s an explicit check in output-line-and-setup-for-next for an “escaped” space (and a comment that acknowledges that this is a heuristic that can be wrong in the other direction). I’m not sure what the best fix for this is; it’s fairly clear that the requirements on the printer aren’t totally consistent. For SBCL, I have merged a one-line change that makes the printer print using character names even for graphic characters, if the *print-readably* printer control variable is true; it may not be ideal that print/read round-tripping was broken in the normal case, but in the case where it’s explicitly been asked for it is clearly wrong.

Syndicated 2014-10-06 20:48:14 from notes

6 Oct 2014 crhodes   » (Master)

settings for gnome-shell extension

A long, long time ago, I configured my window manager. What can I say? I was a student, with too much free time; obviously hoursdays spent learning some configuration file format and tweaking some aspect of behaviour would be repaid many times over the course of a working life. I suppose one thing it led to was my current career, so it’s probably not all a loss.

However, the direct productivity benefits almost certainly were a chimera; unfortunately, systems (hardware and software) changed too often for the productivity benefit (if any) to amortize the fixed set-up time, and so as a reaction I stopped configuring my window manager. In fact, I went the other way, becoming extremely conservative about upgrades of anything at all; I had made my peace with GNOME 2, accepting that there was maybe enough configurability and robustness in the 2.8 era or so for me not to have to change too much, and then not changing anything.

Along comes GNOME 3, and there are howls all over the Internet about the lack of friendly behaviour – “friendly” to people like me, who like lots of terminals and lots of editor buffers; there wasn’t much of an outcry from other people with more “normal” computing needs; who knows why? In any case, I stuck with GNOME 2 for a long time, eventually succumbing at the point of an inadvisable apt-get upgrade and not quite hitting the big red ABORT button in time.

So, GNOME 3. I found that I shared a certain amount of frustration with the vocal crowd: dynamic, vertically-arranged workspaces didn’t fit my model; I felt that clicking icons should generate new instances of applications rather than switch to existing instances, and so on. But, in the same timeframe, I adopted a more emacs-centric workflow, and the improvements of the emacs daemon meant that I was less dependent on particular behaviours of window manager and shell, so I gave it another try, and, with the right extensions, it stuck.

The right extensions? “What are those?” I hear you cry. Well, in common with illustrious Debian Project Leaders past, I found that a tiling extension made many of the old focus issues less pressing. My laptop is big enough, and I have enough (fixed) workspaces, that dividing up each screen between applications mostly works. I also have a bottom panel, customized to a height of 0 pixels, purely to give me the fixed number of workspaces; the overview shows them in a vertical arrangement, but the actual workspace arrangement is the 2x4 that I’m used to.

One issue with a tiled window arrangement, though, is an obvious cue to which window has focus. I have also removed all window decorations, so the titlebar or border don’t help with this; instead, a further extension to shade inactive windows helps to minimize visual distraction. And that’s where the technical part of this blog entry starts...

One of the things I do for my employer is deliver a module on Perception and Multimedia Computing. In the course of developing that module, I learnt a lot about how we see what we see, and also how digital displays work. And one of the things I learnt to be more conscious about was attention: in particular, how my attention can be drawn (for example, I can not concentrate on anything where there are animated displays, such as are often present in semi-public spaces, such as bars or London City airport.)

The shade inactive windows extension adds a brightness-reducing effect to windows without focus. So, that was definitely useful, but after a while I noticed that emacs windows with some text in error-face (bold, bright red) in them were still diverting my attention, even when they were unfocussed and therefore substantially dimmed.

So I worked on a patch to the extension to add a saturation-reducing effect in addition to reducing the brightness. And that was all very well – a classic example of taking code that almost does what you want it to do, and then maintenance-programming it into what you really want it to do – until I also found that the hard-wired time over which the effect took hold (300ms) was a bit too long for my taste, and I started investigating what it would take to make these things configurable.

Some time later, after exploring the world with my most finely-crafted google queries, I came to the conclusion that there was in fact no documentation for this at all. The tutorials that I found were clearly out-dated, and there were answers to questions on various forums whose applicability was unclear. This is an attempt to document the approach that worked for me; I make no claims that this is ‘good’ or even acceptable, but maybe there’s some chance that it will amortize the cost of the time I spent flailing about over other people wanting to customize their GNOME shell.

The first thing that something, anything with a preference needs, is a schema for that preference. In this instance, we’re starting with the shade-inactive-windows in the hepaajan.iki.fi namespace, so our schema will have a path that begins “/fi/iki/hepaajan/shade-inactive-windows”, and we're defining preferences, so let’s add “/preferences” to that.

  <?xml version="1.0" encoding="UTF-8"?>
<schemalist>
  <schema path="/fi/iki/hepaajan/shade-inactive-windows/preferences/"

a schema also needs an id, which should probably resemble the path

            id="fi.iki.hepaajan.shade-inactive-windows.preferences"

except that there are different conventions for hierarchy (. vs /).

            gettext-domain="gsettings-desktop-schemas">

and then here’s a cargo-culted gettext thing, which is probably relevant if the rest of the schema will ever be translated into any non-English language.

In this instance, I am interested in a preference that can be used to change the time over which the shading of inactie windows happens. It’s probably easiest to define this as an integer (the "i" here; other GVariant types are possible):

      <key type="i" name="shade-time">

which corresponds to the number of milliseconds

        <summary>Time in milliseconds over which shading occurs</summary>
      <description>
        The time over which the shading effect is applied, in milliseconds.
      </description>

which we will constrain to be between 0 and 1000, so that the actual time is between 0s and 1s, with a default of 0.3s:

        <range min="0" max="1000"/>
      <default>300</default>

and then there's some XML noise

      </key>
  </schema>
</schemalist>

and that completes our schema. For reasons that will become obvious later, we need to store that schema in a directory data/glib-2.0/schemas relative to our base extension directory; giving it a name that corresponds to the schema id (so fi.iki.hepaajan.shade-inactive-windows.preferences.gschema.xml in this case) is wise, but probably not essential. In order for the schema to be useful later, we also need to compile it: that’s as simple as executing glib-compile-schemas . within the schemas directory, which should produce a gschemas.compiled file in the same directory.

Then, we also need to adapt the extension in question to lookup a preference value when needed, rather than hard-coding the default value. I have no mental model of the namespacing, or other aspects of the environment, applied to GNOME shell extensions’ javascript code, so being simultaneously conservative and a javascript novice I added a top-level variable unlikely to collide with anything:

  var ShadeInactiveWindowsSettings = {};
function init() {

The extension previously didn’t need to do anything on init(); now, however, we need to initialize the settings object, including looking up our schema to discover what settings there are. But where is our schema? Well, if we’re running this extension in-place, or as part of a user installation, we want to look in data/glib-2.0/schemas/ relative to our own path; if we have performed a global installation, the schema will presumably be in a path that is already searched for by the default schema finding methods. So...

      var schemaDir = ExtensionUtils.getCurrentExtension().dir.get_child('data').get_child('glib-2.0').get_child('schemas');
    var schemaSource = Gio.SettingsSchemaSource.get_default();

    if(schemaDir.query_exists(null)) {
        schemaSource = Gio.SettingsSchemaSource.new_from_directory(schemaDir.get_path(), schemaSource, false);
    }

... we distinguish between those two cases by checking to see if we can find a data/glib-2.0/schemas/ directory relative to the extension’s own directory; if we can, we prepend that directory to the schema source search path. Then, we lookup our schema using the id we gave it, and initialize a new imports.gi.Gio.Settings object with that schema.

      var schemaObj = schemaSource.lookup('fi.iki.hepaajan.shade-inactive-windows.preferences', true);
    if(!schemaObj) {
        throw new Error('failure to look up schema');
    }
    ShadeInactiveWindowsSettings = new Gio.Settings({ settings_schema: schemaObj });
}

Then, whenever we use the shade time in the extension, we must make sure to look it up afresh:

  var shade_time = ShadeInactiveWindowsSettings.get_int('shade-time') / 1000;

in order that any changes made by the user take effect immediately. And that’s it. There’s an additional minor wrinkle, in that altering that configuration variable is not totally straightforward; dconf and gettings also need to be told where to look for their schema; that’s done using the XDG_DATA_DIRS configuration variable. For example, once the extension is installed locally, you should be able to run

  XDG_DATA_DIRS=$HOME/.local/gnome-shell/extensions/shade-inactive-windows@hepaajan.iki.fi/data:$XDG_DATA_DIRS dconf

and then navigate to the fi/iki/hepaajan/shade-inactive-windows/preferences schema and alter the shade-time preference entry.

Hooray! After doing all of that, we have wrestled things into being configurable: we can use the normal user preferences user interface to change the time over which the shading animation happens. I’m not going to confess how many times I had to restart my gnome session, insert logging code, look at log files that are only readable by root, and otherwise leave my environment; I will merely note that we are quite a long way away from the “scriptable user interface” – and that if you want to do something similar (not identical, but similar) in an all-emacs world, it might be as simple as evaluating these forms in your *scratch* buffer...

  (set-face-attribute 'default nil :background "#eeeeee")
(defvar my/current-buffer-background-overlay nil)

(defun my/lighten-current-buffer-background ()
  (unless (window-minibuffer-p (selected-window))
    (unless my/current-buffer-background-overlay
      (setq my/current-buffer-background-overlay (make-overlay 1 1))
      (overlay-put my/current-buffer-background-overlay
       'face '(:background "white")))
    (overlay-put my/current-buffer-background-overlay 'window
                 (selected-window))
    (move-overlay my/current-buffer-background-overlay
                  (point-min) (point-max))))
(defun my/unlighten-current-buffer-background ()
  (when my/current-buffer-background-overlay
    (delete-overlay my/current-buffer-background-overlay)))

(add-hook 'pre-command-hook #'my/unlighten-current-buffer-background) 
(add-hook 'post-command-hook #'my/lighten-current-buffer-background)

Syndicated 2014-10-06 19:55:13 from notes

6 Oct 2014 shlomif   » (Master)

Emma Watson’s Visit to Israel&Gaza ; “So, Who the Hell is Qoheleth?”

Here are the recent updates for Shlomi Fish’s Homepage.

  1. “Emma Watson’s Visit to Israel and Gaza” is a work-in-progress Real Person fiction screenplay which aims to bring Shalom to the turbulent Gaza Strip/Israel border:

    Waitress: I hope you’re having a good time, ah…

    EmWatson: Emma… Emma Watson!

    Waitress: Oh! I heard about you, naturally. Are you gonna threaten me with a wand? Heh!

    EmWatson: A wand… yes, the bane of my existence. I’m thinking of collecting money for a public campaign to convert the weapon most associated with me to something more menacing.

    Waitress: Don’t you have enough money for that?

    EmWatson: No, not enough! Heh. And money isn’t everything.

    Waitress: So you’re not playing in films for money?

    EmWatson: Playing in films for money? Of course not! What a preposterous idea.

    Waitress: Ah, nice.

    EmWatson: I’m playing in films for a shitload of money!

  2. “So, who the Hell is Qoheleth?” - is a new illustrated screenplay that tells what I imagine to have happened to the author of the Biblical book of Ecclesiastes / Qoheleth shortly after he has written it. The timing is appropriate because Ecclesiastes is being read during the upcoming Sukkot Jewish holiday.

    Josephus: Anyway, can you share some details about your trip? I never ventured a long way past Damascus.

    Athena: Sure! It was very interesting. Most interesting.

    Athena: We travelled with our own people and some Greek merchants, all the way to Athens, and there we hitchhiked a ride with some Assyrian merchants, hoping it will get us closer to Alexandria. There were some guards escorting us, and at one point they disarmed us and threatened us at sword’s point to have sex with them or else they'll kill us and take all our possessions.

    Josephus: Wow! Rape. So what did you do?

    Athena: Well, we consulted between ourselves and after a long while of being really scared, we calmed down a little, and decided that if we are forced to have sex, we might as well cooperate and try to enjoy it. So we told them that we’ll do it willingly and they agreed.

    Josephus: How clever of you! And then what happened.

    Athena: Well, the three of us and her lover each found their own part of the woods, and we had sex. Then, after one or two times, the three men all lost stamina, while we were not completely satisfied and cried for more!

    [ Josephus laughs. ]

    Alexis: Yes! Then we heard each other’s cries and we gathered at one place together still naked with our clothes as cover, and we bitched about the whole situation - in Greek - and the men stood there ashamed.

    Athena: Yes! Anyway, we continued as couples throughout the trip and the men got better in love making as time went by, and they also taught us a little Aramaic. Then we arrived at the junction - they wanted to go to Assyria, and we wanted to head more south, and then all the 6 of us were completely emotional and offered each other to escort them on the way, so we won’t part, but we eventually cared enough about the others to let them go on their own way.

    Josephus: Wow! That sounds like love.

    Athena: Love! Yes! That’s the word. Eros in action.

  3. A new essay A #SummerNSA’s Reading has been added for summarising the concentrated “#SummerNSA” / Summerschool at the NSA effort during the summer of 2014.

  4. There are new factoids in the Facts Collection:

    “Talk Like a Pirate Day” is the only day of the year when Chuck Norris only talks like a pirate, and does not actually act like one.

    On Yom Kippur (= the Jewish Day of Atonement), Chuck Norris forgives God for his sins.

    Chuck Norris once refactored a 10 million lines C++ program and was done by lunch time. It then took Summer Glau 5 minutes to write the equivalent Perl 10-liner.

  5. There are some new captioned images and aphorisms:

    Every mighty Klingon warrior has Watched Sesame Street

    Every Mighty Klingon warrior has watched Sesame Street!

  6. The screenplay Buffy: A Few Good Slayers has some new scenes:

    [ Faith is teaching Becky and the rest of the class how to throw knives. ]

    Faith: Becky, it’s nice that you hit the mark three times in succession, but you’re not always holding the knife correctly.

    Becky: OK, Ms. Harris. Can you show me how to do that again? [She prepares her phone.]

    Faith: OK, here goes.

    [ Cut to the bullseye - three knives hit it quickly. ]

    Faith: How´s that?

    Becky: That’s very nice, but as my mobile‘s video demonstrates, you didn’t hold the knife “correctly” (in quotes) once.

    Faith: Let me see. [She watches the video.] Oh crap.

    Faith: Becky, Becky… you have a lot of potential. You’re more than a pretty face.

    Becky: Heh, I knew that I have potential, but do you really think I have a pretty face?

    Faith: If my opinion as a straight, married, woman, matters, I think you do.

    Becky: Thanks, Ms. Harris.

    Faith: OK, class dismissed. Please try to practise at your free time, we’re going to have a test soon.

    [ The students rise up and leave. ]

Syndicated 2014-10-05 12:35:17 from shlomif

5 Oct 2014 titus   » (Journeyer)

Putting together an online presence for a diffuse academic community - how?

I would like to build a community site. Or, more precisely, I would like to recognize, collect, and collate information from an already existing but rather diffuse community.

The focus of the community will be academic data science, or "data driven discovery". This is spurred largely by the recent selection of the Moore Data Driven Discovery Investigators, as well as the earlier Moore and Sloan Data Science Environments, and more broadly by the recognition that academia is broken when it comes to data science.

So, where to start?

For a variety of reasons -- including the main practical one, that most academics are not terribly social media integrated and we don't want to try to force them to learn new habits -- I am focusing on aggregating blog posts and Twitter.

So, the main question is... how can we most easily collect and broadcast blog posts and articles via a Web site? And how well can we integrate with Twitter?

First steps and initial thoughts

Following Noam Ross's suggestions in the above storify, I put together a WordPress blog that uses the RSS Multi Importer to aggregate RSS feeds as blog posts (hosted on NFSN). I'd like to set this up for the DDD Investigators who have blogs; those who don't can be given accounts if they want to post something. This site also uses a Twitter feed plugin to pull in tweets from the list of DDD Investigators.

The resulting RSS feed from the DDDI can be pulled into a River of News site that aggregates a much larger group of feeds.

The WordPress setup was fairly easy and I'm going to see how stable it is (I assume it will be very stable, but shrug time will tell :). I'm upgrading my own hosting setup and once that's done, I'll try out River4.

Next steps and longer-term thoughts

Ultimately a data-driven-discovery site that has a bit more information would be nice; I could set up a mostly static site, post it on github, authorize a few people to merge, and then solicit pull requests when people want to add their info or feeds.

One thing to make sure we do is track only a portion of feeds for prolific bloggers, so that I, for example, have to tag a post specifically with 'ddd' to make it show up on the group site. This will avoid post overload.

I'd particularly like to get a posting set up that integrates well with how I consume content. In particular, I read a lot of things via my phone and tablet, and the ability to post directly from there -- probably via e-mail? -- would be really handy. Right now I mainly post to Twitter (and largely by RTing) which is too ephemeral, or I post to Facebook, which is a different audience. (Is there a good e-mail-to-RSS feed? Or should I just do this as a WordPress blog with the postie plug-in?)

The same overall setup could potentially work for a Software Carpentry Instructor community site, a Data Carpentry Instructor community site, trainee info sites for SWC/DC trainees, and maybe also a bioinformatics trainee info site. But I'd like to avoid anything that involves a lot of administration.

Things I want to avoid

Public forums.

Private forums that I have to administer or that aren't integrated with my e-mail (which is where I get most notifications, in the end).

Overly centralized solutions; I'm much more comfortable with light moderation ("what feeds do I track?") than anything else.


Thoughts?

--titus

Syndicated 2014-10-04 22:00:00 from Living in an Ivory Basement

5 Oct 2014 Stevey   » (Master)

Before I forget, a simple virtual machine

Before I forget I had meant to write about a toy virtual machine which I'ce been playing with.

It is register-based with ten registers, each of which can hold either a string or int, and there are enough instructions to make it fun to use.

I didn't go overboard and write a complete grammer, or a real compiler, but I did do enough that you can compile and execute obvious programs.

First compile from the source to the bytecodes:

$ ./compiler examples/loop.in

Mmm bytecodes are fun:

$ xxd  ./examples/loop.raw
0000000: 3001 1943 6f75 6e74 696e 6720 6672 6f6d  0..Counting from
0000010: 2074 656e 2074 6f20 7a65 726f 3101 0101   ten to zero1...
0000020: 0a00 0102 0100 2201 0102 0201 1226 0030  ......"......&.0
0000030: 0104 446f 6e65 3101 00                   ..Done1..

Now the compiled program can be executed:

$ ./simple-vm ./examples/loop.raw
[stdout] register R01 = Counting from ten to zero
[stdout] register R01 = 9 [Hex:0009]
[stdout] register R01 = 8 [Hex:0008]
[stdout] register R01 = 7 [Hex:0007]
[stdout] register R01 = 6 [Hex:0006]
[stdout] register R01 = 5 [Hex:0005]
[stdout] register R01 = 4 [Hex:0004]
[stdout] register R01 = 3 [Hex:0003]
[stdout] register R01 = 2 [Hex:0002]
[stdout] register R01 = 1 [Hex:0001]
[stdout] register R01 = 0 [Hex:0000]
[stdout] register R01 = Done

There could be more operations added, but I'm pleased with the general behaviour, and embedding is trivial. The only two things that make this even remotely interesting are:

  • Most toy virtual machines don't cope with labels and jumps. This does.
    • Even though it was a real pain to go patching up the offsets.
    • Having labels be callable before they're defined is pretty mandatory in practice.
  • Most toy virtual machines don't allow integers and strings to be stored in registers.
    • Now I've done that I'm not 100% sure its a good idea.

Anyway that concludes todays computer-fun.

Syndicated 2014-10-05 08:34:30 from Steve Kemp's Blog

5 Oct 2014 amits   » (Journeyer)

OpenStack Pune Meetup

I participated in the OpenStack Meetup at the Red Hat Pune office a few weekends ago.  I have been too caught up on the lower-level KVM/QEMU layers of the virt stack, and know there aren’t too many people involved in those layers in Pune (or even India); and was curious to learn more about OpenStack and also find out more about the OpenStack community in Pune.  The event was on a Saturday, which means sacrificing one day of rest and relaxation – but I went along because curiousity got the better of me.

This was a small, informal event where we had a few talks and several hallway discussions.  Praveen has already blogged about his experiences, here are my notes about the meetup.

There were a few scheduled talks for the day; speakers nominated themselves on the meetup page and the event organizers allotted slots for them.  The proceedings started off with configuring and setting up OpenStack via DevStack.  I wished (for the audience present there) there would’ve been an introductory talk before a deep-dive into DevStack.  I could spot a few newbies in the crowd, and they would have benefitted by an intro.

In a few discussions with the organizers, I learnt one of their pain points for such meetups: there inevitably are newbies at each meetup, and they can’t move on to advanced topics just because they have to start from scratch for each meetup.  I suggested they have a clear focus for each meetup: tell explicitly what each meetup is about, and the expertise level that’s going to be assumed.  For example, there’s nothing wrong with a newbie-focused event; but then some other event could focus on the networking part of OpenStack, and they assume people are familiar with configuring and deploying openstack and are familiar with basic networking priciples.  This suggestion is based on the Pune FADs we want to conduct and have in the pipeline; and was welcomed by the organizers.

Other talks followed; and I noticed a trend: not many people understood, or even knew about, the lower layers that make up the infrastructure beneath OpenStack.  I asked the organizers if they could spare 10 mins for me to provide a peek into the lower levels, and they agreed.  Right after a short working-lunch break, I took the stage.

I spoke about Linux, KVM and QEMU; dove into details of how each of them co-operate and how libvirt drives the interactions between the upper layers and the lower layers.  Also spoke a little about the alternative hypervisor support that libvirt has, but the advantages of the default hypervisor, QEMU/KVM has over others.  I then spoke about how improvements in Linux in general (e.g. the memory management layer) benefits the thousands of people running Linux, the thousands people running the KVM hypervisor, and in effect, benefit all the OpenStack deployments.  I then mentioned a bit about how features flow from upstream into distributions, and how all the advantages trickle down naturally, without anyone having to bother about particular parts of the infrastructure.

The short talk was well received, and judging by the questions I got asked, it was apparent that some people didn’t know the dynamics involved, and the way I presented it was very helpful to them and they wanted to learn more.  I also got asked a few hypervisor comparison questions.  I had to cut the interaction because I easily overflowed the 15 mins allotted to me, and asked people to follow up with me later, which several did.

One of the results of all those conversations was that I got volunteered to do more in-depth talks on the topic at future meetups.  The organizers lamented there’s a dearth of such talks and subject-matter experts; and many meetups generally end up being just talks from people who have read or heard about things rather than real users or implementers of the technology.  They said they would like to have more people from Red Hat talking about the work we do upstream and all the contributions we make.  I’m just glad our contributions are noticed :-)

Another related topic that came up during discussions with the organizers are hackathons, and getting people to contribute and actually do stuff.  I expect a hackathon to be proposed soon.

I had a very interesting conversation with Sajid, one of the organizers.  He mentioned Reliance Jio are setting up data centres across India, and are going to launch cloud computing services with their 4G rollout.  Their entire infrastructure is based on OpenStack.

There were other conversations as well, but I’ll perhaps talk about them in other posts.

Internally at Red Hat, we had a few discussions on how to improve our organization for such events (even though they’re community events; we should be geared up to serve the attendees better).  Mostly included stuff around making it easier to get people in (ie working with security), getting the AV equipment in place, etc.  All of this was working fine during this event, but basically ensuring all of the things that do go right are also part of the list of things to look at while organizing events so we don’t slip up.

Syndicated 2014-10-05 07:09:10 from Think. Debate. Innovate.

4 Oct 2014 marnanel   » (Journeyer)

multipart

Today I received an email from someone who said they'd attached a file I needed, but I couldn't see the attachment. After some digging, I found that the message was structured like this:

multipart/alternative: (i.e. "these are alternative versions of the same thing")
-- text/plain (a version of the message in plain text)
-- multipart/related: (i.e. "these parts belong together")
-- -- text/html (a version of the message in HTML)
-- -- the attachment

So if your email program shows HTML for preference, you would see the attachment, but if it shows plain text for preference (as mine does), you wouldn't. Of course it *should* have been structured like this:

multipart/related: (i.e. "these parts belong together")
-- multipart/alternative: (i.e. "these are alternative versions of the same thing")
-- -- text/plain (a version of the message in plain text)
-- -- text/html (a version of the message in HTML)
-- the attachment

This entry was originally posted at http://marnanel.dreamwidth.org/313678.html. Please comment there using OpenID.

Syndicated 2014-10-04 21:39:27 (Updated 2014-10-04 22:02:50) from Monument

4 Oct 2014 benad   » (Apprentice)

Eventual Consistency, Squared

Looking back at my article "The Syncing Problem", implementing a generic DVCS seems like a relatively straightforward solution. Actually, if the "data to sync" was simplified to plain text, existing DVCS like git or Mercurial may be sufficient. But there is a fundamental problem I glossed over that has huge ramifications on the design of the DVCS that make existing DVCS implementations dangerous to use.

In modern "Internet-connected" appliances, there are two storage solutions: On-device, and "in the cloud". It is the "cloud" storage that is going to be used for the devices to communicate to each other indirectly when performing data synchronization. There is though a huge behavioural difference between on-device and cloud storage: The cloud storage is "eventually consistent". Beneath its API, the cloud storage itself may also be distributed across machines, and modified data can take a little while to propagate to other machines. Essentially, if you upload a file from one device, it may take a little while for another device to see the change.

Sadly, whatever conflict resolution used by a cloud storage provide is unreliable, because their behaviour is either undocumented or inconsistent. Locking files on such storage may not be possible either. Worse, internal synchronization issues at the cloud provider may make their internal synchronization speed so inconsistent to make it unreliable as a means to communicate information between devices quickly. Almost all VCS (distributed or not) assume a reliable storage area for the version repository. Hosted VCS guarantee ACID. No DVCS was made to push revision information to an unreliable storage, and use that as the primary means to exchange information.

The easiest solution for this is to design a DVCS that supports "write-only" repositories. If the storage key (file name) contains the checksum of the data it contains, it may be possible to have multiple clients writing to the same storage area changes to the shared repository. Even if listing available "data blocks" is inconsistent on the shared storage, all it can do is augment the "knowledge" of what exists in the repository against the repository stored on local storage. The atomicity of the storage blocks should be as close as possible to the atomicity of a transactional version control delta, especially since the cloud storage may make information appear on other devices out-of-order of how they were written. That could make those "patch files" larger than well-optimized VCS, but on cloud storage we may not have any other option.

Sure, a write-only repository may be a big issue if the files in the version control are too large or if storage is limited, but then most VCS tend to avoid deleting historical data, and when they support "cleaning up a repository", the solutions are clumsy and error-prone. In our case, if a device prematurely deletes older historical data in the shared storage, by being unaware that other devices were synced at older versions and may branch from there, then this would be tantamount to using the share storage to host only the latest version and nothing else. All to say, deleting historical data in a shared, eventually-consistent storage is a difficult problem that may involve a lot of tuning based on how long devices can stay unsynchronized before considering them "lost", compared to how fast the cloud storage is expected to be consistent.

Syndicated 2014-10-04 15:05:19 from Benad's Blog

4 Oct 2014 Stevey   » (Master)

Kraków was nice

We returned safely from Kraków, despite a somewhat turbulent flight home.

There were many pictures taken, but thus far I've only posted a random night-time shot. Perhaps more will appear in the future.

In other news I've just made a new release of the chronicle blog compiler, So 5.0.7 should shortly appear on CPAN.

The release contains a bunch of minor fixes, and some new facilities relating to templates.

It seems likely that in the future there will be the ability to create "static pages" along with the blog-entries, tag-clouds & etc. The suggestion was raised on the github issue tracker and as a proof of concept I hacked up a solution which works entirely via the chronicle plugin-system, proving that the new development work wasn't a waste of time - especially when combined with the significant speedups in the new codebase.

(ObRandom: Mailed the Debian package-mmaintainer to see if there was interest in changing. Also mailed a couple of people I know who are using the old code to see if they had comments on the new code, or had any compatibility issues. No replies from either, yet. *shrugs*)

Syndicated 2014-10-04 12:20:45 from Steve Kemp's Blog

3 Oct 2014 prla   » (Apprentice)

Having just recently acquired a brand new Toshiba Satellite Z30-A-130 laptop and after booting Ubuntu 14.04 on it, one thing I immediately wanted to disable was the touchpad. It seems way too sensible and I keep touching while typing, taking the focus away. Turns out it wasn't super easy, at least until I figured it out.

The first obvious idea was to check the System Settings panel for Keyboard and Mouse hoping for a "disable touchpad" option but none was to be found on my fresh Ubuntu install.

$ xinput list

...didn't show the touchpad entry either, even though it was working alright. The idea was to disable it via CLI:

$ xinput set-prop [device number] "Device Enabled" 0

But having no device and hence no device number, I couldn't do it either. The final bit that led me in the right direction was:

$ synclient -l
Couldn't find synaptics properties. No synaptics driver loaded?


So, a bit more poking around and it turns out linux-kernel 3.13 may not be the best bet for driving these touchpads. Someone reported success with 3.17 and it does have indeed much better support. My days of recompiling infinite numbers of kernels everyday (by hand, mind you) were long gone and it's been a good few years since I last had the need to try a different kernel. But I quickly learned that Ubuntu has a kernel team and they wonderfully package every release in all .deb glory. They even package the -low-latency variant too, which I immediately chose over the -generic one. :)

$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc7-utopic/linux-headers-3.17.0-031700rc7_3.17.0-031700rc7.201409281835_all.deb
$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc7-utopic/linux-headers-3.17.0-031700rc7-lowlatency_3.17.0-031700rc7.201409281835_amd64.deb
$ wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.17-rc7-utopic/linux-image-3.17.0-031700rc7-lowlatency_3.17.0-031700rc7.201409281835_amd64.deb
$ sudo dpkg -i *.deb
$ sudo update-grub


And presto. One reboot later and I now have full control over the touchpad, even from the system settings which makes sense as all that was missing was a proper synaptics driver.

Update 2014-10-04: Turns out linux 3.17-rc7, despite fixing the synaptics driver issue, is a bit quirky when it comes to the wifi driver, as the connection suddenly drops and won't reconnect. -rc1 seems to be OK and it's the first version to feature synaptics support for my touchpad, so that's what I'll be using for the time being.

3 Oct 2014 prla   » (Apprentice)

Interesting how having a couple of technical notes to jot down immediately brought this place here back to mind. Even more interesting, at least for me, to look back at the past few entries, how they always seem to be written in pivotal moments in my personal (and professional) life. This present one doesn't deviate from that. More on that (perhaps) later.

So after falling off the end of the contract with my previous employer, I now find myself unemployed and looking forward to the next step. That also meant handing in my work laptop and considering how archaic my own Macbook Pro now seems to be, I felt it was the time to splash some hard-earned money on a new machine. Having had a regular Intel machine for the past few months at work - a Dell Latitude laptop as it was - that meant another stint using (Ubuntu flavored) Linux both for work and fun. Having lived quite a few years (almost a decade) in Macintosh land, I didn't entirely leave the Unix world but it's just not the same thing, from a developer's point of view. I guess I was still pining for the days of Slackware Linux and how tight a grip I had over everything that I thoroughly enjoyed going back to a scenario very much resembling it.

Faced with the task of acquiring a new machine, this meant going back to a Mac was out of the question for now, the plan being to extend my stay in the familiar surroundings of Linux. The work laptop had an Intel Core i7 quad-core processor and, most importantly, an SSD hd. Meaning: fast. Some shop browsing both on and offline yielded my current machine, under 1k€, a 13.3" Toshiba Satellite Z30-A-130, boasting a lesser Core i5 cpu but featuring a 256GB SSD hd. Getting used to it, the natural ongoing process of adaptation, but very happy so far with it.

Which leads us to the next post...

2 Oct 2014 mjg59   » (Master)

Actions have consequences (or: why I'm not fixing Intel's bugs any more)

A lot of the kernel work I've ended up doing has involved dealing with bugs on Intel-based systems - figuring out interactions between their hardware and firmware, reverse engineering features that they refuse to document, improving their power management support, handling platform integration stuff for their GPUs and so on. Some of this I've been paid for, but a bunch has been unpaid work in my spare time[1].

Recently, as part of the anti-women #GamerGate campaign[2], a set of awful humans convinced Intel to terminate an advertising campaign because the site hosting the campaign had dared to suggest that the sexism present throughout the gaming industry might be a problem. Despite being awful humans, it is absolutely their right to request that a company choose to spend its money in a different way. And despite it being a dreadful decision, Intel is obviously entitled to spend their money as they wish. But I'm also free to spend my unpaid spare time as I wish, and I no longer wish to spend it doing unpaid work to enable an abhorrently-behaving company to sell more hardware. I won't be working on any Intel-specific bugs. I won't be reverse engineering any Intel-based features[3]. If the backlight on your laptop with an Intel GPU doesn't work, the number of fucks I'll be giving will fail to register on even the most sensitive measuring device.

On the plus side, this is probably going to significantly reduce my gin consumption.

[1] In the spirit of full disclosure: in some cases this has resulted in me being sent laptops in order to figure stuff out, and I was not always asked to return those laptops. My current laptop was purchased by me.

[2] I appreciate that there are some people involved in this campaign who earnestly believe that they are working to improve the state of professional ethics in games media. That is a worthy goal! But you're allying yourself to a cause that disproportionately attacks women while ignoring almost every other conflict of interest in the industry. If this is what you care about, find a new way to do it - and perhaps deal with the rather more obvious cases involving giant corporations, rather than obsessing over indie developers.

For avoidance of doubt, any comments arguing this point will be replaced with the phrase "Fart fart fart".

[3] Except for the purposes of finding entertaining security bugs

comment count unavailable comments

Syndicated 2014-10-02 16:40:29 from Matthew Garrett

2 Oct 2014 Skud   » (Master)

Read this interview with me about leading AdaCamp Berlin and Bangalore

As I mentioned earlier today, I’m off to Europe shortly for AdaCamp Berlin, then in November I’m going to India for AdaCamp Bangalore. I’ll be leading both events, which means I get to welcome everyone and set the stage for the unconference, make sure the sessions and workshops run smoothly, and that the culture of AdaCamp meets its usual high standards.

The Ada Initiative just posted this announcement and interview where I talk a bit about my experience with AdaCamp, running various community events, and what I’ll bring to these ones.

Syndicated 2014-10-02 11:52:33 from Infotropism

2 Oct 2014 Skud   » (Master)

Travels: London and Berlin (Oct 7th-20th, ish)

I haven’t mentioned this on here yet so I thought I’d better do so before I actually, you know, board the plane.

I’m heading over to Europe next week and the week after. The main reason I’m going is AdaCamp in Berlin, which I will be helping run, but before and after that I’ll also be spending some time in the UK and running this Growstuff event, to get stuck into some serious code with some of our UK-based developers, in London on Oct 18-19.

If you are in the UK and are interested in food innovation, open data, technology for social good, sustainability, inclusive open source projects, or related fields, I would love to meet you! If you can’t make it to the Growstuff code sprint but would like to catch up for a coffee or something, drop me a line.

Syndicated 2014-10-02 04:29:13 from Infotropism

1 Oct 2014 danstowell   » (Journeyer)

Carpenters Estate - Is it viable or not?

Newham Council has handled the current Carpenters Estate protest shockingly badly. Issuing a press release describing the protesting mothers as "agitators and hangers-on" is just idiotically bad handling.

BUT they have also described Carpenters Estate as not "viable", and many commentators (such as Zoe Williams, Russell Brand) have lampooned them for it. After all, they can see the protesting mothers occupying a perfectly decent-looking little home. How can it be not "viable"?

Are they judging viability compared against the market rate for selling off the land? That's what Zoe Williams says, and that's what I assumed too from some conversations. But that's not it at all.

Newham's current problem with the Carpenters Estate is basically caused by the two different types of housing stock on the estate:

  • They have some tall old tower blocks which housed many hundreds of people, but they can't renovate them to a basic decent standard - the council can't afford to do it themselves and the leaseholders couldn't afford to shoulder the costs. (In council reports it's been calculated that the renovation cost per flat would cost more than the value of the flat itself - which means that the private leaseholders totally wouldn't be able to get a mortgage for the renovations.)
  • All the little two-storey houses next to the tower blocks are basically viable, at least in the sense that they should be easy to refurbish. However, they can't just leave people in those houses if they intend to demolish the tower blocks. I'm no expert in demolition but I assume it'd be impossible to demolish the 23-storey block next door while keeping the surrounding houses safe, and that's why Doran Walk is also slated for demolition.

So "not viable" means they can't find any way to refurbish those tower blocks to basic living standards - especially not in the face of the Tory cuts to council budgets - and that affects the whole estate as well as just the tower blocks. This appears to be the fundamental reason they're "decanting" people, in order to demolish and redevelop the whole place. (Discussed eg in minutes from 2012.) It's also the reason they have a big PR problem right now, because those two-storey houses appear "viable" and perfectly decent homes, yet they do indeed have a reason to get everyone out of them!

After the UCL plan for Carpenters Estate fell through it's understandable that they're still casting around for development plans, and we might charitably assume the development plans would be required to include plenty of social housing and affordable housing. You can see from the council minutes that they do take this stuff seriously when they approve/reject plans.

(Could the council simply build a whole new estate there, develop a plan itself, without casting around for partners? Well yes, it's what councils used to do before the 1980s. It's not their habit these days, and there may be financial constraints that make it implausible, but in principle I guess it must be an option. Either way, that doesn't really affect the question of viability, which is about the current un-demolished estate.)

But the lack of a plan has meant that there's no obvious "story" of what's supposed to be happening with the estate, which just leaves space for people to draw their own conclusions. I don't think anyone's deliberately misrepresenting what the council means when they talk about viability. I think the council failed badly in some of its early communication, and that led to misunderstandings that fed too easily into a narrative of bureaucratic excuses.

Syndicated 2014-10-01 16:15:10 (Updated 2014-10-01 16:35:26) from Dan Stowell

1 Oct 2014 mdz   » (Master)

Join me in supporting The Ada Initiative

When I first read that Linux kernel developer Valerie Aurora would be changing careers to work full-time on behalf of women in open source communities, I never imagined it would lead so far so fast. Today, The Ada Initiative is a non-profit organization with global reach, whose programs have helped create positive change for women in a wide range of communities beyond open source. Building on this foundation, imagine how much more they can do in the next four years! That’s why I’m pledging my continuing support, and asking you to join me.

For the next 7 days, I will personally match your donations up to $4,096. My employer, Heroku (Salesforce.com), will match my donations too, so every dollar you contribute will be tripled!

My goal is that together we will raise over $12,000 toward The Ada Initiative’s 2014 fundraising drive.

Donate now

Since about 1999, I had been working in open source communities like Debian and Ubuntu, where women are vastly underrepresented even compared to the professional software industry. Like other men in these communities, I had struggled to learn what I could do to change this. Such a severe imbalance can only be addressed by systemic change, and I hardly knew where to begin. I worked to raise awareness by writing and speaking, and joined groups like Debian Women, Ubuntu Women and Geek Feminism. I worked on my own bias and behavior to avoid being part of the problem myself. But it never felt like enough, and sometimes felt completely hopeless.

Perhaps worst of all, I saw too many women burning out from trying to change the system. It was often taxing just to participate as a woman in a male-dominated community, and the extra burden of activism seemed overwhelming. They were all volunteers, doing this work in evenings and weekends around work or study, and it took a lot of time, energy and emotional reserve to deal with the backlash they faced for speaking out about sexism. Valerie Aurora and Mary Gardiner helped me to see that an activist organization with full-time staff could be part of the solution. I joined the Ada Initiative advisory board in February 2011, and the board of directors in April.

founders_laughing

Today, The Ada Initiative is making a difference not only in my community, but in my workplace as well. When I joined Heroku in 2012, none of the engineers were women, and we clearly had a lot of work to do to change that. In 2013, I attended AdaCamp SF along with my colleague Peter van Hardenberg, joining the first “allies track”, open to participants of any gender, for people who wanted to learn the skills to support the women around them. We’ve gone on to host two ally skills workshops of our own for Heroku employees, one taught by Ada Initiative staff and another by a member of our team, security engineer Leigh Honeywell. These workshops taught interested employees simple, everyday ways to take positive action to challenge sexism and create a better workplace for women. The Ada Initiative also helped us establish a policy for conference sponsorship which supports our gender diversity efforts. Today, Heroku engineering includes about 10% women and growing. The Ada Initiative’s programs are helping us to become the kind of company we want to be.leigh-eeoc-ally-skills-workshop

I attended the workshop with a group of Heroku colleagues, and it was a powerful experience to see my co-workers learning tactics to support women and intervene in sexist situations. Hearing them discuss power and privilege in the workplace, and the various “a-ha!” moments people had, were very encouraging and made me feel heard and supported.
– Leigh Honeywell

If you want to see more of these programs from The Ada Initiative, please contribute now:
Donate now


Syndicated 2014-10-01 16:30:23 from We'll see | Matt Zimmerman

1 Oct 2014 bagder   » (Master)

Good bye Rockbox

I’m officially not taking part in anything related to Rockbox anymore. I’ve unsubscribed and I’m out.

In the fall of 2001, my friend Linus and my brother Björn had both bought the portable Archos Player, a harddrive based mp3 player and slightly underwhelmed by its firmware decided they would have a go at trying to improve it. All three of us had been working with embedded systems for many years already and I was immediately attracted to the idea of reverse engineering this kind of device and try to improve it. It sounded like a blast to me.

In December 2001 we had the first test program actually running on the device and flashing a led. The first little step of what would become a rather big effort. We wrote a GPLed mp3 player firmware replacement, entirely from scratch without re-using any original parts. A full home-grown tiny multitasking operating system with a UI.

Fast-forwarding through history: we managed to get a really good firmware done for the early Archos players and we managed to move on to follow-up mp3 players too. After a decade or so, we supported well over 60 different mp3 player models and we played every music format known to man, we usually had better battery life than the original firmwares. We could run doom and we had a video player, a plugin system and a system full of crazy things.

We gathered large amounts of skilled and intelligent hackers from all over the world who contributed to make this possible. We had yearly meetups, or developer conferences, and we hung out on IRC every day of the week. I still hang out on our off-topic IRC channel!

Over time, smart phones emerged as the preferred devices people would use to play music while on the go. We ported Rockbox over to Android as an app, but our pixel-based UI was never really suitable for the flexible Android world and I also think that most contributors were more interested in hacking devices than writing Android apps. The app never really attracted many users or developers so while functional it never “took off”.

mp3 players are now already a thing of the past and will soon fall into the cave of forgotten old things our children will never even know or care about.

Developers and users of Rockbox have mostly moved on to other ventures. I too stopped actually contributing to the project several years ago but I was running build clients for a long while and I’ve kept being subscribed to the development mailing list. Until now. I’m now finally cutting off the last rope. Good bye Rockbox, it was fun while it lasted. I had a massive amount of great fun and I learned a lot while in the project.

Rockbox

Syndicated 2014-10-01 08:53:48 from daniel.haxx.se

1 Oct 2014 mikal   » (Journeyer)

On layers

There's been a lot of talk recently about what we should include in OpenStack and what is out of scope. This is interesting, in that many of us used to believe that we should do ''everything''. I think what's changed is that we're learning that solving all the problems in the world is hard, and that we need to re-focus on our core products. In this post I want to talk through the various "layers" proposals that have been made in the last month or so. Layers don't directly address what we should include in OpenStack or not, but they are a useful mechanism for trying to break up OpenStack into simpler to examine chunks, and I think that makes them useful in their own right.

I would address what I believe the scope of the OpenStack project should be, but I feel that it makes this post so long that no one will ever actually read it. Instead, I'll cover that in a later post in this series. For now, let's explore what people are proposing as a layering model for OpenStack.

What are layers?

Dean Troyer did a good job of describing a layers model for the OpenStack project on his blog quite a while ago. He proposed the following layers (this is a summary, you should really read his post):

  • layer 0: operating system and Oslo
  • layer 1: basic services -- Keystone, Glance, Nova
  • layer 2: extended basics -- Neutron, Cinder, Swift, Ironic
  • layer 3: optional services -- Horizon and Ceilometer
  • layer 4: turtles all the way up -- Heat, Trove, Moniker / Designate, Marconi / Zaqar


Dean notes that Neutron would move to layer 1 when nova-network goes away and Neutron becomes required for all compute deployments. Dean's post was also over a year ago, so it misses services like Barbican that have appeared since then. Services are only allowed to require services from lower numbered layers, but can use services from higher number layers as optional add ins. So Nova for example can use Neutron, but cannot require it until it moves into layer 1. Similarly, there have been proposals to add Ceilometer as a dependency to schedule instances in Nova, and if we were to do that then we would need to move Ceilometer down to layer 1 as well. (I think doing that would be a mistake by the way, and have argued against it during at least two summits).

Sean Dague re-ignited this discussion with his own blog post relatively recently. Sean proposes new names for most of the layers, but the intent remains the same -- a compute-centric view of the services that are required to build a working OpenStack deployment. Sean and Dean's layer definitions are otherwise strongly aligned, and Sean notes that the probability of seeing something deployed at a given installation reduces as the layer count increases -- so for example Trove is way less commonly deployed than Nova, because the set of people who want a managed database as a service is smaller than the set of of people who just want to be able to boot instances.

Now, I'm not sure I agree with the compute centric nature of the two layers proposals mentioned so far. I see people installing just Swift to solve a storage problem, and I think that's a completely valid use of OpenStack and should be supported as a first class citizen. On the other hand, resolving my concern with the layers model there is trivial -- we just move Swift to layer 1.

What do layers give us?

Sean makes a good point about the complexity of OpenStack installs and how we scare away new users. I agree completely -- we show people our architecture diagrams which are deliberately confusing, and then we wonder why they're not impressed. I think we do it because we're proud of the scope of the thing we've built, but I think our audiences walk away thinking that we don't really know what problem we're trying to solve. Do I really need to deploy Horizon to have working compute? No of course not, but our architecture diagrams don't make that obvious. I gave a talk along these lines at pyconau, and I think as a community we need to be better at explaining to people what we're trying to do, while remembering that not everyone is as excited about writing a whole heap of cloud infrastructure code as we are. This is also why the OpenStack miniconf at linux.conf.au 2015 has pivoted from being a generic OpenStack chatfest to being something more solidly focussed on issues of interest to deployers -- we're just not great at talking to our users and we need to reboot the conversation at community conferences until its something which meets their needs.


We intend this diagram to amaze and confuse our victims


Agreeing on a set of layers gives us a framework within which to describe OpenStack to our users. It lets us communicate the services we think are basic and always required, versus those which are icing on the cake. It also let's us explain the dependency between projects better, and that helps deployers work out what order to deploy things in.

Do layers help us work out what OpenStack should focus on?

Sean's blog post then pivots and starts talking about the size of the OpenStack ecosystem -- or the "size of our tent" as he phrases it. While I agree that we need to shrink the number of projects we're working on at the moment, I feel that the blog post is missing a logical link between the previous layers discussion and the tent size conundrum. It feels to me that Sean wanted to propose that OpenStack focus on a specific set of layers, but didn't quite get there for whatever reason.

Next Monty Taylor had a go at furthering this conversation with his own blog post on the topic. Monty starts by making a very important point -- he (like all involved) both want the OpenStack community to be as inclusive as possible. I want lots of interesting people at the design summits, even if they don't work directly on projects that OpenStack ships. You can be a part of the OpenStack community without having our logo on your product.

A concrete example of including non-OpenStack projects in our wider community was visible at the Atlanta summit -- I know for a fact that there were software engineers at the summit who work on Google Compute Engine. I know this because I used to work with them at Google when I was a SRE there. I have no problem with people working on competing products being at our summits, as long as they are there to contribute meaningfully in the sessions, and not just take from us. It needs to be a two way street. Another concrete example is Ceph. I think Ceph is cool, and I'm completely fine with people using it as part of their OpenStack deploy. What upsets me is when people conflate Ceph with OpenStack. They are different. They're separate. And that is fine. Let's just not confuse people by saying Ceph is part of the OpenStack project -- it simply isn't because it doesn't fall under our governance model. Ceph is still a valued member of our community and more than welcome at our summits.

Do layers help us work our what to focus OpenStack on for now? I think they do. Should we simply say that we're only going to work on a single layer? Absolutely not. What we've tried to do up until now is have OpenStack be a single big thing, what we call "the integrated release". I think layers gives us a tool to find logical ways to break that thing up. Perhaps we need a smaller integrated release, but then continue with the other projects but on their own release cycles? Or perhaps they release at the same time, but we don't block the release of a layer 1 service on the basis of release critical bugs in a layer 4 service?

Is there consensus on what sits in each layer?

Looking at the posts I can find on this topic so far, I'd have to say the answer is no. We're close, but we're not aligned yet. For example, one proposal has a tweak to the previously proposed layer model that adds Cinder, Designate and Neutron down into layer 1 (basic services). The author argues that this is because stateless cloud isn't particularly useful to users of OpenStack. However, I think this is wrong to be honest. I can see that stateless cloud isn't super useful by itself, but we are assuming that OpenStack is the only piece of infrastructure that a given organization has. Perhaps that's true for the public cloud case, but the vast majority of OpenStack deployments at this point are private clouds. So, you're an existing IT organization and you're deploying OpenStack to increase the level of flexibility in compute resources. You don't need to deploy Cinder or Designate to do that. Let's take the storage case for a second -- our hypothetical IT organization probably already has some form of storage -- a SAN, or NFS appliances, or something like that. So stateful cloud is easy for them -- they just have their instances mount resources from those existing storage pools like they would any other machine. Eventually they'll decide that hand managing that is horrible and move to Cinder, but that's probably later once they've gotten through the initial baby step of deploying Nova, Glance and Keystone.

The first step to using layers to decide what we should focus on is to decide what is in each layer. I think the conversation needs to revolve around that for now, because it we drift off into whether existing in a given layer means you're voted off the OpenStack island, when we'll never even come up with a set of agreed layers.

Let's ignore tents for now

The size of the OpenStack "tent" is the metaphor being used at the moment for working out what to include in OpenStack. As I say above, I think we need to reach agreement on what is in each layer before we can move on to that very important conversation.

Conclusion

Given the focus of this post is the layers model, I want to stop introducing new concepts here for now. Instead let me summarize where I stand so far -- I think the layers model is useful. I also think the layers should be an inverted pyramid -- layer 1 should be as small as possible for example. This is because of the dependency model that the layers model proposes -- it is important to keep the list of things that a layer 2 service must use as small and coherent as possible. Another reason to keep the lower layers as small as possible is because each layer represents the smallest possible increment of an OpenStack deployment that we think is reasonable. We believe it is currently reasonable to deploy Nova without Cinder or Neutron for example.

Most importantly of all, having those incremental stages of OpenStack deployment gives us a framework we have been missing in talking to our deployers and users. It makes OpenStack less confusing to outsiders, as it gives them bite sized morsels to consume one at a time.

So here are the layers as I see them for now:

  • layer 0: operating system, and Oslo
  • layer 1: basic services -- Keystone, Glance, Nova, and Swift
  • layer 2: extended basics -- Neutron, Cinder, and Ironic
  • layer 3: optional services -- Horizon, and Ceilometer
  • layer 4: application services -- Heat, Trove, Designate, and Zaqar


I am not saying that everything inside a single layer is required to be deployed simultaneously, but I do think its reasonable for Ceilometer to assume that Swift is installed and functioning. The big difference here between my view of layers and that of Dean, Sean and Monty is that I think that Swift is a layer 1 service -- it provides basic functionality that may be assumed to exist by services above it in the model.

I believe that when projects come to the Technical Committee requesting incubation or integration, they should specify what layer they see their project sitting at, and the justification for a lower layer number should be harder than that for a higher layer. So for example, we should be reasonably willing to accept proposals at layer 4, whilst we should be super concerned about the implications of adding another project at layer 1.

In the next post in this series I'll try to address the size of the OpenStack "tent", and what projects we should be focussing on.

Tags for this post: openstack kilo technical committee tc layers
Related posts: My candidacy for Kilo Compute PTL; Juno TC Candidacy; Juno nova mid-cycle meetup summary: nova-network to Neutron migration; Juno Nova PTL Candidacy; Juno nova mid-cycle meetup summary: scheduler; Juno nova mid-cycle meetup summary: ironic

Comment

Syndicated 2014-09-30 18:57:00 from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

1 Oct 2014 Skud   » (Master)

Why I just stopped using IM (hint: fucking Google)

tl;dr – if we usually talk on IM/GTalk you won’t see me around any more. Use IRC, email, or other mechanisms (listed at bottom of this post) to contact me.


Background: Google stopped supporting open standards for IM a few years ago.

Other background: when I changed my name in 2011 I grabbed a GMail account with that name, just in case it would be useful. I didn’t use it, though — instead I forwarded any mail from it to my actual email address, the one I’ve had since the turn of the century: skud@infotrope.net, and set that address as my default for everything I could find.

Unfortunately Google didn’t honour those preferences, and kept exposing my unused GMail address to people. When I signed up for Google Groups, it would be exposed. When I shared Google Docs, it would be exposed. I presume it was being exposed all kinds of other ways, too, because people kept seeing my GMail address and thinking it was the right way to contact me. So in addition to the forwarding I also set up a vacation reminder telling anyone who emailed me there to use my actual address and not to use the Google one.

But Google wasn’t done yet. They kept dropping stuff into my GMail account and not forwarding it. Comments on Google docs. Invitations. Administrative notices. IM logs that I most definitely did not want archived. These were all piling up silently in an account I never logged into.

Eventually, after I missed out on several messages from a volunteer offering to help with Growstuff, I got fed up and found out how to completely delete a GMail account. I did this few weeks ago.

Fast forward to last night, when my Internet connection flaked out right before I went to bed. I looked at all my disconnected, blank windows, shrugged, and crashed for the night. This morning, everything was better and all my apps set about reconnecting.

Except that Adium, the app I use for instant messaging, was asking me for the GTalk password for skud@infotrope.net. Weird, I thought, but I had the password saved in my keychain and resubmitted it. Adium, or more properly GTalk, didn’t like it. I tried a few more times, including resetting my app password (I use two-factor auth). No luck.

Eventually I found the problem. Via this Adium bug report I learned that a GMail account is required to use GTalk. Even if you don’t use (and have never used) your GMail address to login to it, and don’t give people a GMail address to add you as a contact.

So, my choices at this point are:

  1. Sign up again for GMail, continue to have an unused and unwanted email address exposed to the public, miss important messages, and risk security/privacy problems with archiving of stuff I don’t want archived; or,
  2. Set up Jabber/XMPP, which will take a fair amount of messing around (advice NOT wanted, I know what is involved), and which will only let me talk to friends who don’t use GMail/GTalk (a small minority); or,
  3. Not be available on IM.

For now I am going with option 3. If you are used to talking to me via IM at my skud@infotrope.net address, you can now contact me as follows.

IRC: I am Skud on irc.freenode.net and on some other specialist networks. On Freenode I habitually hang around on #growstuff and intermittently on other channels. Message me any time; if I’m not awake/online I’ll see it when I return.

Email: skud@infotrope.net as ever, or skud@growstuff.org for Growstuff and related work.

Social media: I’m on social media hiatus and won’t be using it to chat at length, but still check mentions/messages semi-regularly.

Text/SMS: If you have my number, you know where to find me.

Voice/video (including phone, Skype, etc): By arrangement. Email me if you want to set something up.

To my good friends who I used to chat to all the time and now won’t see around so much: please let me know if you use Jabber/XMPP and if so what your address is; if you do, then I’ll prioritise getting that set up.

Syndicated 2014-09-30 23:57:30 from Infotropism

30 Sep 2014 etbe   » (Master)

Links September 2014

Matt Palmer wrote a short but informative post about enabling DNS in a zone [1]. I really should setup DNSSEC on my own zones.

Paul Wayper has some insightful comments about the Liberal party’s nasty policies towards the unemployed [2]. We really need a Basic Income in Australia.

Joseph Heath wrote an interesting and insightful article about the decline of the democratic process [3]. While most of his points are really good I’m dubious of his claims about twitter. When used skillfully twitter can provide short insights into topics and teasers for linked articles.

Sarah O wrote an insightful article about NotAllMen/YesAllWomen [4]. I can’t summarise it well in a paragraph, I recommend reading it all.

Betsy Haibel wrote an informative article about harassment by proxy on the Internet [5]. Everyone should learn about this before getting involved in discussions about “controversial” issues.

George Monbiot wrote an insightful and interesting article about the referendum for Scottish independence and the failures of the media [6].

Mychal Denzel Smith wrote an insightful article “How to know that you hate women” [7].

Sam Byford wrote an informative article about Google’s plans to develop and promote cheap Android phones for developing countries [8]. That’s a good investment in future market share by Google and good for the spread of knowledge among people all around the world. I hope that this research also leads to cheap and reliable Android devices for poor people in first-world countries.

Deb Chachra wrote an insightful and disturbing article about the culture of non-consent in the IT industry [9]. This is something we need to fix.

David Hill wrote an interesting and informative article about the way that computer game journalism works and how it relates to GamerGate [10].

Anita Sarkeesian shares the most radical thing that you can do to support women online [11]. Wow, the world sucks more badly than I realised.

Michael Daly wrote an article about the latest evil from the NRA [12]. The NRA continues to demonstrate that claims about “good people with guns” are lies, the NRA are evil people with guns.

Related posts:

  1. Links July 2014 Dave Johnson wrote an interesting article for Salon about companies...
  2. Links May 2014 Charmian Gooch gave an interesting TED talk about her efforts...
  3. Links September 2013 Matt Palmer wrote an insightful post about the use of...

Syndicated 2014-09-30 13:55:48 from etbe - Russell Coker

30 Sep 2014 mikal   » (Journeyer)

Blueprints implemented in Nova during Juno

As we get closer to releasing the RC1 of Nova for Juno, I've started collecting a list of all the blueprints we implemented in Juno. This was mostly done because it helps me write the release notes, but I am posting it here because I am sure that others will find it handy too.

Process



Ongoing behind the scenes work

Object conversion

Scheduler
  • Support sub-classing objects. launchpad specification
  • Stop using the scheduler run_instance method. Previously the scheduler would select a host, and then boot the instance. Instead, let the scheduler select hosts, but then return those so the caller boots the instance. This will make it easier to move the scheduler to being a generic service instead of being internal to nova. launchpad specification
  • Refactor the nova scheduler into being a library. This will make splitting the scheduler out into its own service later easier. launchpad specification
  • Move nova to using the v2 cinder API. launchpad specification
  • Move prep_resize to conductor in preparation for splitting out the scheduler. launchpad specification


API
  • Use JSON schema to strongly validate v3 API request bodies. Please note this work will later be released as v2.1 of the Nova API. launchpad specification
  • Provide a standard format for the output of the VM diagnostics call. This work will be exposed by a later version of the v2.1 API. launchpad specification
  • Move to the OpenStack standard name for the request id header, in a backward compatible manner. launchpad specification
  • Implement the v2.1 API on the V3 API code base. This work is not yet complete. launchpad specification


Other
  • Refactor the internal nova API to make the nova-network and neutron implementations more consistent. launchpad specification


General features

Instance features

Networking

Scheduling
  • Extensible Resource Tracking. The set of resources tracked by nova is hard coded, this change makes that extensible, which will allow plug-ins to track new types of resources for scheduling. launchpad specification
  • Allow a host to be evacuated, but with the scheduler selecting destination hosts for the instances moved. launchpad specification
  • Add support for host aggregates to scheduler filters. launchpad: disk; instances; and IO ops specification


Other
  • i18n Enablement for Nova, turn on the lazy translation support from Oslo i18n and updating Nova to adhere to the restrictions this adds to translatable strings. launchpad specification
  • Offload periodic task sql query load to a slave sql server if one is configured. launchpad specification
  • Only update the status of a host in the sql database when the status changes, instead of every 60 seconds. launchpad specification
  • Include status information in API listings of hypervisor hosts. launchpad specification
  • Allow API callers to specify more than one status to filter by when listing services. launchpad specification
  • Add quota values to constrain the number and size of server groups a users can create. launchpad specification


Hypervisor driver specific

Hyper-V

Ironic

libvirt

vmware
  • Move the vmware driver to using the oslo vmware helper library. launchpad specification
  • Add support for network interface hot plugging to vmware. launchpad specification
  • Refactor the vmware driver's spawn functionality to be more maintainable. This work was internal, but is mentioned here because it significantly improves the supportability of the VMWare driver. launchpad specification


Tags for this post: openstack juno blueprints implemented

Comment

Syndicated 2014-09-30 05:05:00 (Updated 2014-09-30 21:08:59) from stillhq.com : Mikal, a geek from Canberra living in Silicon Valley (no blather posts)

30 Sep 2014 Skud   » (Master)

Open food interoperability: entities, unique IDs, and semantic equivalence

This is a post I made on Growstuff Talk to propose some initial steps towards interoperability for open food projects. If you have comments, probably best to make them on that post.


I wanted to post about some concepts from my past open data work which have been very much in my mind when working on Growstuff, but which I’m not sure I’ve ever expressed in a way that helps everyone understand their importance.

Just for background: from 2007-2011 I worked on Freebase, a massive general-purpose open data repository which was acquired by Google in 2010 and now forms part of their “Knowledge” area. While working at Google I also worked as a liaison between Google search/knowledge and the Wikimedia Foundation, and presented at a Wikimedia data summit where we proposed the first stages of what would become Wikidata — an entity-based data store for all of Wikimedia’s other projects.

Freebase and Wikidata are part of what is broadly known as the Semantic Web, which has to do with providing data and meaning via web technologies, using common data formats etc.


The Semantic Web movement has several different branches, ranging from the extremely abstract and academic, to the quite mundane and pragmatic. Some of the more common bits of Semantic Web technology you might have come across are microformats, for instance, which let you add semantic meaning to your HTML markup, for instance for defining the meanings of links to things like licenses or for marking up recipes on food blogs and the like. There is also Semantic Mediawiki which adds some semantic features on top of a wiki, to allow you to query for information in interesting ways; Practical Plants uses SMW and its search is based on this semantic data.

At the more academic end of the Semantic Web world are things like RDF which creates a directed graph of semantic data which can be queried via a language called SPARQL, and attempts to define data standards and ontologies for a wide range of purposes. These are generally heavyweight and mostly of interest to researchers, academics, etc, though some aspects of this work are starting to seep through into consumer technology.

This is all background, however. What I wanted to talk about was the single most important thing we learned while working on Freebase, which is this:

Entities must have unique identifiers.

Here’s what I mean. Let’s say you know three people all called Mary Smith. Then someone says, “It’s Mary Smith’s birthday today.” Which one are they referring to? You don’t know. In any system based around knowledge, you need to have some kind of unique ID for each entity to avoid ambiguity. So instead you might say, “Mary Smith, whose employee number is E453425″ or “Mary Smith, whose email address is mary@example.com”, or “Mary Smith, whose primary key in our database is 789″.

When working on our proposal for phase 1 of Wikidata, one of the things we realised is that the Wikimedia community — all the languages of Wikipedia, the Wikimedia Commons, etc — lacked unique identifiers for real-world entities. For instance, Barack Obama was http://en.wikipedia.org/wiki/Barack_Obama on English Wikipedia and http://de.wikipedia.org/wiki/Barack_Obama on German Wikipedia and http://commons.wikimedia.org/wiki/Barack_Obama on Wikimedia Commons and http://en.wikinews.org/wiki/Category:Barack_Obama on Wikinews, but none of these was his definitive identifier.

Meanwhile, interwiki links — the links between English and German and French and Swahili and Korean wikipedias — were maintained by hand (or, actually, by a bot) that had to update every wikipedia whenever a page was added or changed on any of them. This was a combinatoric exercise: with 2 wikis, there are two links (A -> B and B <- A). With 5 wikis there are (4 + 3 + 2 + 1) * 2 links. With N wikis, there are N-1! * 2 links, or to put it another way, 50 wikis would mean 1.2165637e+63 links between them. This was wildly inefficient to maintain!

Wikidata’s “phase 1″ was to create an entity store for Wikimedia projects, where each concept or entity — “Barack Obama” or “semantic web” or “tomato” — would have a central identity which could be linked to. Then, each Wikimedia project could say “This page describes entity XYZ”, or conversely Wikidata could say “this entity is described on these pages”, and suddenly the work of the interwiki bot became much easier: it meant that each new wiki added would only mean one new link, not an exponentially-expanding web of links.

We are in a similar position with open food data at present. There are dozens of open source food projects and that list doesn’t even touch on the ones that are more connected to recipes/eating/nutrition. We’re talking about how to interoperate between our various projects, but the key to interoperability is entity identification. If someone wants to mash up Growstuff’s harvest data with Openrecipes recipe search or the US FDA’s nutrition data, they need to know that Growstuff’s tomato is the same as the tomato you use in spaghetti sauce or the tomato that contains some percent of your RDA of potassium.

So how do we do this? None of our projects are sufficiently established, mature, or complete to claim the right to be the central ID repository. Apart from that, many of us have different focuses — edible plants, all types of plants, all types of living things, and all types of food (including non-animal/non-plant food) are some of the scopes I can mention offhand. Even the wide-ranging species databases like the Encyclopedia of Life don’t capture such information as crop varieties (eg. roma tomato, habanero pepper) that are important to veggie gardeners like Growstuff’s members.

Here’s what I would propose as an interim measure.

All open food projects need to link their major entities (eg. “crops” in Growstuff’s case) to one or more large, open, API-accessible data stores.

Examples of these include:

  • Wikipedia (any language, but English has the most articles)
  • Wikidata
  • Freebase
  • Encyclopedia of Life

By doing this, we can match data between projects. For instance, if Growstuff’s “tomato” links to the same entity as OpenFarm’s “tomato” and OpenFoodNetwork’s “tomato” and OpenRecipes’ “tomato” then we can reasonably assume they’re all talking about the same thing.

Also, some of the above data sources provide APIs which allow us to pivot easily between data sets. For instance, Freebase’s query language allows you to ask questions like “given an entity that is identified as ‘tomato’ on English Wikipedia, what is its identify on the Encyclopedia of Life?”

To see this in action, paste the following query into Freebase’s interactive query editor:

    [{
      "a:key": [{
        "namespace": "/wikipedia/en",
        "value": "Tomato"
      }],
      "b:key": [{
        "namespace": "/biology/eol",
        "value": null
      }]    
    }]

As you’ll see, the result is “392557” or to put it another way http://eol.org/pages/392557 — the EOL page on tomatoes.

From day 1, Growstuff has been tracking Wikipedia links for all our crops, to enable this sort of query against Freebase and so easily pivot to other data sets that Freebase knows about. If other projects take similar steps, this means that we are well on our way toward interoperability.

(As an aside, this is why we’re also having this other discussion about what to do about crop varieties that don’t have their own Wikipedia page, as this messes up the 1-to-1 relationship between Wikipedia entities and Growstuff entities. This may be something we just have to deal with, however, as no external data set will exactly match ours.)

Next steps

  1. I strongly encourage all open food projects to link their “crops” or similar entities to one or more major, open-licensed, API-accessible data source (ideally one which has its keys in Freebase).
  2. We should all expose these links via our APIs, data dumps, or whatever other mechanisms we use to make our open data available.
  3. Developers should be able to request data from our APIs based on these identifiers, either through query parameters or through REST API resources like eg. /crops/eol/392557.json
  4. We should use semantic markup/links to denote this entity equivalence on our webpages, eg. if Growstuff links to a Practical Plants page on the same crop, there should be a standard way to say “we consider these pages to refer to the same entity”. I’m not sure exactly what this is, yet, but if we do this it will benefit web crawlers, search engines, and other non-API consumers of our websites.
  5. We should look into developing a microformat for expressing crop information on a webpage, in collaboration with microformats.org. I expect, however, that it will be very hard to develop a workable ontology, since (for instance) some of our projects are interested in planting information and some aren’t, some are interested in sale and distribution and others aren’t, some are dealing with non-edible plants and others aren’t, etc. It may have to be as simple as “this is a crop and here are the names we have for it”.
  6. It would be great to put together some kind of visualisation like the linked open data cloud to show which open food projects are providing interoperable identities and how they connect to each other.

I’d like to get buy-in from other open food data projects on at least the general idea of matching our “crop” entities (whatever we call them) against some of the big databases. Who’s in?

Syndicated 2014-09-30 02:11:13 from Infotropism

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Advogato User Stats
Users13999
Observer9884
Apprentice745
Journeyer2337
Master1029

New Advogato Members

Recently modified projects

20 Jun 2014 Ultrastudio.org
13 Apr 2014 Babel
13 Apr 2014 Polipo
19 Mar 2014 usb4java
8 Mar 2014 Noosfero
17 Jan 2014 Haskell
17 Jan 2014 Erlang
17 Jan 2014 Hy
17 Jan 2014 clj-simulacrum
17 Jan 2014 Haskell-Lisp
17 Jan 2014 lfe-disco
17 Jan 2014 clj-openstack
17 Jan 2014 lfe-openstack
17 Jan 2014 LFE
10 Jan 2014 libstdc++

New projects

8 Mar 2014 Noosfero
17 Jan 2014 Haskell
17 Jan 2014 Erlang
17 Jan 2014 Hy
17 Jan 2014 clj-simulacrum
17 Jan 2014 Haskell-Lisp
17 Jan 2014 lfe-disco
17 Jan 2014 clj-openstack
17 Jan 2014 lfe-openstack
17 Jan 2014 LFE
1 Nov 2013 FAQ Linux
15 Apr 2013 Gramps
8 Apr 2013 pydiction
28 Mar 2013 Snapper
5 Jan 2013 Templer