Recent blog entries

5 Feb 2016 zeenix   » (Journeyer)

FOSDEM & Dev-x Hackfest 2016

Last week I travelled to Brussels to attend and present at FOSDEM. Since I was going there anyway, I decided to also join 2.5 days of GNOME Developer Experience Hackfest.

Travelling to Brussels is usually pretty easy and fast, thanks to Eurostar but I turned it into a bit of nightmare this time. I had completely forgotten how London public transport is a total disasters in peak hours and hence ended-up arriving too late at the station. Not a big deal, they put me on the next train for free. I decided to go through security already and that's when I realized that I have forgotten my laptop at home. :( Fortunately my nephew (who is also my flatmate) was still at home and was going to travel to city centre anyway so I asked him to bring it with him. After two hours of anxiously waiting, he managed to arrive just in time for train staff to let in the very last late arriving passenger. Phew!

While I didn't have a particular agenda for the hackfest, I had a discussion with Alexander Larsson about sandboxing in xdg-app and how we will implement per-app authorization to location information from Geoclue. The main problem has always been that we have no means of reliably identifying apps and turns out that xdg-app already solved that problem. Each xdg-app has it's ID (i-e the name of it's desktop file w/o the .desktop suffix) in /proc/PID/cgroup file and app can not change that.

So I sat down and started working on this. I was able to finish off the Geoclue part of the solution already before the hackfest ended and now working on gnome-shell (currently the only geoclue app authorizing agent) part. Once done I'll then add settings in gnome-control-center so users can change their mind about whether or not they want an app to be able to access their location. Other than that, I helped test a few xdg-app bundles.

It's important to keep in mind that this solution will still involve trusting the system (non-xdg-app) application as there is no way to reliably identify those. i-e if you download a random script from internet and run it, we can not possibly guarantee that it won't access your location without your consent. Let's hope that xdg-app becomes very ubiquitous and becomes a de-facto standard for distributing your Linux apps in the near future.

FOSDEM was a fun weekend as usual. I didn't attend a lot of talks but met many interesting people and we had chat about various different open source technologies. I was glad to hear that a project I started off as a simple proof-of-concept for GUPnP, is now a days used in automobiles.

My own talk about Geospacial technologies in GNOME went fine except for the fact that I ran out of time towards the end and my Raspberry Pi demo didn't work because I forgot to plug-in the WiFi adaptor. :( Still, I was able to cover most of the topics and Maps demo worked pretty smoothly (there was  weird libchamplain bug I hit but it wasn't very critical at all).

While I came back home pumped with a lot of motivation, unfortunately I managed to catch the infamous FOSDEM flu. I've been resting most of the week and today I started to feel better so I'm writing this late blog post as the first thing, before I completely forget what happened last week.

Oh and last but not the least, many thanks to GNOME foundation for sponsoring my train tickets.


Syndicated 2016-02-05 17:19:00 (Updated 2016-02-05 17:19:33) from zeenix

5 Feb 2016 sdodji   » (Master)

4 Feb 2016 wingo   » (Master)

guile compiler tasks

Hey! We released Guile 2.1.2, including the unboxing work, and we fixed the slow bootstrap problem by shipping pre-built bootstraps in tarballs. A pretty OK solution in my opinion; check it out!

future work

At this point I think I'm happy with Guile's compiler and VM, enough for now. There is a lot more work to do but it's a good point at which to release a stable series. There will probably be a number of additional pre-releases, but not any more significant compiler/VM work that must be done before a release.

However, I was talking with Guilers at FOSDEM last weekend and we realized that although we do a pretty good job at communicating the haps in compiler-land, we don't do a good job at sharing a roadmap or making it possible for other folks to join the hack. And indeed, it's been difficult to do so while things were changing so much: I had to get things right in my head before joining in the confusion of other people's heads.

In that spirit I'd like to share a list of improvements that it would be nice to make at some point. If you take one of these tasks, be my guest: find me on IRC (wingo on freenode) and let me know, and I'll help as I am able. You need to be somewhat independent; I'm not offering a proper mentoring or anything, more like office hours or something, where you come with the problem you are having and I commiserate and give context/background/advice as I am able.

So with that out of the way, here's a huge list of stuff! Following this, more details on each one.

  1. stripping binaries

  2. full source in binaries

  3. cps in in binaries

  4. linking multiple modules together

  5. linking a single executable

  6. instruction explosion

  7. elisp optimizations

  8. prompt removal

  9. basic register allocation

  10. optimal register allocation

  11. unboxed record fields

  12. textual CPS

  13. avoiding arity checks

  14. unboxed calls and returns

  15. module-level inlining

  16. cross-module inlining

As a bonus, in the end I'll give some notes on native compilation. But first, the hacks!

stripping binaries

Guile uses ELF as its object file format, and currently includes source location information as DWARF data. On space-constrained devices this might be too much. Your task: add a hack to the linker that can strip existing binaries. Read Ian Lance Taylor's linker articles for more background, if you don't know things about linkers yet.

full source in binaries

Wouldn't it be nice if the ELF files that Guile generates actually included the source as well as the line numbers? We could do that, in a separate strippable ELF section. This point is like the reverse of the previous point :)

cps in in binaries

We could also include the CPS IR in ELF files too. This would enable some kinds of link-time optimization and cross-module inlining. You'd need to define a binary format for CPS, like LLVM bitcode or so. Neat stuff :)

linking multiple modules together

Currently in Guile, just about every module is a separate .go file. Loading a module will cause a few stat calls and some seeks and reads and all that. Wouldn't it be nice if you could link together all the .go files that were commonly used into one object? Again this is a linker hack, but it needs support from the run-time as well: when the run-time goes to load a file, it should first check in a registry if that file has been logically provided by some other file. We'd be able to de-duplicate constant data from various modules. However there is an initialization phase when loading a .go file which effectively performs all the relocations needed by constants that need a fix-up at load-time; see the ELF article I linked to above for more. For some uses, it would be OK to produce one relocation/initialization procedure. For others, if you expected to only load a fraction of the modules in a .go file, it would be a lose on startup time,
so you would probably need to support lazy relocation when a module is first loaded.

Anyway, your task would be to write a linker hack that loads a bunch of .go files, finds the relocations in them, de-duplicates the constants, and writes out a combined .go file that includes a table of files contained in it. Good luck :) This hack would work great for Emacs, where it's effectively a form of unexec that doesn't actually rely on unexec.

linking a single executable

In the previous task, you could end up with the small guile binary that links to libguile (or your binary linking to libguile), and then a .go file containing all the modules you are interestd in. It sure would be nice to be able to link those together into just one binary, or at least to link the .go into the Guile binary. If the Guile is statically linked itself, you would have a statically linked application. If it's dynamically linked, it would remain dynamically linked. Again, a linker hack, but one that could provide a nicer way to distribute Guile binaries.

instruction explosion

Now we get more to the compiler side of things. Currently in Guile's VM there are instructions like vector-ref. This is a little silly: there are also instructions to branch on the type of an object (br-if-tc7 in this case), to get the vector's length, and to do a branching integer comparison. Really we should replace vector-ref with a combination of these test-and-branches, with real control flow in the function, and then the actual ref should use some more primitive unchecked memory reference instruction. Optimization could end up hoisting everything but the primitive unchecked memory reference, while preserving safety, which would be a win. But probably in most cases optimization wouldn't manage to do
this, which would be a lose overall because you have more instruction dispatch.

Well, this transformation is something we need for native compilation anyway. I would accept a patch to do this kind of transformation on the master branch, after version 2.2.0 has forked. In theory this would remove most all high level instructions from the VM, making the bytecode closer to a virtual CPU, and likewise making it easier for the compiler to emit native code as it's working at a lower level.

elisp optimizations

Guile implements Emacs Lisp, and does so well. However it hasn't been the focus of a lot of optimization. Emacs has a lot of stuff going on on its side, and so have we, so we haven't managed to replace the Elisp interpreter in Emacs with one written in Guile, though Robin Templeton has brought us a long way forward. We need someone to do both the integration work but also to poke the compiler and make sure it's a clear win.

prompt removal

It's pretty natural to use delimited continuations when compiling some kind of construct that includes a break statement to Guile, whether that compiler is part of Elisp or just implemented as a Scheme macro. But, many instances of prompts can be contified, resulting in no overhead at run-time. Read up on contification and contify the hell out of some prompts!

basic register allocation

Guile usually tries its best to be safe-for-space: only the data which might be used in the future of a program is kept alive, and the rest is available for garbage collection. Notably, this applies to function arguments, temporaries, and lexical variables: if a value is dead, the GC can collect it and re-use its space. However this isn't always what you want. Sometimes you might want to have all variables that are in scope to be available, for better debugging. Your task would be to implement a "slot allocator" (which is really register allocation) that keeps values alive in the parts of the programs that they dominate.

optimal register allocation

On the other hand, our slot allocator -- which is basically register allocation, but for stack slots -- isn't so great. It does OK but you can often end up shuffling values in a loop, which is the worst. Your task would be to implement a proper register allocator: puzzle-solving, graph-coloring, iterative coalescing, something that really tries to do a good job. Good luck!

unboxed record fields

Guile's "structs", on which records are implemented, support unboxed values, but these values are untyped, not really integrated with the record layer, and always boxed in the VM. Your task would be to design a language facility that allows us to declare records with typed fields, and to store unboxed values in those fields, and to cause access to their values to emit boxing/unboxing instructions around them. The optimizer will get rid of those boxing/unboxing instructions if it can. Good luck!

textual CPS

The CPS language is key to all compiler work in Guile, but it doesn't have a nice textual form like LLVM IR does. Design one, and implement a parser and an unparser!

avoiding arity checks

If you know the procedure you are calling, like if it's lexically visible, then if you are calling it with the right number of arguments you can skip past the argument check and instead do a call-label directly into the body. Would be pretty neat!

unboxed calls and returns

Likewise if a function's callers are all known, it might be able to unbox its arguments or return value, if that's a good idea. Tricky! You could start with a type inference pass or so, and maybe that could produce some good debugging feedback too.

module-level inlining

Guile currently doesn't inline anything that's not lexically visible. Unfortunately this restriction extends to top-level definitions in a module: they are treated as mutable and so never inlined/optimized/etc. Probably we need to change the semantics here such that a module can be compiled as a unit, and all values which are never mutated can be assumed to be constant. Probably you also want a knob to turn off this behavior, but really you can always re-compile and re-load a module as a whole if re-loading a function at run-time doesn't work because it was inlined. Anyway. Some semantic work here, but some peval work as well. Be careful!

cross-module inlining

Likewise Guile currently doesn't inline definitions from other modules. However for small functions this really hurts. Guile should probably serialize tree-il for small definitions in .go files, and allow peval to speculatively inline imported definitions. This is related to the previous point and has some semantic implications.

bobobobobobonus! native compilation

Thinking realistically, native compilation is the next step. We have the object file format, cool. We will need the ability to call out from machine code in .go files to run-time functions, so we need to enhance the linker, possibly even with things like PLT/GOT sections to avoid dirtying too many pages. We need to lower the CPS even further, to get closer to some kind of machine model, then go specific, with an assembler for each architecture. The priority in the beginning will be simplicity and minimal complexity; good codegen will come later. This is obviously the most attractive thing but it's also the most tricky, design-wise. I want to do at least part of this, so though you can't have it all, you are welcome to help :)

That's it for now. I'll amend the post with more things as and when I think of them. Comments welcome too, as always. Happy hacking!

Syndicated 2016-02-04 21:38:05 from wingolog

4 Feb 2016 etbe   » (Master)

Unikernels

At LCA I attended a talk about Unikernels. Here are the reasons why I think that they are a bad idea:

Single Address Space

According to the Unikernel Wikipedia page [1] a significant criteria for a Unikernel system is that it has a single address space. This gives performance benefits as there is no need to change CPU memory mappings when making system calls. But the disadvantage is that any code in the application/kernel can access any other code directly.

In a typical modern OS (Linux, BSD, Windows, etc) every application has a separate address space and there are separate memory regions for code and data. While an application can request the ability to modify it’s own executable code in some situations (if the OS is configured to allow that) it won’t happen by default. In MS-DOS and in a Unikernel system all code has read/write/execute access to all memory. MS-DOS was the least reliable OS that I ever used. It was unreliable because it performed tasks that were more complex than CP/M but had no memory protection so any bug in any code was likely to cause a system crash. The crash could be delayed by some time (EG corrupting data structures that are only rarely accessed) which would make it very difficult to fix. It would be possible to have a Unikernel system with non-modifyable executable areas and non-executable data areas and it is conceivable that a virtual machine system like Xen could enforce that. But that still wouldn’t solve the problem of all code being able to write to all data.

On a Linux system when an application writes to the wrong address there is a reasonable probability that it will not have write access and you will immediately get a SEGV which is logged and informs the sysadmin of the address of the crash.

When Linux applications have bugs that are difficult to diagnose (EG buffer overruns that happen in production and can’t be reproduced in a test environment) there are a variety of ways of debugging them. Tools such as Valgrind can analyse memory access and tell the developers which code had a bug and what the bug does. It’s theoretically possible to link something like Valgrind into a Unikernel, but the lack of multiple processes would make it difficult to manage.

Debugging

A full Unix environment has a rich array of debugging tools, strace, ltrace, gdb, valgrind and more. If there are performance problems then tools like sysstat, sar, iostat, top, iotop, and more. I don’t know which of those tools I might need to debug problems at some future time.

I don’t think that any Internet facing service can be expected to be reliable enough that it will never need any sort of debugging.

Service Complexity

It’s very rare for a server to have only a single process performing the essential tasks. It’s not uncommon to have a web server running CGI-BIN scripts or calling shell scripts from PHP code as part of the essential service. Also many Unix daemons are not written to run as a single process, at least threading is required and many daemons require multiple processes.

It’s also very common for the design of a daemon to rely on a cron job to clean up temporary files etc. It is possible to build the functionality of cron into a Unikernel, but that means more potential bugs and more time spent not actually developing the core application.

One could argue that there are design benefits to writing simple servers that don’t require multiple programs. But most programmers aren’t used to doing that and in many cases it would result in a less efficient result.

One can also argue that a Finite State Machine design is the best way to deal with many problems that are usually solved by multi-threading or multiple processes. But most programmers are better at writing threaded code so forcing programmers to use a FSM design doesn’t seem like a good idea for security.

Management

The typical server programs rely on cron jobs to rotate log files and monitoring software to inspect the state of the system for the purposes of graphing performance and flagging potential problems.

It would be possible to compile the functionality of something like the Nagios NRPE into a Unikernel if you want to have your monitoring code running in the kernel. I’ve seen something very similar implemented in the past, the CA Unicenter monitoring system on Solaris used to have a kernel module for monitoring (I don’t know why). My experience was that Unicenter caused many kernel panics and more downtime than all other problems combined. It would not be difficult to write better code than the typical CA employee, but writing code that is good enough to have a monitoring system running in the kernel on a single-threaded system is asking a lot.

One of the claimed benefits of a Unikernel was that it’s supposedly risky to allow ssh access. The recent ssh security issue was an attack against the ssh client if it connected to a hostile server. If you had a ssh server only accepting connections from management workstations (a reasonably common configuration for running servers) and only allowed the ssh clients to connect to servers related to work (an uncommon configuration that’s not difficult to implement) then there wouldn’t be any problems in this regard.

I think that I’m a good programmer, but I don’t think that I can write server code that’s likely to be more secure than sshd.

On Designing It Yourself

One thing that everyone who has any experience in security has witnessed is that people who design their own encryption inevitably do it badly. The people who are experts in cryptology don’t design their own custom algorithm because they know that encryption algorithms need significant review before they can be trusted. The people who know how to do it well know that they can’t do it well on their own. The people who know little just go ahead and do it.

I think that the same thing applies to operating systems. I’ve contributed a few patches to the Linux kernel and spent a lot of time working on SE Linux (including maintaining out of tree kernel patches) and know how hard it is to do it properly. Even though I’m a good programmer I know better than to think I could just build my own kernel and expect it to be secure.

I think that the Unikernel people haven’t learned this.

No related posts.

Syndicated 2016-02-04 09:48:52 from etbe – Russell Coker

4 Feb 2016 gpoo   » (Master)

Johnson Street Bridge at night

Puente Johnson Johnson Street Bridge at night. Victoria, British Columbia, Canada.

It has been a while since I wanted to take this picture. Tonight there was a light rain, I was in the mood of taking long-exposure pictures, and I had a tripod with me.

Although iconic, this bridge is going to be replaced by a new one in 2017. Half of the bridge will be dedicated for pedestriand and cyclists.

Syndicated 2016-02-04 06:59:00 from Moving a castle

4 Feb 2016 hypatia   » (Journeyer)

Joining Stripe

I’ve been searching for a new position since finishing at the Ada Initiative at the end of September 2015. On January 11, I was very happy to join Stripe in Australia as a Partner Engineer, working as a technical expert with Stripe’s partners.

Stripe is the best way to accept payments online and in mobile apps. (It’s pretty cool to see the change in payments since the last time I worked in a payments company.) My job will involve working closely with Australian companies, which I am especially looking forward to after ending up with a lot of US and Silicon Valley focus over the past few years of my life.

I’ll mostly be based remotely in Sydney, with regular visits to the Australian team in Melbourne. I’m thrilled to work closely with Susan Wu, Mac Wang, and the team in Australia, as well of course as with the company as a whole. I spent my first two weeks with Stripe in San Francisco and love how friendly and welcoming my colleagues are.

Work at Stripe

Stripe is just starting to build a Sales and Partner Engineering team to go with their strong Support Engineering team. If you’re interested in joining me in the Field Engineering team at Stripe, there are multiple positions open, and they include the Head of Field Engineering and Sales Engineering Manager (to whom I will report), both San Francisco-based. If you want to work in Australia. there is a Sales Engineer position open in Melbourne.

If you want to talk to me about working at Stripe, email mary@stripe.com (hey look at that, there’s still firstname@ opportunities too!)

Syndicated 2016-02-03 23:50:54 from puzzling.org

4 Feb 2016 marnanel   » (Journeyer)

"Who listens to the radio through a bleedin` satellite anyway?"

I don't understand all the physics, but I read Justin Smith's aerialsandtv.com as much for his writing style as the technical stuff. Here's a sample:

"If you`re using the Sky free channels (as opposed to Freesat) in order to receive all the available free channels sometimes need a Sky card . At the moment this is quite cheap but it is only available from Sky and anyone who has had dealings with them can testify that it can be a frustrating business..... In fact when Which? researched call centres in Jan 11 they found Sky was the worst, and they`ve got some decent opposition in that department, particularly Royal Mail, and (ironically) BT, plus all the broadband providers, obviously. That`s the modern trend, companies don`t actually want to talk to their customers, not unless it`s a voice activated computer. I never talk to them. Well actually I do, I swear at them till they put me through to a human being. You should try it, it`s very satisfying.

Some of the of the programmes on Freeview are not available on Freesat. As far as I am aware Dave or the UK History channel are not available on Freesat although the situation could change so you are advised to check. Apparently UKTV History changed its name in March 09 to “Yesterday”, and it also changed its Freeview MUX allocation. Yet another example of name changing bollox. Isn`t all this digital TV complicated enough......

On the other hand there are a few more channels on Freesat than on Freeview. So you might get 120 odd channels of crap *, instead of the 80 odd channels of crap on Freeview. Big deal. So you can waste even more time going through the TV guide confirming there`s nothing actually worth watching anyway. Life`s wonderful.

* Remember they aren`t all TV channels, some are radio channels. Who listens to the radio through a bleedin` satellite anyway ? That`s what I want to know. Whatever next ? Gas companies selling electricity ? And I bet they`d charge too much for it. The world`s gone mad."


This entry was originally posted at http://marnanel.dreamwidth.org/356247.html. Please comment there using OpenID.

Syndicated 2016-02-03 23:45:18 from Monument

3 Feb 2016 dmarti   » (Master)

World's Simplest Privacy Tool

Here's the world's simplest Firefox add-on, which just turns on Tracking Protection (ordinarily buried somewhere in about:config) and sets third-party cookie policy to a sane value.

install pq from addons.mozilla.org

So far it has 15 users and one review -- five stars. It doesn't do much, or for very many people, but what it does do it does with five-star quality.

Bonus link: How do I turn on Tracking Protection? Let me count the ways.

Syndicated 2016-02-03 03:52:31 from Don Marti

2 Feb 2016 etbe   » (Master)

Compatibility and a Linux Community Server

Compatibility/interoperability is a good thing. It’s generally good for systems on the Internet to be capable of communicating with as many systems as possible. Unfortunately it’s not always possible as new features sometimes break compatibility with older systems. Sometimes you have systems that are simply broken, for example all the systems with firewalls that block ICMP so that connections hang when the packet size gets too big. Sometimes to take advantage of new features you have to potentially trigger issues with broken systems.

I recently added support for IPv6 to the Linux Users of Victoria server. I think that adding IPv6 support is a good thing due to the lack of IPv4 addresses even though there are hardly any systems that are unable to access IPv4. One of the benefits of this for club members is that it’s a platform they can use for testing IPv6 connectivity with a friendly sysadmin to help them diagnose problems. I recently notified a member by email that the callback that their mail server used as an anti-spam measure didn’t work with IPv6 and was causing mail to be incorrectly rejected. It’s obviously a benefit for that user to have the problem with a small local server than with something like Gmail.

In spite of the fact that at least one user had problems and others potentially had problems I think it’s clear that adding IPv6 support was the correct thing to do.

SSL Issues

Ben wrote a good post about SSL security [1] which links to a test suite for SSL servers [2]. I tested the LUV web site and got A-.

This blog post describes how to setup PFS (Perfect Forward Secrecy) [3], after following it’s advice I got a score of B!

From the comments on this blog post about RC4 etc [4] it seems that the only way to have PFS and not be vulnerable to other issues is to require TLS 1.2.

So the issue is what systems can’t use TLS 1.2.

TLS 1.2 Support in Browsers

This Wikipedia page has information on SSL support in various web browsers [5]. If we require TLS 1.2 we break support of the following browsers:

The default Android browser before Android 5.0. Admittedly that browser always sucked badly and probably has lots of other security issues and there are alternate browsers. One problem is that many people who install better browsers on Android devices (such as Chrome) will still have their OS configured to use the default browser for URLs opened by other programs (EG email and IM).

Chrome versions before 30 didn’t support it. But version 30 was released in 2013 and Google does a good job of forcing upgrades. A Debian/Wheezy system I run is now displaying warnings from the google-chrome package saying that Wheezy is too old and won’t be supported for long!

Firefox before version 27 didn’t support it (the Wikipedia page is unclear about versions 27-31). 27 was released in 2014. Debian/Wheezy has version 38, Debian/Squeeze has Iceweasel 3.5.16 which doesn’t support it. I think it is reasonable to assume that anyone who’s still using Squeeze is using it for a server given it’s age and the fact that LTS is based on packages related to being a server.

IE version 11 supports it and runs on Windows 7+ (all supported versions of Windows). IE 10 doesn’t support it and runs on Windows 7 and Windows 8. Are the free upgrades from Windows 7 to Windows 10 going to solve this problem? Do we want to support Windows 7 systems that haven’t been upgraded to the latest IE? Do we want to support versions of Windows that MS doesn’t support?

Windows mobile doesn’t have enough users to care about.

Opera supports it from version 17. This is noteworthy because Opera used to be good for devices running older versions of Android that aren’t supported by Chrome.

Safari supported it from iOS version 5, I think that’s a solved problem given the way Apple makes it easy for users to upgrade and strongly encourages them to do so.

Log Analysis

For many servers the correct thing to do before even discussing the issue is to look at the logs and see how many people use the various browsers. One problem with that approach on a Linux community site is that the people who visit the site most often will be more likely to use recent Linux browsers but older Windows systems will be more common among people visiting the site for the first time. Another issue is that there isn’t an easy way of determining who is a serious user, unlike for example a shopping site where one could search for log entries about sales.

I did a quick search of the Apache logs and found many entries about browsers that purport to be IE6 and other versions of IE before 11. But most of those log entries were from other countries, while some people from other countries visit the club web site it’s not very common. Most access from outside Australia would be from bots, and the bots probably fake their user agent.

Should We Do It?

Is breaking support for Debian/Squeeze, the built in Android browser on Android <5.0, and Windows 7 and 8 systems that haven’t upgraded IE as a web browsing platform a reasonable trade-off for implementing the best SSL security features?

For the LUV server as a stand-alone issue the answer would be no as the only really secret data there is accessed via ssh. For a general web infrastructure issue it seems that the answer might be yes.

I think that it benefits the community to allow members to test against server configurations that will become more popular in the future. After implementing changes in the server I can advise club members (and general community members) about how to configure their servers for similar results.

Does this outweigh the problems caused by some potential users of ancient systems?

I’m blogging about this because I think that the issues of configuration of community servers have a greater scope than my local LUG. I welcome comments about these issues, as well as about the SSL compatibility issues.

Related posts:

  1. Name Server IP and a Dead Server About 24 hours ago I rebooted the system that runs...
  2. Server Costs vs Virtual Server Costs The Claim I have seen it claimed that renting a...
  3. My Blog Server was Cracked On the 1st of August I noticed that the server...

Syndicated 2016-02-02 05:44:52 from etbe – Russell Coker

31 Jan 2016 softkid   » (Journeyer)

Fosdem 2016 day 2

Day 2 was a bit different than day 1, has I was less tired. It started by me visiting a few booths in order to decorate my bag and get a few more T-shirts, thanks to wiki-mania, Apache, Open Stack. I got the mini-port to VGA cable I had left in the conference room and then  headed for the conferences.

The first one was “Active supervision and monitoring with Salt, Graphite and Grafana“ was interesting because I knew nothing about any of these, except for graphite, but I knew so little that I learned a lot.

The second one titled “War Story: Puppet in a Traditional Enterprise” was someone implementing puppet at an enterprise scale in a big company. It reminded me all the big company I had consulted to a few years back - nothing surprising. It was quiet interesting anyway.

The Third talk I attend was about hardening and securing configuration management software. It was more about general principle than an howto. Quite interesting specially the hardening.io link given at the end of the documentation and the idea to remove ssh if possible on all servers and enable it thru conf. management to investigate issues. I didn’t learn much but it was a good refresher.

I then attend a talk in a very small room that was packed packed packed , about mapping with your phone. As I’ve started contributing to OSM, it was nice to listen and discover all the other apps that I can run on my droid phone in order to add data to the maps. I’ll probably share that next month at the local OSM meeting that got announced this week-end.

Last but not least I attended the key signing party. According to my paperwork, I’ll have sot sign twice 98 keys (twice because I’m creating a new key).

I’ve of course added a few pictures to my Fosdem set.

Syndicated 2016-01-31 22:22:05 from NaN

31 Jan 2016 amits   » (Journeyer)

FOSDEM 2016 Talk: Live Migration of Virtual Machines From The Bottom Up

I just did a talk titled ‘Live Migration of Virtual Machines From The Bottom Up‘ at the FOSDEM conference in Brussels, Belgium.  The slides are available at this location.

The talk introduced the KVM stack (Linux, KVM, QEMU, libvirt) and live migration; introduced ways the higher layers (especially oVirt and OpenStack) use KVM and migration, and what challenges the KVM team faces in working with varying use-cases and new features added to make migration work, and work faster.

There was a video recording, I will post the link to it in a separate post.

Syndicated 2016-01-31 15:00:33 (Updated 2016-01-31 00:03:43) from Think. Debate. Innovate.

31 Jan 2016 Pizza   » (Master)

Sinfonia CHC-S6145 (CS2) and Ciaat Brava 21, working!

Over the past year, I've written a bit about the situation involving the Sinfonia CHC-S6145 printer and its rebadged sibling, the Ciaat Brava 21. To summarize, the printers worked but required use of a proprietary, binary-only library ('libS6145ImageProcess') to perform thermal compensation and other transformations to the image data in order to generate sane output.

To make a long story short, I set out to reverse-engineer how that library worked... and a couple of weeks ago, I succeeded, with my reimplemented library generating completely identical results.

After some back and forth with Sinfonia, I'm quite pleased to announce that my re-implmented library, called 'libS6145ImageReProcess', is now released to the public under a GPLv3+ license. Except for the differing name, it is a drop-in replacement for the Sinfonia library.

Just to be absolutely clear, Sinfonia is not responsible for this library in any way, and will not support you if you complain that the output is somehow deficient or your printer catches fire when you print images of Donald Trump biting the heads off of adorable kittens.

Now in order to actually utilize these printers, you'll need to compile and install three components:

  • Gutenprint 5.2.11 (just released!)
  • libS6145ImageReProcess library
  • Latest selphy_print backend code

I should have the necessary backend code in the Gutenprint development repo soon, but due to licensing complications the library will probably remain separately distributed.

Particular thnaks go to Sinfonia and Ciaat for providing documentation on the printer communication protocols, and Matt Koglin for his SinfoniaCam(tm) and many, many rounds of testing.

This has been a long time coming, and is the culmination of quite a bit of work. I hope it proves useful, and if you do purchase one of these printers intending to use it with Linux (or a more obscure OS), please let your Sinfonia distributor know. :)

Syndicated 2016-01-30 20:24:40 from I Dream of Rain (free_software)

31 Jan 2016 marnanel   » (Journeyer)

The Dolorous Stroke, which is not about sex

Q: How does the term 'dolorous stroke' fit into the stories surrounding the search for the Holy Grail?

Answer (by me):

The Dolorous Stroke was when Sir Balin stabbed King Pelles in the genitals with a spear. This was even more unfortunate than it sounds, because the spear had also been used to stab Jesus during his crucifixion, and was therefore magic, and Bad Things begin to happen.

Pelles becomes unable to have sex, and he is therefore unable to have a son to be the next king. However, he has a daughter named Elaine, and he decides he'll use her sexuality instead.

Elaine has been trapped in a bath of boiling water by a sorceress. Sir Lancelot arrives, being all heroic, and gets her out of the bath, "naked as a needle" as Malory puts it. Pelles and Elaine get Sir Lancelot drunk, and Elaine uses her magic ring (honestly) to trick Lancelot into having sex with her. "Wit you well that Sir Launcelot was glad, and so was that lady Elaine that she had gotten Sir Launcelot in her arms." says Malory.

Next morning, Lancelot is angry because he thinks his strength depended on his being a virgin. I have no idea why-- girl cooties or something. Elaine then "skips out of bed all naked" (Malory is getting quite excited here) to tell Lancelot that she's pregnant. Pregnancy tests have evidently become less quick and reliable since the days of King Arthur.

Lancelot jumps out of the window and runs off. Nine months later, Elaine bears a son, Galahad, who has the same hangups about virginity as his father, and is insufferably priggish in every version I've read.

Elaine then vanishes from the narrative, since she's just a plot element. I would have said that the story was only interested in you if you have a penis, but then again there's Pelles.

Anyway, Pelles is frustrated and sad and infertile, and because of the magic, the kingdom starts becoming frustrated and sad and infertile as well. So Pelles has nothing to do except sit outside his castle fishing, and holding on to his long wooden rod all day. This is why Pelles is called the Fisher King.

When Galahad grows up, he becomes a Knight of the Round Table, and goes in search of the Grail along with all the others. Guess what? Pelles was actually looking after it the whole time. Lancelot and Galahad both find it, but Galahad gets to keep it because he's a virgin.

Now we find out why Pelles wanted Galahad to be conceived:

Galahad finds the same spear as before, which is now bleeding. He cures Pelles by thrusting the spear into the wound between his legs. No, seriously. Suddenly Pelles is no longer sad and frustrated, and the land becomes fertile again, and everyone's happy. But Galahad is still totally a virgin, of course.

(Bear in mind that these stories have been told over and over again for a thousand years, and the details change in the telling. Sometimes it's Percival rather than Galahad who finds the grail, for example.)

This entry was originally posted at http://marnanel.dreamwidth.org/355424.html. Please comment there using OpenID.

Syndicated 2016-01-30 23:08:02 (Updated 2016-01-30 23:20:04) from Monument

30 Jan 2016 softkid   » (Journeyer)

Fosdem 2016 day 1

This year I’m attending fosdem, after skipping it last year. It’s good to be back even if I was very tired when I arrived yesterday night and managed to visit three of Brussels train station. I was up early and the indications in bus 71 where fucked up so it took me a short walk under some rain to get to the campus - but I made it early and was able to take interesting empty pictures.

The first talk I attended was about MIPS for the embedded world. It was interesting for some tidbids, but felt more like a marketing speech to use MIPS on future embedding project.

After that I wanders and found a bunch of ex-joosters and had very interesting conversation with all of them.

I delivered my talk in 10 minutes and then answered question for the next 20 minutes.

The http2 talk was interesting and the room was packed. But probably not deep enough for me. Still I think we should think about enabling http/2 on mozfr.org.

I left to get some rest after talking to otto about block chain and bitcoins.

Syndicated 2016-01-30 15:58:51 from NaN

30 Jan 2016 lloydwood   » (Journeyer)

SaVi seasonal software development.

It's become my habit to use some of the downtime over the year-end break to... oh, said that about a year ago.

Once again, I did some software work over Christmas and released SaVi 1.4.9, for simulating satellite constellations.

This is mostly a maintenance release, which was needed to remove some old code that used tmpnam(), which modern linkers dislike. But it also adds some approximate simulations of LeoSat and OneWeb, which are "megaconstellations" currently competing to get constructed. (Mega? They're not even kilo...)

I'm having flashbacks to the 1990s, when I first started on SaVi, and big constellations like Teledesic and Skybridge were all the rage.

30 Jan 2016 hacker   » (Master)

No more criminals for president. Not this time. Never again.

I rarely speak out on specific presidential candidates, but I’ve pretty much had enough of this comedy unfolding that we call our political process. With Trump making a complete arse of himself, Ted Cruz saying “we need another war president” and other candidates flat-out lying, making up facts and paying third-party companies to create false […]

No related posts.

Syndicated 2016-01-30 04:28:18 from random neuron misfires

28 Jan 2016 johnnyb   » (Journeyer)

My new programming, New Programmers Start Here, just came out today! I'm very excited to get this in the hands of new programmers.

This book tries to be a "pull up by your bootstraps" guide to programming. It introduces more than just the language (JavaScript in this case) by taking the reader through how all of the pieces of data on a computer connect together, and how the computer itself works. I believe that this is the best first-programming book for new programmers.

If you have friends who are designers who want to start transitioning to programming, they should pick up this book. If you have young friends who want to get into programming, they should pick up this book.

27 Jan 2016 amits   » (Journeyer)

Interviews On My FOSDEM 2016 Talk

I was interviewed by the FOSDEM folks on my upcoming talk on KVM Live Migration.

I was also interviewed by Brian Proffitt, who has written an article that serves as a preview for the talk.

Looking forward to the talk this Sunday!

Syndicated 2016-01-27 22:09:33 from Think. Debate. Innovate.

27 Jan 2016 etbe   » (Master)

Using LetsEncrypt

Lets Encrypt is a new service to provide free SSL keys [1]. I’ve just set it up on a few servers that I run.

Issues

The first thing to note is that the client is designed to manage your keys and treat all keys on a server equally with a single certificate. It shouldn’t be THAT difficult to do things in other ways but it would involve extra effort. The next issue that can make things difficult is that it is designed that the web server will have a module to negotiate new keys automatically. Automatically negotiating new keys will be really great when we get that all going, but as I didn’t feel like installing a slightly experimental Apache module on my servers that meant I had to stop Apache while I got the keys – and I’ll have to do that again every 3 months as the keys have a short expiry time.

There are some other ways of managing keys, but the web servers I’m using Lets Encrypt with at the moment aren’t that important and a couple of minutes of downtime is acceptable.

When you request multiple keys (DNS names) for one server to make it work without needless effort you have to get them all in the one operation. That gives you a single key file for all DNS names which is very convenient for services that don’t support getting the hostname before negotiating SSL. But it could be difficult if you wanted to have one of the less common configurations such as having a mail server and a web server on the same IP addess but using different keys

How To Get Keys

deb http://mirror.internode.on.net/pub/debian/ testing main

The letsencrypt client is packaged for Debian in Testing but not in Jessie. Adding the above to the /etc/apt/sources.list file for a Jessie system allows installing it and a few dependencies from Testing. Note that there are problems with doing this, you can’t be certain that all the other apps installed will be compatible with the newer versions of libraries that are installed and you won’t get security updates.

letsencrypt certonly --standalone-supported-challenges tls-sni-01

The above command makes the letsencrypt client listen on port 443 to talk to the Lets Encrypt server. It prompts you for server names so if you want to minimise the downtime for your web server you could specify the DNS names on the command-line.

If you run it on a SE Linux system you need to run “setsebool allow_execmem 1” before running it and “setsebool allow_execmem 0” afterwards as it needs execmem access. I don’t think it’s a problem to temporarily allow execmem access for the duration of running this program, if you use KDE then you will be forced to allow such access all the time for the desktop to operate correctly.

How to Install Keys

[ssl:emerg] [pid 9361] AH02564: Failed to configure encrypted (?) private key www.example.com:443:0, check /etc/letsencrypt/live/www.example.com/fullchain.pem

The letsencrypt client suggests using the file fullchain.pem which has the key and the full chain of certificates. When I tried doing that I got errors such as the above in my Apache error.log. So I gave up on that and used the separate files. The only benefit of using the fullchain.pem file is to have a single line in a configuration file instead of 3. Trying to debug issues with fullchain.pem took me a lot longer than copy/paste for the 3 lines.

Under /etc/letsencrypt/live/$NAME there are symlinks to the real files. So when you get new keys the old keys will be stored but the same file names can be used.

SSLCertificateFile "/etc/letsencrypt/live/www.example.com/cert.pem"
SSLCertificateChainFile "/etc/letsencrypt/live/www.example.com/chain.pem"
SSLCertificateKeyFile "/etc/letsencrypt/live/www.example.com/privkey.pem"

The above commands are an example for configuring Apache 2.

smtpd_tls_cert_file = /etc/letsencrypt/live/smtp.example.com/cert.pem
smtpd_tls_key_file = /etc/letsencrypt/live/smtp.example.com/privkey.pem
smtpd_tls_CAfile = /etc/letsencrypt/live/smtp.example.com/chain.pem

Above is an example of Postfix configuration.

ssl_cert = </etc/letsencrypt/live/smtp.example.com/cert.pem
ssl_key = </etc/letsencrypt/live/smtp.example.com/privkey.pem
ssl_ca = </etc/letsencrypt/live/smtp.example.com/chain.pem

Above is an example for Dovecot, it goes in /etc/dovecot/conf.d/10-ssl.conf in a recent Debian version.

Conclusion

At this stage using letsencrypt is a little fiddly so for some commercial use (where getting the latest versions of software in production is difficult) it might be a better option to just pay for keys. However some companies I’ve worked for have had issues with getting approval for purchases which would make letsencrypt a good option to avoid red tape.

When Debian/Stretch is released with letsencrypt I think it will work really well for all uses.

No related posts.

Syndicated 2016-01-27 13:15:20 from etbe – Russell Coker

25 Jan 2016 amits   » (Journeyer)

SCaLE14x Talk: KVM Weather Report

I did a talk titled ‘KVM Weather Report‘ at the SCaLE conference in Pasadena, California yesterday.  The slides are available at this location.

The talk introduced the KVM stack (Linux, KVM, QEMU, libvirt); briefly went over some features and the communities around the projects, and discussed some of the new features added to the KVM stack in the last year.

Next up is my talk on live migration of VMs at FOSDEM in Belgium.

Syndicated 2016-01-25 13:18:21 (Updated 2016-01-25 13:19:46) from Think. Debate. Innovate.

23 Jan 2016 elwell   » (Journeyer)

Growatt inverter monitoring with Raspberry Pi

At home we have a small (2.5KW - 10*250w panels) PV system to try and offset our daytime electricity usage. This is connected to a 'Growatt' inverter that handily has both RS485 (wierd 2 pin plugs) and RS232 (9 pin D connector buried under a screwplate) outputs.

With the firmware on ours (installed Sept 2013) it supports modbus-rtu over serial 9600 8N1.

I had done some initial digging and experimentation (as announced on Whirlpool) but never really got sensible values out.When my guruplug (via a long USB to serial adaptor) finally died and I shelved the whole thing. With the completion of the structured wiring though I finally got round to reconnecting it and starting again.

Blue serial cable attached to structured wiring.
Small D9 Gender changer, + cisco console cable (all hail fleabay) gives a nice neat look on the outside, and in the garage I have another console cable plugged into the relevant patch outlet and a cheap usb-serial adaptor in a Raspberry Pi (which also has a GPS module connected, acting as a PPS NTP master)

Anyway, in the intervening time, someone had worked on my hacky scripts and wrapped the posting to PVoutput in an exec(curl) call -- first up I ripped that out and swapped for python requests.

I then went through the growatt modbus guide and made sure that it correctly calculated high and low byte values where these are split. The resulting script can be found on github,  and todays output can be seen on pvoutput. - a couple of charts are duplicated below.
Todays output v insolation prediction
As you can see, we had a couple of clouds going overhead today, so only generated 13KWh  vs 13.7 yesterday. Also the pvoutput fields are somewhat vague - 'Voltage' I've chosen to upload the array DC voltage rather than the grid AC volts (actually, I upload that and the grid frequency as extended data), and 'temperature' - I'd ideally like to have the panel temperatures, but upload the inverter temp so I can see if it's getting toasty. These can be seen on the 'all info' plot below

The observant of you will notice that the Etoday figure was slow to take off - this is because I didn't RTFM and discover that it's uploaded in watts, not kW...



Syndicated 2016-01-23 15:35:00 (Updated 2016-01-23 15:36:51) from Andrew Elwell

22 Jan 2016 marnanel   » (Journeyer)

Maths joke

It all began with a family trip to Königsberg...
Seven Bridges for Seven Brothers



This entry was originally posted at http://marnanel.dreamwidth.org/355089.html. Please comment there using OpenID.

Syndicated 2016-01-22 22:18:44 from Monument

21 Jan 2016 marnanel   » (Journeyer)

This is Godot



"This is Godot.

Godot is not here.

Godot is somewhere else.

Be like Godot.
Please."

This entry was originally posted at http://marnanel.dreamwidth.org/354824.html. Please comment there using OpenID.

Syndicated 2016-01-21 16:31:39 from Monument

21 Jan 2016 wingo   » (Master)

talks i would like to give in 2016

Every year I feel like I'm trailing things in a way: I hear of an amazing conference with fab speakers, but only after the call for submissions had closed. Or I see an event with exactly the attendees I'd like to schmooze with, but I hadn't planned for it, and hey, maybe I could have even spoke there.

But it's a new year, so let's try some new things. Here's a few talks I would love to give this year.

building languages on luajit

Over the last year or two my colleagues and I have had good experiences compiling in, on, and under LuaJIT, and putting those results into production in high-speed routers. LuaJIT has some really interesting properties as a language substrate: it has a tracing JIT that can punch through abstractions, it has pretty great performance, and it has a couple of amazing escape hatches that let you reach down to the hardware in the form of the FFI and the DynASM assembly generator. There are some challenges too. I can tell you about them :)

try guile for your next project!

This would be a talk describing Guile, what it's like making programs with it, and the kind of performance you can expect out of it. If you're a practicing programmer who likes shipping small programs that work well, are fun to write, and run with pretty good performance, I think Guile can be a great option.

I don't get to do many Guile talks because hey, it's 20 years old, so we don't get the novelty effect. Still, I judge a programming language based on what you can do with it, and recent advances in the Guile implementation have expanded its scope significantly, allowing it to handle many problem sizes that it couldn't before. This talk will be a bit about the language, a bit about the implementation, and a bit about applications or problem domains.

compiling with persistent data structures

As part of Guile's recent compiler improvements, we switched to a somewhat novel intermediate language. It's continuation-passing-style, but based on persistent data structures. Programming with it is interesting and somewhat different than other intermediate languages, and so this would be a talk describing the language and what it's like to work in it. Definitely a talk for compiler people, by a compiler person :)

a high-performance networking with luajit talk

As I mentioned above, my colleagues and I at work have been building really interesting things based on LuaJIT. In particular, using the Snabb Switch networking toolkit has let us build an implementation of a "lightweight address family translation router" -- the internet-facing component of an IPv4-as-a-service architecture, built on an IPv6-only network. Our implementation flies.

It sounds a bit specialized, and it is, but this talk could go two ways.

One version of this talk could be for software people that aren't necessarily networking specialists, describing the domain and how with Snabb Switch, LuaJIT, compilers, and commodity x86 components, we are able to get results that compete well with offerings from traditional networking vendors. Building specialized routers and other network functions in software is an incredible opportunity for compiler folks.

The other version would be more for networking people. We'd explain the domain less and focus more on architecture and results, and look more ahead to challenges of 100Gb/s ports.

let me know!

I'll probably submit some of these to a few conferences, but if you run an event and would like me to come over and give one of these talks, I would be flattered :) Maybe that set of people is empty, but hey, it's worth a shot. Probably contact via the twitters has the most likelihood of response.

There are some things you need to make sure are covered before reaching out, of course. It probably doesn't need repeating in 2016, but make sure that you have a proper code of conduct, and that that you'll be able to put in the time to train your event staff to create that safe space that your attendees need. Getting a diverse speaker line-up is important to me too; conferences full of white dudes like me are not only boring but also serve to perpetuate an industry full of white dudes. If you're reaching out, reach out to woman and people of color too, and let me know that you're working on it. This old JSConf EU post has some ideas too. Godspeed, and happy planning!

Syndicated 2016-01-21 11:59:18 from wingolog

21 Jan 2016 danstowell   » (Journeyer)

An executive summary of Islam in Britain

Just finished this really useful little book: "Medina in Birmingham, Najaf in Brent: Inside British Islam" by Innes Bowen. It could well be subtitled "An executive summary of Islam in Britain", because that's exactly what it feels like - a brief, breezy and dispassionate summary of the main Muslim groups in the UK, what they believe, how they interact with the world, etc.

Very handy reading, if you're a non-Muslim British person like me who might be wondering: the Muslims in my neighbourhood, are they sunni or shia? Does it matter? How do they relate to the various Muslim groups that are making the news these days? Which ones dress in special ways, and how significant is it? - All those naive questions that you can't just come out and ask.

All kinds of interesting stuff comes up while answering these questions. For example I learnt about the Tablighi Jamaat and why they wanted to build the "mega-mosque" that has been back and forth in the news trying to get planning permission. I learnt which groups have a voice in the Muslim Council of Britain. And even though the book doesn't spend much time on women's issues, it gives lots of titbits about different groups' conventions on veiling, staying in the house, marriage, and mosque provision - so it gives me some "local" insight to complement this other reading on veiling practices.

As in that other book, one thing that might surprise you is that some seemingly "traditional" things (like clothing practices) are borne of quite modern movements within Islam; really, you realise that "traditional" vs "modern" is not a particuarly helpful way to distinguish different strands of Islam practiced in Britain today.

Syndicated 2016-01-10 11:43:31 from Dan Stowell

21 Jan 2016 etbe   » (Master)

Finding Storage Performance Problems

Here are some basic things to do when debugging storage performance problems on Linux. It’s deliberately not an advanced guide, I might write about more advanced things in a later post.

Disk Errors

When a hard drive is failing it often has to read sectors several times to get the right data, this can dramatically reduce performance. As most hard drives aren’t monitored properly (email or SMS alerts on errors) it’s quite common for the first notification about an impending failure to be user complaints about performance.

View your kernel message log with the dmesg command and look in /var/log/kern.log (or wherever your system is configured to store kernel logs) for messages about disk read errors, bus resetting, and anything else unusual related to the drives.

If you use an advanced filesystem like BTRFS or ZFS there are system commands to get filesystem information about errors. For BTRFS you can run “btrfs device stats MOUNTPOINT” and for ZFS you can run “zpool status“.

Most performance problems aren’t caused by failing drives, but it’s a good idea to eliminate that possibility before you continue your investigation.

One other thing to look out for is a RAID array where one disk is noticeably slower than the others. For example if you have a RAID-5 or RAID-6 array every drive should have almost the same number of reads and writes, if one disk in the array is at 99% performance capacity and the other disks are at 5% then it’s an indication of a failing disk. This can happen even if SMART etc don’t report errors.

Monitoring IO

The iostat program in the Debian sysstat package tells you how much IO is going to each disk. If you have physical hard drives sda, sdb, and sdc you could run the command “iostat -x 10 sda sdb sdc” to tell you how much IO is going to each disk over 10 second periods. You can choose various durations but I find that 10 seconds is long enough to give results that are useful.

By default iostat will give stats on all block devices including LVM volumes, but that usually gives too much data to analyse easily.

The most useful things that iostat tells you are the %util (the percentage utilisation – anything over 90% is a serious problem), the reads per second “r/s“, and the writes per second “w/s“.

The parameters to iostat for block devices can be hard drives, partitions, LVM volumes, encrypted devices, or any other type of block device. After you have discovered which block devices are nearing their maximum load you can discover which of the partitions, RAID arrays, or swap devices on that disk are causing the load in question.

The iotop program in Debian (package iotop) gives a display that’s similar to that of top but for disk io. It generally isn’t essential (you can run “ps ax|grep D” to get most of that information), but it is handy. It will tell you which programs are causing IO on a busy filesystem. This can be good when you have a busy system and don’t know why. It isn’t very useful if you have a system that is used for one task, EG a database server that is known to be busy doing database stuff.

It’s generally a good idea to have sysstat and iotop installed on all systems. If a system is experiencing severe performance problems you might not want to wait for new packages to be installed.

In Debian the sysstat package includes the sar utility which can give historical information on system load. One benefit of using sar for diagnosing performance problems is that it shows you the time of day that has the most load which is the easiest time to diagnose performance problems.

Swap Use

Swap use sometimes confuses people. In many cases swap use decreases overall disk use, this is the design of the Linux paging algorithms. So if you have a server that accesses a lot of data it might swap out some unused programs to make more space for cache.

When you have multiple virtual machines on one system sharing the same disks it can be difficult to determine the best allocation for RAM. If one VM has some applications allocating a lot of RAM but not using it much then it might be best to give it less RAM and force those applications into swap so that another VM can cache all the data it accesses a lot.

The important thing is not the amount of swap that is allocated but the amount of IO that goes to the swap partition. Any significant amount of disk IO going to a swap device is a serious problem that can be solved by adding more RAM.

Reads vs Writes

The ratio of reads to writes depends on the applications and the amount of RAM. Some applications can have most of their reads satisfied from cache. For example an ideal configuration of a mail server will have writes significantly outnumber reads (I’ve seen ratios of 5:1 for writes to reads on real mail servers). Ideally a mail server will cache all new mail for at least an hour and as the most prolific users check their mail more frequently than that most mail will be downloaded before it leaves the cache. If you have a mail server with reads outnumbering writes then it needs more RAM. RAM is cheap nowadays so if you don’t want to compete with Gmail it should be cheap to buy enough RAM to cache all recent mail.

The ratio of reads to writes is important because it’s one way of quickly determining if you have enough RAM and adding RAM is often the cheapest way of improving performance.

Unbalanced IO

One common performance problem on systems with multiple disks is having more load going to some disks than to others. This might not be a problem (EG having cron jobs run on disks that are under heavy load while the web server accesses data from lightly loaded disks). But you need to consider whether it’s desirable to have some disks under more load than others.

The simplest solution to this problem is to just have a single RAID array for all data storage. This is also the solution that gives you the maximum available disk space if you use RAID-5 or RAID-6.

A more complex option is to use some SSDs for things that require performance and disks for things that don’t. This can be done with the ZIL and L2ARC features of ZFS or by just creating a filesystem on SSD for the data that is most frequently accessed.

What Did I Miss?

I’m sure that I missed something, please let me know of any other basic things to do – or suggestions for a post on more advanced things.

Related posts:

  1. Strange SATA Disk Performance Below is a GNUPlot graph of ZCAV output from a...
  2. Vibration and Strange SATA Performance Almost two years ago I blogged about a strange performance...
  3. New Storage Developments Eweek has an article on a new 1TB Seagate drive....

Syndicated 2016-01-21 03:05:01 from etbe – Russell Coker

21 Jan 2016 mjg59   » (Master)

Linux Foundation quietly drops community representation

The Linux Foundation is an industry organisation dedicated to promoting, protecting and standardising Linux and open source software[1]. The majority of its board is chosen by the member companies - 10 by platinum members (platinum membership costs $500,000 a year), 3 by gold members (gold membership costs $100,000 a year) and 1 by silver members (silver membership costs between $5,000 and $20,000 a year, depending on company size). Up until recently individual members ($99 a year) could also elect two board members, allowing for community perspectives to be represented at the board level.

As of last Friday, this is no longer true. The by-laws were amended to drop the clause that permitted individual members to elect any directors. Section 3.3(a) now says that no affiliate members may be involved in the election of directors, and section 5.3(d) still permits at-large directors but does not require them[2]. The old version of the bylaws are here - the only non-whitespace differences are in sections 3.3(a) and 5.3(d).

These changes all happened shortly after Karen Sandler announced that she planned to stand for the Linux Foundation board during a presentation last September. A short time later, the "Individual membership" program was quietly renamed to the "Individual supporter" program and the promised benefit of being allowed to stand for and participate in board elections was dropped (compare the old page to the new one). Karen is the executive director of the Software Freedom Conservancy, an organisation involved in the vitally important work of GPL enforcement. The Linux Foundation has historically been less than enthusiastic about GPL enforcement, and the SFC is funding a lawsuit against one of the Foundation's members for violating the terms of the GPL. The timing may be coincidental, but it certainly looks like the Linux Foundation was willing to throw out any semblance of community representation just to ensure that there was no risk of someone in favour of GPL enforcement ending up on their board.

Much of the code in Linux is written by employees paid to do this work, but significant parts of both Linux and the huge range of software that it depends on are written by community members who now have no representation in the Linux Foundation. Ignoring them makes it look like the Linux Foundation is interested only in promoting, protecting and standardising Linux and open source software if doing so benefits their corporate membership rather than the community as a whole. This isn't a positive step.

[1] Article II of the bylaws
[2] Other than in the case of the TAB representative, an individual chosen by a board elected via in-person voting at a conference

comment count unavailable comments

Syndicated 2016-01-20 23:21:53 from Matthew Garrett

19 Jan 2016 joey   » (Master)

git-annex v6

Version 6 of git-annex, released last week, adds a major new feature; support for unlocked large files that can be edited as usual and committed using regular git commands.

For example:

  git init
git annex init --version=6
mv ~/foo.iso .
git add foo.iso
git commit -m "added hundreds of megabytes to git annex (not git)"
git remote add origin ssh://sever/dir
git annex sync origin --content # uploads foo.iso

Compare that with how git-annex has worked from the beginning, where git annex add is used to add a file, and then the file is locked, preventing further modifications of it. That is still a very useful way to use git-annex for many kinds of files, and is still supported of course. Indeed, you can easily switch files back and forth between being locked and unlocked.

This new unlocked file mode uses git's smudge/clean filters, and I was busy developing it all through December. It started out playing catch-up with git-lfs somewhat, but has significantly surpassed it now in several ways.

So, if you had tried git-annex before, but found it didn't meet your needs, you may want to give it another look now.


Now a few thoughts on git-annex vs git-lfs, and different tradeoffs made by them.

After trying it out, my feeling is that git-lfs brings an admirable simplicity to using git with large files. File contents are automatically uploaded to the server when a git branch is pushed, and downloaded when a branch is merged, and after setting it up, the user may not need to change their git workflow at all to use git-lfs.

But there are some serious costs to that simplicity. git-lfs is a centralized system. This is especially problimatic when dealing with large files. Being a decentralized system, git-annex has a lot more flexability, like transferring large file contents peer-to-peer over a LAN, and being able to choose where large quantities of data are stored (maybe in S3, maybe on a local archive disk, etc).

The price git-annex pays for this flexability is you have to configure it, and run some additional commands. And, it has to keep track of what content is located where, since it can't assume the answer is "in the central server".

The simplicity of git-lfs also means that the user doesn't have much control over what files are present in their checkout of a repository. git-lfs downloads all the files in the work tree. It doesn't have facilities for dropping the content of some files to free up space, or for configuring a repository to only want to get a subset of files in the first place. On the other hand, git-annex has excellent support for alll those things, and this comes largely for free from its decentralized design.

If git has showed us anything, it's perhaps that a little added complexity to support a fully distributed system won't prevent people using it. Even if many of them end up using it in a mostly centralized way. And that being decentralized can have benefits beyond the obvious ones.


Oh yeah, one other advantage of git-annex over git-lfs. It can use half as much disk space!

A clone of a git-lfs repository contains one copy of each file in the work tree. Since the user can edit that file at any time, or checking out a different branch can delete the file, it also stashes a copy inside .git/lfs/objects/.

One of the main reasons git-annex used locked files, from the very beginning, was to avoid that second copy. A second local copy of a large file can be too expensive to put up with. When I added unlocked files in git-annex v6, I found it needed a second copy of them, same as git-lfs does. That's the default behavior. But, I decided to complicate git-annex with a config setting:

  git config annex.thin true
git annex fix

Run those two commands, and now only one copy is needed for unlocked files! How's it work? Well, it comes down to hard links. But there is a tradeoff here, which is why this is not the default: When you edit a file, no local backup is preserved of its old content. So you have to make sure to let git-annex upload files to another repository before editing them or the old version could get lost. So it's a tradeoff, and maybe it could be improved. (Only thin out a file after a copy has been uploaded?)

This adds a small amount of complexity to git-annex, but I feel it's well worth it to let unlocked files use half the disk space. If the git-lfs developers are reading this, that would probably be my first suggestion for a feature to consider adding to git-lfs. I hope for more opportunities to catch-up to git-lfs in turn.

Syndicated 2016-01-19 17:28:50 from see shy jo

19 Jan 2016 marnanel   » (Journeyer)

horsewhipping for Jesus

On 23 January 1867, the Rev. Edward Dodd, a fellow of Magdalene College Cambridge and the vicar of St Giles' church, was caught and publicly horsewhipped by the Rev. J Sumner Brockhurst, of Emmanuel College, as Dodd was leaving formal hall. When asked by a court to explain his actions, Brockhurst said that Dodd had said grace without mentioning the name of Jesus, because a Jewish man was present, and that any reasonable person would have whipped Dodd under such provocation. The court did not agree.

The "Saturday Review" said that this was the result of "muscular Christianity". This was a movement among certain evangelicals at the time, who were worried that religion in general was losing its focus on Jesus by trying to be nice to everyone. (I suspect there was a fair amount of misogyny mixed in: trying to accommodate people was seen as womanly, weak, and unworthy of men.)

Article in the "Spectator": https://books.google.co.uk/books?id=GU3hAAAAMAAJ&pg=PA124&redir_esc=y#v=onepage&q&f=false

Article in the "Saturday Review": https://books.google.co.uk/books?id=Z9UcAQAAMAAJ&pg=PA142&redir_esc=y#v=onepage&q&f=false

This entry was originally posted at http://marnanel.dreamwidth.org/354449.html. Please comment there using OpenID.

Syndicated 2016-01-19 12:41:05 (Updated 2016-01-19 12:44:40) from Monument

19 Jan 2016 wingo   » (Master)

unboxing in guile

Happy snowy Tuesday, hackfolk! I know I said in my last dispatch that I'd write about Lua soon, but that article is still cooking. In the meantime, a note on Guile and unboxing.

on boxen, on blitzen

Boxing is a way for a programming language implementation to represent a value.

A boxed value is the combination of a value along with a tag providing some information about the value. Both the value and the tag take up some space. The value can be thought to be inside a "box" labelled with the tag and containing the value.

A value's tag can indicate whether the value's bits should be interpreted as an unsigned integer, as a double-precision floating-point number, as an array of words of a particular data type, and so on. A tag can also be used for other purposes, for example to indicate whether a value is a pointer or an "immediate" bit string.

Whether values in a programming language are boxed or not is an implementation consideration. It can be the case that in languages with powerful type systems that a compiler can know what the representation of all values are in all parts of all programs, and so boxing is never needed. However, it's much easier to write a garbage collector if values have a somewhat uniform representation, with tag bits to tell the GC how to trace any pointers that might be contained in the object. Tags can also carry run-time type information needed by a dynamically typed language like Scheme or JavaScript, to allow for polymorphic predicates like number? or pair?.

Boxing all of the values in a program can incur significant overhead in space and in time. For example, one way to implement boxes is to allocate space for the tag and the value on the garbage-collected heap. A boxed value would then be referred to via a pointer to the corresponding heap allocation. However, most memory allocation systems align their heap allocations on word-sized boundaries, for example on 8-byte boundaries. That means that the low 3 bits of a heap allocation will always be zero. If you make a bit string whose low 3 bits are not zero, it cannot possibly be a valid pointer. In that case you can represent some types within the set of bit strings that cannot be valid pointers. These values are called "immediates", as opposed to "heap objects". In Guile, we have immediate representations for characters, booleans, some special values, and a subset of the integers. Alternately, a programming language implementation can represent values as double-precision floating point numbers, and shove pointers into the space of the NaN values. And for heap allocations, some systems can associate one tag with a whole page of values, minimizing per-value boxing overhead.

The goal of these optimizations is to avoid heap allocation for some kinds of boxes. While most language implementations have good garbage collectors that make allocation fairly cheap, the best way to minimize allocation cost is to refrain from it entirely.

In Guile's case, we currently use a combination of low-bit tagging for immediates, including fixnums (a subset of the integers), and tagged boxes on the heap for everything else, including floating-point numbers.

Boxing floating-point numbers obviously incurs huge overhead on floating-point math. You have to consider that each intermediate value produced by a computation will result in the allocation of another 8 bytes for the value and 4 or 8 bytes for the tag. Given that Guile aligns allocations on 8-byte boundaries, the result is a 16-byte allocation in either case. Consider this loop to sum the doubles in a bytevector:

(use-modules (rnrs bytevectors))
(define (f64-sum v)
  (let lp ((i 0) (sum 0.0))
    (if (< i (bytevector-length v))
        (lp (+ i 8)
            (+ sum (bytevector-ieee-double-native-ref v i)))
        sum)))

Each trip through the loop is going to allocate not one but two heap floats: one to box the result of bytevector-ieee-double-native-ref (whew, what a mouthful), and one for the sum. If we have a bytevector of 10 million elements, that will be 320 megabytes of allocation. Guile can allocate short-lived 16-byte allocations at about 900 MB/s on my machine, so summing this vector is going to take at least 350ms, just for the allocation. Indeed, without unboxing I measure this loop at 580ms for a 10 million element vector:

> (define v (make-f64vector #e10e6 1.0))
> ,time (f64-sum v)
$1 = 1.0e7
;; 0.580114s real time, 0.764572s run time.  0.268305s spent in GC.

The run time is higher than the real time due to parallel marking. I think in this case, allocation has even higher overhead because it happens outside the bytecode interpreter. The add opcode has a fast path for small integers (fixnums), and if it needs to work on flonums it calls out to a C helper. That C helper doesn't have a pointer to the thread-local freelist so it has to go through a more expensive allocation path.

Anyway, in the time that Guile takes to fetch one f64 value from the vector and add it to the sum, the CPU ticked through some 150 cycles, so surely we can do better than this.

unboxen, unblitzen

Let's take a look again at the loop to see where the floating-point allocations are produced.

(define (f64-sum v)
  (let lp ((i 0) (sum 0.0))
    (if (< i (bytevector-length v))
        (lp (+ i 8)
            (+ sum (bytevector-ieee-double-native-ref v i)))
        sum)))

It turns out there's no reason for the loquatiously-named bytevector-ieee-double-native-ref to return a boxed number. It's a monomorphic function that is well-known to the Guile compiler and virtual machine, and it even has its own opcode. In Guile 2.0 and until just a couple months ago in Guile 2.2, this function did box its return value, but that was because the virtual machine had no facility for unboxed values of any kind.

To allow bytevector-ieee-double-native-ref to return an unboxed double value, the first item of business was then to support unboxed values in Guile's VM. Looking forward to unboxed doubles, we made a change such that all on-stack values are 64 bits wide, even on 32-bit systems. (For simplicity, all locals in Guile take up the same amount of space. For the same reason, fetching 32-bit floats also unbox to 64-bit doubles.)

We also made a change to Guile's "stack maps", which are data structures that tell the garbage collector which locals are live in a stack frame. There is a stack map recorded at every call in a procedure, to be used when an activation is pending on the stack. Stack maps are stored in a side table in a separate section of the compiled ELF library. Live values are traced by the garbage collector, and dead values are replaced by a special "undefined" singleton. The change we made was to be able to indicate that live values were boxed or not, and if they were unboxed, what type they were (e.g. unboxed double). Knowing the type of locals helps the debugger to print values correctly. Currently, all unboxed values are immediates, so the GC doesn't need to trace them, but it's conceivable that we could have unboxed pointers at some point. Anyway, instead of just storing one bit (live or dead) per local in the stack map, we store two, and reserve one of the bit patterns to indicate that
the local is actually an f64 value.

But the changes weren't done then: since we had never had unboxed locals, there were quite a few debugging-related parts of the VM that assumed that we could access the first slot in an activation to see if it was a procedure. This dated from a time in Guile where slot 0 would always be the procedure being called, but the check is bogus ever since Guile 2.2 allowed local value slots corresponding to the closure or procedure arguments to be re-used for other values, if the closure or argument was dead. Another nail in the coffin of procedure-in-slot-0 was driven by closure optimizations, in which closures whose callees are all visible could specialize the representation of their closure in non-standard ways. It took a while, but unboxing f64 values flushed out these bogus uses of slot 0.

The next step was to add boxing and unboxing operations to the VM (f64->scm and scm->f64, respectively). Then we changed bytevector-ieee-double-native-ref to return an unboxed value and then immediately box it via f64->scm. Similarly for bytevector-ieee-double-native-set!, we unbox the value via scm->f64, potentially throwing a type error. Unfortunately our run-time type mismatch errors got worse; although the source location remains the same, scm->f64 doesn't include the reason for the unboxing. Oh well.

(define (f64-sum v)
  (let lp ((i 0) (sum 0.0))
    (if (< i (bytevector-length v))
        (lp (+ i 8)
            (let ((f64 (bytevector-ieee-double-native-ref v i))
                  (boxed (f64->scm f64)))
              (+ sum boxed))
        sum)))

When we lower Tree-IL to CPS, we insert the needed f64->scm and scm->f64 boxing and unboxing operations around bytevector accesses. Cool. At this point we have a system with unboxed f64 values, but which is slower than the original version because every f64 bytevector access involves two instructions instead of one, although the instructions themselves together did the same amount of work. However, telling the optimizer about these instructions could potentially eliminate some of them. Let's keep going and see where we get.

Let's attack the other source of boxes, the accumulation of the sum. We added some specialized instuctions to the virtual machine to support arithmetic over unboxed values. Doing this is potentially a huge win, because not only do you avoid allocating a box for the result, you also avoid the type checks on the incoming values. So we add f64+, f64-, and so on.

Unboxing the + to f64+ is a tricky transformation, and relies on type analysis. Our assumption is that if type analysis indicates that we are in fact able to replace a generic arithmetic instruction with a combination of operand unboxing, unboxed arithmetic, and a boxing operation, then we should do it. Separating out the boxes and the monomorphic arithmetic opens the possibility to remove the resulting box, and possibly remove the unboxing of operands too. In this case, we run an optimization pass and end up with something like:

(define (f64-sum v)
  (let lp ((i 0) (sum 0.0))
    (if (< i (bytevector-length v))
        (lp (+ i 8)
            (let ((f64 (bytevector-ieee-double-native-ref v i))
                  (boxed (f64->scm f64)))
              (f64->scm
               (f64+ (scm->f64 sum)
                     (scm->f64 boxed)))))
        sum)))

Scalar replacement via fabricated expressions will take the definition of boxed as (f64->scm f64) and fabricate a definition of f64 as (scm->f64 boxed), which propagates down to the f64+ so we get:

(define (f64-sum v)
  (let lp ((i 0) (sum 0.0))
    (if (< i (bytevector-length v))
        (lp (+ i 8)
            (let ((f64 (bytevector-ieee-double-native-ref v i))
                  (boxed (f64->scm f64)))
              (f64->scm
               (f64+ (scm->f64 sum)
                     f64))))
        sum)))

Dead code elimination can now kill boxed, so we end up with:

(define (f64-sum v)
  (let lp ((i 0) (sum 0.0))
    (if (< i (bytevector-length v))
        (lp (+ i 8)
            (let ((f64 (bytevector-ieee-double-native-ref v i)))
              (f64->scm
               (f64+ (scm->f64 sum)
                     f64))))
        sum)))

Voilà, we removed one allocation. Yay!

As we can see from the residual code, we're still left with one f64->scm boxing operation. That expression is one of the definitions of sum, one of the loop variables. The other definition is 0.0, the starting value. So, after specializing arithmetic operations, we go through the set of multiply-defined variables ("phi" variables) and see what we can do to unbox them.

A phi variable can be unboxed if all of its definitions are unboxable. It's not always clear that you should unbox, though. For example, maybe you know via looking at the definitions for the value that it can be unboxed as an f64, but all of its uses are boxed. In that case it could be that you throw away the box when unboxing each definition, only to have to re-create them anew when using the variable. You end up allocating twice as much instead of not at all. It's a tricky situation. Currently we assume a variable with multiple definitions should only be unboxed if it has an unboxed use. The initial set of unboxed uses is the set of operands to scm->f64. We iterate this set to a fixed point: unboxing one phi variable could cause others to be unbox as well. As a heuristic, we only require one unboxed use; it could be there are other uses that are boxed, and we could indeed hit that pessimal double-allocation case. Oh well!

In this case, the intermediate result looks something like:

(define (f64-sum v)
  (let lp ((i 0) (sum (scm->f64 0.0)))
    (let ((sum-box (f64->scm sum)))
      (if (< i (bytevector-length v))
          (lp (+ i 8)
              (let ((f64 (bytevector-ieee-double-native-ref v i)))
                (scm->f64
                 (f64->scm
                  (f64+ (scm->f64 sum-box)
                        f64))))
          sum-box)))

After the scalar replacement and dead code elimination passes, we end up with something more like:

(define (f64-sum v)
  (let lp ((i 0) (sum (scm->f64 0.0)))
    (let ((sum-box (f64->scm sum)))
      (if (< i (bytevector-length v))
          (lp (+ i 8)
              (f64+ sum
                    (bytevector-ieee-double-native-ref v i)))
          sum-box)))

Well this is looking pretty good. There's still a box though. Really we should sink this to the exit, but as it happens there's something else that accidentally works in our favor: loop peeling. By peeling the first loop iteration, we create a control-flow join at the loop exit that defines a phi variable. That phi variable is subject to the same optimization, sinking the box down to the join itself. So in reality the result looks like:

(define (f64-sum v)
  (let ((i 0)
        (sum (scm->f64 0.0))
        (len (bytevector-length v)))
    (f64->scm
     (if (< i len)
         sum
         (let ((i (+ i 8))
               (sum (f64+ sum
                          (bytevector-ieee-double-native-ref v i))))
           (let lp ((i i) (sum sum))
             (if (< i len)
                 (lp (+ i 8)
                     (f64+ sum (bytevector-ieee-double-native-ref v i)))
                 sum)))))))

As you can see, the peeling lifted the length computation up to the top too, which is a bonus. We should probably still implement allocation sinking, especially for loops for which peeling isn't an option, but the current status often works well. Running f64-sum on a 10-million-element packed double array goes down from 580ms to 99ms, or to some 25 or 30 CPU cycles per element, and of course no time in GC. Considering that this loop still has the overhead of bytecode interpretation and cache misses, I think we're doing A O K.

limits

It used to be that using packed bytevectors of doubles was an easy way to make your program slower using types (thanks to Sam Tobin-Hochstadt for that quip). The reason is that although a packed vector of doubles uses less memory, every access to it has to allocate a new boxed number. Compare to "normal" vectors where sure, it uses more memory, but fetching an element fetches an already-boxed value. Now with the unboxing optimization, this situation is properly corrected... in most cases.

The major caveat is that for unboxing to work completely, each use of a potentially-unboxable value has to have an alternate implementation that can work on unboxed values. In our example above, the only use was f64+ (which internally is really called fadd), so we win. Writing an f64 to a bytevector can also be unboxed. Unfortunately, bytevectors and simple arithmetic are currently all of the unboxable operations. We'll implement more over time, but it's a current limitation.

Another point is that we are leaning heavily on the optimizer to remove the boxes when it can. If there's a bug or a limitation in the optimizer, it could be the box stays around needlessly. It happens, hopefully less and less but it does happen. To be sure you get the advantages, you need to time the code and see if it's spending significant time in GC. If it is, then you need to disassemble your code to see where that's happening. It's not a very nice thing, currently. The Scheme-like representations I gave above were written by hand; the CPS intermediate language is much more verbose than that.

Another limitation is that function arguments and return values are always boxed. Of course, the compiler can inline and contify a lot of functions, but that means that to use abstraction, you need to build up a mental model of what the inliner is going to do.

Finally, it's not always obvious to the compiler what the type of a value is, and that necessarily limits unboxing. For example, if we had started off the loop by defining sum to be 0 instead of 0.0, the result of the loop as a whole could be either an exact integer or an inexact real. Of course, loop peeling mitigates this to an extent, unboxing sum within the loop after the first iteration, but it so happens that peeling also prevents the phi join at the loop exit from being unboxed, because the result from the peeled iteration is 0 and not 0.0. In the end, we are unable to remove the equivalent of sum-box, and so we still allocate once per iteration. Here is a clear case where we would indeed need allocation sinking.

Also, consider that in other contexts the type of (+ x 1.0) might actually be complex instead of real, which means that depending on the type of x it might not be valid to unbox this addition. Proving that a number is not complex can be non-obvious. That's the second way that fetching a value from a packed vector of doubles or floats is useful: it's one of the rare times that you know that a number is real-valued.

on integer, on fixnum

That's all there is to say about floats. However, when doing some benchmarks of the floating-point unboxing, one user couldn't reproduce some of the results: they were seeing huge run-times for on a microbenchmark that repeatedly summed the elements of a vector. It turned out that the reason was that they were on a 32-bit machine, and one of the loop variables used in the test was exceeding the fixnum range. Recall that fixnums are the subset of integers that fit in an immediate value, along with their tag. Guile's fixnum tag is 2 bits, and fixnums have a sign bit, so the most positive fixnum on a 32-bit machine is 229—1, or around 500 million. It sure is a shame not to be able to count up to #xFFFFFFFF without throwing an allocation party!

So, we set about seeing if we could unbox integers as well in Guile. Guile's compiler has a lot more visibility as to when something is an integer, compared to real numbers. Anything used as an index into a vector or similar data structure must be an exact integer, and any query as to the length of a vector or a string or whatever is also an integer.

Note that knowing that a value is an exact integer is insufficient to unbox it: you have to also know that it is within the range of your unboxed integer data type. Here we take advantage of the fact that in Guile, type analysis also infers ranges. So, cool. Because the kinds of integers that can be used as indexes and lengths are all non-negative, our first unboxed integer type is u64, the unsigned 64-bit integers.

If Guile did native compilation, it would always be a win to unbox any integer operation, if only because you would avoid polymorphism or any other potential side exit. For bignums that are within the unboxable range, the considerations are similar to the floating-point case: allocation costs dominate, so unboxing is almost always a win, provided that you avoid double-boxing. Eliminating one allocation can pay off a lot of instruction dispatch.

For fixnums, though, things are not so clear. Immediate tagging is such a cheap way of boxing that in an interpreter, the extra instructions you introduce could outweigh any speedup from having faster operations.

In the end, I didn't do science and I decided to just go ahead and unbox if I could. We are headed towards native compilation, this is a necessary step along that path, and what the hell, it seemed like a good idea at the time.

Because there are so many more integers in a typical program than floating-point numbers, we had to provide unboxed integer variants of quite a number of operations. Of course we could unconditionally require unboxed arguments to vector-ref, string-length and so on, but in addition to making u64 variants of arithmetic, we also support bit operations like logand and such. Unlike the current status with floating point numbers, we can do test-and-branch over unboxed u64 comparisons, and we can compare u64 values to boxed SCM values.

In JavaScript, making sure an integer is unboxed is easy: you just do val | 0. The bit operation | truncates the value to a uint32. In Guile though, we have arbitrary-precision bit operations, so although (logior val 0) would assert that val is an integer, it wouldn't necessarily mean that it's unboxable.

Instead, the Guile idiom for making sure you have an unboxed integer in a particular range should go like this:

(define-inlinable (check-uint-range x mask)
  (let ((x* (logand x mask)))
    (unless (= x x*)
      (error "out of range" x))
    x*))

A helper like this is useful to assert that an argument to a function is of a particular type, especially given that arguments to functions are always boxed and treated as being of unknown type. The logand asserts that the value is an integer, and the comparison asserts that it is within range.

For example, if we want to implement a function that does modular 8-bit addition, it can go like:

(define-inlinable (check-uint8 x)
  (check-uint-range x #xff))
(define-inlinable (truncate-uint8 x)
  (logand x #xff))
(define (uint8+ x y)
  (truncate-uint8 (+ (check-uint8 x) (check-uint8 y))))

If we disassemble this function, we get something like:

Disassembly of #<procedure uint8+ (x y)> at #xa8d0f8:

   0    (assert-nargs-ee/locals 3 2)    ;; 5 slots (2 args)
   1    (scm->u64/truncate 4 3)
   2    (load-u64 1 0 255)
   5    (ulogand 4 4 1)
   6    (br-if-u64-=-scm 4 3 #f 17)     ;; -> L1
;; [elided code to throw an error if x is not in range]
L1:
  23    (scm->u64/truncate 3 2)
  24    (ulogand 3 3 1)
  25    (br-if-u64-=-scm 3 2 #f 18)     ;; -> L2
;; [elided code to throw an error if y is not in range]
L2:
  43    (uadd 4 4 3)
  44    (ulogand 4 4 1)
  45    (u64->scm 3 4)
  46    (return-values 2)               ;; 1 value

The scm->u64/truncate instructions unbox an integer, but truncating it to the u64 range. They are used when we know that any additional bits won't be used, as in this case where we immediately do a logand of the unboxed value. All in all it's not a bad code sequence; there are two possible side exits for each argument (not an integer signalled by the unboxing, and out of range signalled by the explicit check), and no other run-time dispatch. For now I think we can be pretty happy with the code.

That's about it for integer unboxing. We also support unboxed signed 64-bit integers, mostly for use as operands or return values from bytevector-s8-ref and similar unboxed accessors on bytevectors. There are fewer operations that have s64 variants, though, compared to u64 variants.

summary

Up until now in Guile, it could be that you might have to avoid Scheme if you needed to do some kinds of numeric computation. Unboxing floating-point and integer numbers makes it feasible to do more computation in Scheme instead of having to rely in inflexible C interfaces. At the same time, as a Scheme hacker I feel much more free knowing that I can work on 64-bit integers without necessarily allocating bignums. I expect this optimization to have a significant impact on the way I program, and what I program. We'll see where this goes, though. Until next time, happy hacking :)

Syndicated 2016-01-19 11:57:53 from wingolog

19 Jan 2016 marnanel   » (Journeyer)

Disasters and retreats

I'm reading Orwell's essay "The Lion and the Unicorn", and this bit made me laugh out loud:

"English literature, like other literatures, is full of battle-poems, but it is worth noticing that the ones that have won for themselves a kind of popularity are always a tale of disasters and retreats. There is no popular poem about Trafalgar or Waterloo, for instance. Sir John Moore's army at Corunna, fighting a desperate rearguard action before escaping overseas (just like Dunkirk!) has more appeal than a brilliant victory. The most stirring battle-poem in English is about a brigade of cavalry which charged in the wrong direction."


This entry was originally posted at http://marnanel.dreamwidth.org/354073.html. Please comment there using OpenID.

Syndicated 2016-01-19 10:35:50 (Updated 2016-01-19 10:38:05) from Monument

19 Jan 2016 bagder   » (Master)

“Subject: Urgent Warning”

Back in December I got a desperate email from this person. A woman who said her Instagram had been hacked and since she found my contact info in the app she mailed me and asked for help. I of course replied and said that I have nothing to do with her being hacked but I also have nothing to do with Instagram other than that they use software I’ve written.

Today she writes back. Clearly not convinced I told the truth before, and now she strikes back with more “evidence” of my wrongdoings.

Dear Daniel,

I had emailed you a couple months ago about my “screen dumps” aka screenshots and asked for your help with restoring my Instagram account since it had been hacked, my photos changed, and your name was included in the coding. You claimed to have no involvement whatsoever in developing a third party app for Instagram and could not help me salvage my original Instagram photos, pre-hacked, despite Instagram serving as my Photography portfolio and my career is a Photographer.

Since you weren’t aware that your name was attached to Instagram related hacking code, I thought you might want to know, in case you weren’t already aware, that your name is also included in Spotify terms and conditions. I came across this information using my Spotify which has also been hacked into and would love your help hacking out of Spotify. Also, I have yet to figure out how to unhack the hackers from my Instagram so if you change your mind and want to restore my Instagram to its original form as well as help me secure my account from future privacy breaches, I’d be extremely grateful. As you know, changing my passwords did nothing to resolve the problem. Please keep in mind that Facebook owns Instagram and these are big companies that you likely don’t want to have a trail of evidence that you are a part of an Instagram and Spotify hacking ring. Also, Spotify is a major partner of Spotify so you are likely familiar with the coding for all of these illegally developed third party apps. I’d be grateful for your help fixing this error immediately.

Thank you,

[name redacted]

P.S. Please see attached screen dump for a screen shot of your contact info included in Spotify (or what more likely seems to be a hacked Spotify developed illegally by a third party).

Spotify credits screenshot

Here’s the Instagram screenshot she sent me in a previous email:

Instagram credits screenshot

I’ve tried to respond with calm and clear reasonable logic and technical details on why she’s seeing my name there. That clearly failed. What do I try next?

Syndicated 2016-01-19 08:37:32 from daniel.haxx.se

18 Jan 2016 dreier   » (Journeyer)

Just how to Improve Your Technical Skills

Mites’ research is named acarology.

The Response-to-Literature Essay At any time when I’ve presented this train to students, I have observed that some have difficulty in distinguishing in between a response-to-literature essay plus a summary essay. Know the primary difference Let us mention that you have concluded reading a novel. You may be presented two assignments on that one novel-its a bad desire, so never get worried. On the first of all assignment, you will be requested to present a summary; around the 2nd, you may have to put in writing a reaction. For your to begin with training, you will definitely summarize the plot inside your have words. When you drop by the write-up How Do I Be able to write a Summary Essay? you’ll see anything you need to do. For your second working out, you happen to be offering up a critique-your criticism-of the novel.

Just as the eighth century’s prophets b.c.

This doesn’t mean the response essay is totally not having a synopsis from the operate. It must give a temporary summary, particularly where it provides the background on the level or perhaps the concept that is the totally focus of your respective reaction. Whilst some instructors desire it, this kind of summarizing absolutely need not stand like a distinctive portion belonging to the essay. It can be woven into your examination to look as desired for the qualifications to the precise position that you’re doing while you produce your critique. Whereas a summary essay will clearly show your comprehension in the novel and its plot, the reaction essay need to display your very important analysis of the literary get the job done. Be described as a prepared scholar Regardless if you are crafting a summary essay or maybe a reaction essay, the geared up scholar is a single that’s within the practice of creating notes even when browsing literature. Use webs, charts, diagrams, maps, and tables for your notes.

Next part: briefly express what is the usp of one’s company.

For those who try this, you will see that it rather a lot less complicated to handle your assignment. At any time you make this happen as half of the regimen, regardless of whether no assignment is supplied at the end of the reading, you are likely to build-up a invaluable personal-development source. You are going to be improving your research skills and equipping you to manage initiatives in any area of life. Select your focal point In a very reaction essay, you’ll find a variety of parts that could acquire your focus. Some that you simply could publish about are the authors style; the authors purpose in penning this explicit get the job done; the track record matters that prompted the composing: historical, social, economic, and political dilemmas; the characterization; the symbolism used by the writer; the success of any foreshadowing inside of the tale; the figurative equipment utilized by the author to inform the tale also to carry out its further which means: simile, metaphor, alliteration, hyperbole, and so forth; a comparison among this novel and also other novels from the exact creator; a comparison somewhere between this novel in addition to other novels by other authors in the exact genre. It is wise to pick only one of those aspects and also to target on that time by generating a thesis statement and supporting it carefully all through your essay. When you are reviewing a non-fiction show results that is definitely dealing that has a practical challenge, you might wish to take into consideration regardless if the authors thesis has contributed usefully to the discussion and to the resolution of the predicament. Ask thoughts Inside distinct region that you have picked out to always be your focus, you have got to talk to various doubts: In this particular make any difference, lets say the qualifications dilemmas that prompted the writing, has the writer succeeded or failed? Are there weaknesses here or strengths within the the authors procedure of such matters?

We’re living together fundamentally as roommates.

Is there clarity or is it missing? Does it provide enlightenment about very much the same worries at the moment? Could the author have handled the subject additional correctly? Are there other is effective of the author where this specified position gets even better (or worse) treatment solution? Are there other writers from the comparable genre who definitely have quite possibly dealt with this point in a significantly better way? By asking these types of inquiries and working on what ever investigation is critical to find the answers, it is possible for you to to grow a essential response to literature. Not surprisingly, you are able to make this happen provided that you’ve got check out the work with attention to its detail and when you have grasped its message. The more you can learn in the authors other will work in addition to of publications in the equivalent style by other authors, the better equipped you need to be to provide an acceptable, intellectual reaction. Your reaction isn’t really intended to generally be just a description of the way you actually feel regarding the novel.

Locate a bible-reading plan to follow.

You may comprise of that ingredient, as expected, but it really types merely a small-scale piece of one’s in general reaction. In my view, the response-to-literature essay seriously isn’t a beginners work out. It truly is for students who definitely have experienced exposure with the writings of various authors in varied genres plus much more than a particular get the job done of each of individuals authors. Reaction essays really don’t deal only with literature. Assignments may be provided to reply to plays and films, but on this publish and its sequel we’ve been concerned only with literature.

Tweet

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

Advogato User Stats
Users13995
Observer9881
Apprentice745
Journeyer2335
Master1030

New Advogato Members

Recently modified projects

5 Feb 2016 Abigail
29 Dec 2015 mod_virgule
20 Jul 2015 Justice4all
25 May 2015 Beobachter
7 Mar 2015 Ludwig van
18 Dec 2014 AshWednesday
11 Nov 2014 respin
20 Jun 2014 Ultrastudio.org
13 Apr 2014 Babel
13 Apr 2014 Polipo
19 Mar 2014 usb4java
8 Mar 2014 Noosfero

New projects

5 Feb 2016 Abigail
2 Dec 2014 Justice4all
11 Nov 2014 respin
8 Mar 2014 Noosfero
17 Jan 2014 Haskell
17 Jan 2014 Erlang
17 Jan 2014 Hy
17 Jan 2014 clj-simulacrum
17 Jan 2014 Haskell-Lisp
17 Jan 2014 lfe-disco
17 Jan 2014 clj-openstack
17 Jan 2014 lfe-openstack
17 Jan 2014 LFE
1 Nov 2013 FAQ Linux
15 Apr 2013 Gramps