Advogato blog for mikal

Bad science: Ben Goldacre vs Rentokil, a twitter timeline

Sun, 14 Mar 2010 21:11:14 GMT

Beyond This Horizon

Sun, 7 Mar 2010 11:22:06 GMT

ISBN: 0743435613
Baen (2002), Mass Market Paperback, 256 pages
LibraryThing

This book is only the second book I have failed to finish in the last few years. This is Heinlein's second novel, and that's the problem -- its clumsily written in a style which moves at a lethargic pace. Its also pretty badly dated, and possesses that creepy EE "Doc" Smith eugenics theme that is so disturbing. I made it to page 67 before I stopped caring.

Tags for this post: book() Robert_A_Heinlein()
Related posts: Triplanetary

Comment

Books read in February 2010

Thu, 4 Mar 2010 04:07:09 GMT

Tags for this post: book() read()

Comment

The Colour of Magic

Thu, 4 Mar 2010 04:07:09 GMT

ISBN: 0061020711
LibraryThing

There isn't much point in writing a review of this book, because it seems that you'd have to have been living under a rock to not have read it. On the other hand, I feel this book isn't as good as people make out. Its not as funny as Douglas Adams, it jumps around, and the plot is sketchy at best. Overall, not Pratchett's best work.

Tags for this post: book() Terry_Pratchett()
Related posts: Better Than Life; Bill the Galactic Hero Series; Life, the Universe and Everything; The Restaurant at the End of the Universe; So Long, and Thanks For All the Fish; The Man in the Rubber Mask; Red Dwarf; Colony; Last Human; Bill the Galactic Hero on the Planet of Bottled Brains; Mostly Harmless; Bill The Galactic Hero; The Hitchhikers Guide to the Galaxy; Bill the Galactic Hero: The Planet of the Robot Slaves

Comment

Burning Chrome

Sun, 28 Feb 2010 09:11:27 GMT

ISBN: 0441089348
Ace Books (1986), Mass Market Paperback, 208 pages
LibraryThing

This is a collection of William Gibson short stories. They're gritty and real, and make more sense now that I've been to both the US and Japan. I read these stories as a kid and loved them, although their vision for the future isn't a happy one. Its good to see I still like them as an adult. Stories in this collection:

Johnny Mnemonic
The Gernsback Continuum
Fragments of a Hologram Rose
The Belonging Kind
Hinterlands
Red Star, Winter Orbit
New Rose Hotel
The Winter Market
Dogfight
Burning Chrome

Tags for this post: book() William_Gibson()
Related posts: Isaac Asimov's Robot Short Stories; Robot Visions; Galactic Dreams; Isaac Asimov's Nightfall Short Stories; The Complete Hammer's Slammers Volume 1; Body Armor: 2000; The Complete Robot; Nightfall (short stories); I, Robot; Bolos 4: Last Stand; The Complete Hammer's Slammers Volume 2; The Complete Hammer's Slammers Volume 3; Bolos 2: The Unconquerable; Bolos 1: Honor of the Regiment; The Rest of the Robots; Bolos 6: Cold Steel; Battlefields Beyond Tomorrow; Dogs of War; Bolo!; Bolos 3: The Triumphant; The Compleat Bolo

Comment

Bolo Series

Fri, 26 Feb 2010 03:07:11 GMT

Created by Keith Laumer and then followed on by many others, this series was very entertaining whilst not being fine literature.

1986: The Compleat Bolo by Keith Laumer (includes Bolo, and Rogue Bolo)
1990: The Stars Must Wait by Keith Laumer
1993: Bolos Book I: Honor of the Regiment by S.M. Stirling, S.N. Lewitt, J. Andrew Keith, Todd Johnson, Mike Resnick, Barry N Malzberg, Christopher Stasheff, Mercedes Lackey, Larry Dickson and David Drake.
1994: Bolos Book II: The Unconquerable by S.M. Stirling, Shirley Meier, Christopher Stasheff, Karen Wehrstein, Todd Johnson, and William R. Forstchen
1995: Bolos Book III: The Triumphant by David Weber and Linda Evans
1997: Bolos Book IV: Last Stand by S.M. Stirling, David Weber, William R. Forstchen, Robert Greenberger, William H Keith Jr, Todd Johnson, Steve Perry, John Decamp, Mark Thies, and Linda Evans
2001: Bolos Book V: Old Guard by Mark Thies, John Mina, J. Steven York, Dean Wesley Smith and William H. Keith Jr
2002: Bolos Book VI: Cold Steel by J. Steven York, Dean Wesley Smith, and Linda Evans
1997: Bolo Brigade by William H. Keith Jr
1998: Bolo Rising by William H. Keith Jr
2002: Bolo Strike by William H. Keith Jr
2006: Bolo! by David Weber
2005: The Road to Damascus by John Ringo and Linda Evans
2007: Old Soldiers"> by David Weber

Tags for this post: book() Keith_Laumer()
Related posts: Bolo Strike; Bolos 4: Last Stand; The Stars Must Wait; Bolos 2: The Unconquerable; Bolos 1: Honor of the Regiment; Bolo Brigade; Bolos 6: Cold Steel; Battlefields Beyond Tomorrow; The Road to Damascus; Bolo Rising; Bolo!; Bolos 3: The Triumphant; The Compleat Bolo; Bolos 5: Old Guard; Old Soldiers; Bill the Galactic Hero Series; First Family; On Basilisk Station; Death Bringer; The Complete Hammer's Slammers Volume 1; Body Armor: 2000

Comment

Old Soldiers

Wed, 24 Feb 2010 11:20:21 GMT

ISBN: 1416521046
Baen (2007), Mass Market Paperback, 400 pages
LibraryThing

This is a more traditional bolo book than The Road to Damascus, which means I enjoyed it more. I'm a little sad that its the last one of the series, but there is another one coming soon (entitled Their Finest Hour due out in August) which gives me something to hang out for. This book gives some nice closure to operation Ragnarok, whilst not being too predictable. Overall I enjoyed it.

Tags for this post: book() Keith_Laumer()
Related posts: Bolo!; Bolos 3: The Triumphant; Bolo Strike; Bolos 4: Last Stand; The Stars Must Wait; Bolos 2: The Unconquerable; Bolos 1: Honor of the Regiment; Bolo Brigade; Bolos 6: Cold Steel; Battlefields Beyond Tomorrow; The Road to Damascus; Bolo Rising; The Compleat Bolo; Bolos 5: Old Guard; Bill the Galactic Hero Series; First Family; On Basilisk Station; Death Bringer; The Complete Hammer's Slammers Volume 1; Body Armor: 2000; Without Warning

Comment

The Road to Damascus

Sun, 21 Feb 2010 06:08:25 GMT

ISBN: 0743499166
Baen (2005), Mass Market Paperback, 768 pages
LibraryThing

I didn't like this book. Bolo books should be about combat and human interactions with AI machines, which this book starts nicely with. However, the book then diverts from that and spends hundreds of pages ranting about how bad socialism is. Sure, the baddies are nasty, but its not a very believable rendition of communist Russia. Worse, the story is told with a whiney teenager I found deeply annoying as a character (she's meant to be annoying, but I just didn't care about her). Finally, after about 400 pages of such drivel, we finally get around to having some more actual bolo story. The finish of the story is ok, I just wished it didn't take so long to get there. This is by far the weakest Bolo book I have read so far.

Tags for this post: book() Keith_Laumer()
Related posts: Bolos 4: Last Stand; Bolos 3: The Triumphant; Bolo Strike; The Stars Must Wait; Bolos 2: The Unconquerable; Bolos 1: Honor of the Regiment; Bolo Brigade; Bolos 6: Cold Steel; Battlefields Beyond Tomorrow; Bolo Rising; Bolo!; The Compleat Bolo; Bolos 5: Old Guard; Bill the Galactic Hero Series; First Family; On Basilisk Station; Death Bringer; The Complete Hammer's Slammers Volume 1; Body Armor: 2000; Without Warning; Blood River

Comment

Dragon's Egg

Thu, 11 Feb 2010 21:12:44 GMT

ISBN: 034528349x
LibraryThing

This book was strongly recommended by a friend, and has been sitting on my shelf waiting to be read for a while. Its my first real taste of "hard" science fiction, and I enjoyed it. The story is surprisingly fast moving given it has to cover the entire development of an intelligent species, and the process is interesting. There are strong parallels with our own past, which I guess isn't that surprising a thing for an author to do. I've seen complaints that the human characters in this book are a bit one dimensional, but I think that's unfair. The book isn't about the humans, and they are there only to progress the plot. I think the human characters are fine. If you want to see poor character development, checkout Robot City and Robots and Aliens series.

Tags for this post: book() Robert_L_Forward()
Related posts: Isaac Asimov's Robot City: Robots and Aliens: Humanity; Isaac Asimov's Robot City: Robots and Aliens: Maverick; Isaac Asimov's Robot City: Robots and Aliens: Alliance; The Robot City, Robots and Aliens Series; Isaac Asimov's Robot City: Robots and Aliens: Intruder; Isaac Asimov's Robot City: Robots and Aliens: Changeling; Isaac Asimov's Robot City: Robots and Aliens: Renegade

Comment

Methodology for my SMTP survey

Wed, 10 Feb 2010 00:11:01 GMT

Most of the questions raised by yesterday's initial post about my SMTP survey were about methodology. Unfortunately, the questions are all on Facebook, which means they're not publicly visible and I'm not keen on making other peoples comments visible to y'all without permission. However, I already have a lot written about the methodology which I can include here. Hopefully it will either address most questions, or lead us off in an interesting direction.

I'll start off by talking about the other surveys I have discovered, although it should be noted that I became aware of another, possibly larger survey yesterday. I have some questions about that new survey that I will address today, so let's ignore it for this discussion until I know more. The Bernstein surveys are the oldest SMTP survey data available. These surveys have used a variety of methodologies for selecting servers to probe -- the first survey selected 500,000 random IP addresses from a combination of a DNS walk, glue records from zone files, and MX lookups from zone files, the second survey similarly used 200,000 random addresses from a DNS walk, the third survey performed an MX lookup on .net domains, and then probed the resulting IP addresses, the fourth survey did the same as the third, but only looked up 1/256th of .com, the fifth survey looked up PTR records for 1 million random IP addresses, and then attempted to connect to those which had a valid PTR record, and finally the sixth survey followed the same selection methodology as the fifth.

The Credentia surveys, which are more recent, have probed relatively small random samples of the IP address space. This has resulted in extremely small numbers of actual SMTP servers being reported on, which makes these surveys less reliable than those with larger samples. The MailChannels survey, which was run at the time that I started developing my own survey implementation, used a list of 400,000 domain names registered by large companies. This has resulted in a selection bias in their results towards solutions favored by large for profit companies.

All of the surveys I have seen so far use the same method to determine the SMTP software in use -- the 220 status line is logged, and then run through a series of rules to attempt to determine what software is running. Its surprising how many implementations will just tell you what they are, and some even include the version of the software. Even if they don't tell you outright, often they use slightly different banner strings than other implementations, which can be used as a fingerprint. All of the various survey implementations use their own rule sets, although I am willing to publish mine if anyone is interested. I also today became aware of smtpscan, which is a perl script that implements the same form of fingerprinting. I need to dig through its source and see how its rules compare with mine.

(As an aside, I was keeping complete logs of my scans until I had a very embarrassing disk failure a few months ago. Whilst I no longer have complete logs, I do have complete sets of the collected data, which makes it possible for me to tweak the detection rules I use and then rerun the processing to see how that changes the results over time.)

Onto my target selection methodology. My survey tracks IP addresses, not domain names. This means if many domain names are hosted on a single IP address, then there is still only one result counted for those domain names. This probably skews my results away from large installations, because they often have many servers sitting behind a single shared IP address to handle load. In that case, only one of the servers in their cluster will be scanned and counted by my survey. One challenge with my probing technique is that it is not possible to simply scan all 4,294,967,296 IPv4 addresses. Instead, it is important to only scan IP addresses likely to be running a SMTP server. To that end, I have experimented with six techniques to determine candidate IP addresses for scanning:

Entries in the RBL (http://dsbl.org/main) and CBL (http://cbl.abuseat.org/) real time spam black lists were surveyed in the April 2008, July 2008 and October 2008 surveys. These blacklists contain lists of IP addresses suspected to run open relays or mail proxies, and are aggregated into one list of IP addresses per source. At its peak, there were 105 million IP addresses derived with this technique.
IP addresses were also collected from the received headers an email collection consisting of emails sent to Gmane (http://www.gmane.org), mail-archive.com, and my personal mail boxes. Overall, 1,101,066 IP addresses were collected with this method.
A number of domain registrars make their zone files available to researchers. These zone files contain lists of domains currently registered in their top level or second level domains. I currently have access to the zone files for .com, .net, .root, .root, .arpa, .mobi, and .asia. These zone files are used to perform Domain Name System queries for Mail Exchanger (MX) entries. These entries are used by domains to advertise to email senders which servers handle email for a given domain. The IP addresses of these servers are then added to the list to be probed as part of my survey. So far over 95 million domains have been scanned in this manner, resulting in 2.1 million unique mail server IP addresses. It should be noted that there is some overlap between the mail servers discovered by my RBL analysis, and the mail servers identified by this MX lookup. The table below provides a comparison of the various zone files we have used over time. This is important, as .com and .net are significantly larger than the other zones.
One of the limitations of the MX lookups described above is that I do not possess a complete list of all domains registered on the Internet. This is because many domain registrars do not make their DNS zone files available for analysis. I have therefore developed two other methods for locating mail exchangers. The first is to use the DNS reverse lookup results for IP addresses I have surveyed -- these DNS names are recursively queried for MX records. The second technique performs DNS reverse lookups on IPs on the boundary of IP allocations, on the theory that an IP allocation is likely to share a common domain name across its IPs. The results of these lookups are then recursively queried for MX records as well.

Here's that table of zone file sizes as promised in a bullet point above:

Date	arpa	asia	com	mobi	net	root
19 Dec 2007	5	...	71,545,277	783,199	10,660,216	281
29 Mar 2008	5	112,366	74,996,409	892,916	11,174,045	281
14 Jul 2008	5	194,824	76,312,322	928,736	11,591,319	280
29 Sep 2008	5	227,451	77,661,007	954,888	11,822,954	280
1 Jan 2009	5	241,131	78,701,644	866,590	11,960,758	280
1 Apr 2009	5	244,164	80,324,644	840,019	12,170,629	280
2 Oct 2009	5	212,054	82,259,663	858,503	12,433,457	280

Again, there is much more to say here -- for example a discussion of the effectiveness of each IP discovery technique would be useful. However, I'm once again over my 1,000 word limit for a single post, so I'll leave that discussion for later.

Tags for this post: research()
Related posts: Measuring the popularity of SMTP server implementations on the Internet; Initial SMTP survey poster results in a pie chart; Interesting paper: "YouTube Traffic Characterization: A View From the Edge"; RemoteWorker v74; Microsoft Exchange the most popular SMTP server on the Internet?; I think I've worked out the problem with the hotel network; Mikal, the massive domain squatter; Internet traffic; Domain name lookup helper for python?; Normalising mail server package names; Satellite internet at Walmart; Long time not much write; RemoteWorker v70; Announcing early results of my survey of SMTP servers; Noticed that smtpsurvey.stillhq.com is down?; Mikal, tell something I didn't know about SMTP servers on the Internet; The witty worm with Vern Paxson; Why does every man and his dog put man pages online?; Sensis Australian search

Comment