This is the monthly report after the usual battles fought in cyberspace and beyond!
Third encounter with AIX
Mauri was playing with hanging puppets and Marti was already eating Spaghetti, but we were all waiting my father for supper on friday evening. The phone rang once, I called back but it was busy. It turned out that my father phoned, however he had to stop the phone call as someone from IBM support finally called him. He explained me that he had a strange black box in a corner of a huge hall full with wine boxes (each with 12 bottles of prestigious wine!) which refused to boot up. On the black box one could read IBM 520, and closer there was a label saying "Type 8203-E4A N/S G5-208 B4". The server went down on friday morning at 9:45 when an electrician doing maintenance work removed fuses to work safely. The whistling UPS run unheard for a while, and the black beast died right after UPS finished its juice.
The whole company could not work without this server through friday. (on which a DHCP server, a file server and a billing software run), so my father decided to keep trying and to skip supper. I could not resist when he mentioned the box was running AIX, so dangermouse mounted in his car and drove to the huge wine hall equipped with ananas pieces, biscuits and the two laptops virgibuntu and andromeda. Intermezzo for the fans: the Calibra car lost the front spoiler in a close encounter with the guard rail, which explains the broken blizzard sign, too.
But back to the problem: the local network was not lying neither on the usual 192.168.x.x IP range, nor on 10.x.x.x. It was between 172.16.x.x and 172.31.x.x which was unusual setting for us, and the confusion was great. On the LCD panel of the IBM 520 one could find out that it was started in modality A which was sort of diagnostic modality. The box did not have neither keyboard nor screen and was silent like the monolith of Space Odissey.
It had a strange HMS port, but we did not know what HMS was. Either the manual of 500 pages we found in Internet about HMS forgot to introduce what HMS was, or we were so tired that could not figure out.
The next day, when the kind technician of IBM came, we learned how to use HMS to revive the server, and learned about the smitty utility, and why one of the hard disk never lighted. For short time, I had the root password of the box, but when I tried to login to have a ride, the software owner already changed it. It is the third time I encounter an AIX, but never had the chance to login and discover its inners! In the end, our job reduced to get the printers back up and running, and to add a samba network name to the host file, which for some reason the resurrected blackbox did not want to resolve. When we left, I glanced one last time at the misterious IBM 520 box, and I felt like it was laughing.
I discovered that Virtualbox 4 has much better network support. I used this fact to setup a Fedora Linux 16 with the Sage libraries, a phantastic collection of mathematical software. Additionally, I found out that GPU libraries worked well under Linux, especially the Synapse support made by Lukas Gebauer. There is still some hope for the version II of GPU then! And from the outside, this Fedora thing should appear like a phantom computer on its network segment!
There were two lessons I learned with this puzzle: the first one is "let others help you"! In fact, I could not even solve the first level of the puzzle, while Virgi made it up to level 27. She explained me the rules of the game (we could not see them as we were logged in), and also some of the heuristics she was using, and to which mathematical problem it reduced (this can be found in the forum as well). I had a bad time passing the first level after that, but finally managed to do it! contagious told me that he used an external library to solve the Crossflip challenge. As my brute force solver barely brought me to level 60 and I could not find a way to implement Posa search, I decided to go with external libraries to defeat OneOfUs. I tried out many, including one of an author who made the claim P=NP. He was so kind to provide the C++ implementation, which you can't say of most of so called scientific authors. Still, his library did up to level 460, but then started going very slow. I teamed up with userfriendly, sharing what we discovered on the problem. After our briefing, he then skyrocketed to level 1000.
I kept stucked at level 460 knowing all what userfriendly knew. I worked through the night, but could not find the mistake in my solver implementation. I supposed an obscure error in the POST implementation of Synapse (the submitting part of the solver), but could not find any error after I created a test page to see what I submitted. Defeated, I went to sleep. I thought on all what I observed (including the fact that the "P=NP" solver in its slowlyness passed level 460) and then had the Zen flash, which each hacker.org puzzle hides. It turns out that my solver was in fact correct and it was solving the problem, just it was solving another problem which was not the one expected by hacker.org! I had a nasty mistake in the board generation for level over 242, but this mistake showed up only on bigger boards! The second lesson learnt is "if you think your code is right, it is. Look somewhere else!". For the record, the last level was solved in 20 seconds. The whole story can be seen here, when charting will work again.
Cheat Engine is an amazing Open Source tool which you can attach to any running process. Among other things, it identifies where integer variables are held in memory, so that you can modify them while you are playing. This helped me out a lot in a challenge I could not disassemble. I also tried "What in the World" without luck, but knowing what the sequences probably are. And failed on "Slim Image", too. On hacker.org, I often feel at the limit of what my mind can do and also what current software and hardware can do, which is a good introduction for the next section :-)
I read the Geometric Visions article of Michael Crawford, which is a great article not only about mental disease, but also about philosophy and about how people mind works in general.
I only had acoustic allucinations once, when i was living in a place close to a noisy rail station where i could not sleep. i never had problems with depressions or so, also because i can't focus on details, i do not care about details (which can be also a problem in life :-)
To tell the difference between voices and real life, i used Occam's rasor principle: if my reasoning started getting too complex, and one easier explanation (which might be hurting to myself) is easier, then i decided to believe the easier explanation and to fix my behaviour in positive ways. I found out that most of the time, complex reasoning was just an excuse to hide my misbehaviour. Occam's principle worked very well through my whole life.
I found the OpenFlashChart library for PHP and I am now pepping up deltasql with charts. The hope is that such chart experiments will help the GPU project as well!
As final links the history of Commodore International and of its founder, and a note on the strange Kaprekar numbers.
That's all folks!
for f in $FILES
if [ $f != "./resize.sh" ]
echo "Resizing $f file down to $1 per cent..."
# take action on each file. $f store current file name
convert $f -resize $1% "./resized/$f"
These are the latest from my hackerland journey. I feared that either me or the challenge site would go down, and in fact hacker.org is now temporarily unavailable since three days. Maybe it is under a Denial of Services attack of some user who is angry because he can't solve some challenge... Who knows?
Close to the last castle
With 190 challenges solved, I am close to the last castle on the upper left corner of the map. The challenges around it are very hard, none of them is solved, and after there will be a super-hard challenge anyway to protect this castle.
Being a dangermouse it was reasonably easy to solve some quests of the serie "Really Small Mouse". I brought the mouse down to 21 instructions, but there are some hackers who can do it in 20 and less! Cool challenges encountered on the road were 'No full ACK in SEPT' which refreshed modular algebra a bit, 'Maelstrom' to train image recognition, 'HVM cipher' and 'Really Simple Access' to revive my reminescences of cryptanalysis. While solving "Shattered and Shuffled" and "Shredded and Scrambled", I felt cold down my neck, it was the breath of the dark side who strucked me.
I suffered deceiving in "Such Much" and only an answer on the forum brought me back on track. I almost brought down hacker.org by submitting huge numbers calculated with Python! I attacked with brute force 'Soviet Intercept' and I almost locked me out of this challenge; now I have to wait one hour between submitting of answers for this particular challenge, though now I prepared a sort of key which should allow the identification of the spies.
For the cryptographic challenges involving unkown codes, I developed a tool to perform the kappa-test, and another one to perform the chi-test on vigeneres which have substitutions in columns instead of simple caesar ciphers. I wrote them according to this article. I found a good tool to perform first image analysis on Steganographic challenges here made by Caesum, the author of the Challenger's handbook. To perform coprime factorization I definitely recommend MSieve. Other discoveries were Binary Coded Decimals (BCD) and Setun, a Russian computer which was working with ternary balanced circuits :-)
deltasql 1.4.2 and beyond
1.4.2 was a tactical release, to get hands on code which I did not touch since three months and also to advertise deltasql through the Mantra of Open Source Release often, release early. The development of 1.4.3 is struggled by technical accidents (the server I am using to test went down, because the friend of mine who kindly hosts deltasql forgot to pay the bill). Additionally, I face now trouble pushing my code with git to sourceforge. But 1.4.3 will be fresh wind with several minor bugfixes, user preferences and the ability to inform users of new scripts via sendmail.
Finally, to remember myself on how to
Set default operating system in Grub
1. edit DEFAULT variable in /etc/default/grub
2. run sudo grub-update
There are many challenge sites on the net, and one to unify them all.
I decided to focus on hacker.org as there are many high quality challenges involving programming, cryptography and maths, and a bit less of ugly exploits and similar. The site organizes challenges on a map. Harder challenges are on mountains, easier ones on grass. Each time a challenge is solved, it opens the road to new challenges. Castles protect challenges that are on crossroads. Winning against a fortified castle opens new possibilities and new areas to be discovered.
From easy to hard or the other way round
Sometimes i get stuck solving a harder challenge. At first time, it looks like i am wasting my time, but the point is that if by chance i manage to solve it, several other challenges suddenly get much easier.
This was the case with the Labyrinth challenge, which forced me to develop new techniques for the HVM. In the end, I invested more than one month of time on it. Once Labyrinth was solved, I could solve the similar Deluge and the sequel "Flash Flood warmup" in one day. Other challenges like "King Mouse", "Brokenest Keys" and "Mus Minutoides" finally found a solution through new ugly techniques (mainly calls with no returns and what i call the statistical approach, to spoil them a bit).
To sharpen my weapons further, I put Caesum's Challengers handbook in my journeyer's backpack. And here I found a good explanation of the chi and kappa test, so that I added the index of coincidence to the magic spells I am aware of.
It was a great satisfaction to besiege the castle protected by 'Spiral bits'! The path following algorithm did his job well, but the decoding part was a complete nightmare, so that i turned the Lazarus GUI into a complete editor for inserting, deleting and flipping bits. I also had an integrated lens on board and a tool to compare the decoding on slightly different paths (which did not work very well though). And a good bit of luck was also part of solving it!
I then succesfully attacked 'No full ACK in HEX' and 'No full ACK in DEC', learning Knuth arrow notation, the tower of powers, and in particular learning Python's ability of computing with large numbers, which is unrivalled. Maple couldn't do what Python did. Now I know why some numeric people I worked with were so happy with Python :-)
And still, there is work to do. My armies failed several times against 'Blizzard' and 'Descartes'. The 'Captcha' prototype needs further refinement as well...
(current ranking on hacker.org is 188 with 180 challenges solved/on Khan 200 challenges solved and Copernicus badge reached, next goal is one million points)
Mauri is born! Marti now has a brother and we are all very happy to welcome him in our family :-) Might the source be with you, Luke.
In the following, some updates on our lone hackerland journey:
Probably some smart people go straight to the correct solution. For me, I walked through all stages of enlightment. I first developed a maze parser in Frepascal, then got a brute force solver in place which could solve up to about maze level 120. The solver started taking long times for each new level (up to 6 hours). I am a bit out of gear right now as we moved house, lost connection to civilization for some time, and still have to reactivate the server room, which will move from pavement to cellar. I remembered that spartacusII server got burned in a similar challenge and decided to upgrade the algorithm as soon as possible. With dynamic programming I restricted a bit the solution space. A first attempt with combinations failed, I tried to debug the way I create combinations, but could not find the mistake. I rewrote the algorithm and got a working solver for levels up to 153. Then again, the solver was taking too much time. Finally, I got a sort of Zen flash and found a way to retrieve the solution, as it would be just popping out from the maze. The new solver could solve any maze in less than a second, and easily moved up to level 513. When it solved level 514 (a level noone has solved), and was waiting the one minute delay to avoid server pounding before the solution is submitted, I got very proud. Too much proud, as the server answered: "Your solution sucked, maximum level for this puzzle is 513". This site is amazing, it does not forget to teach you to be modest :-) The whole story can be seen in this graph as well.
I then decided to attack another puzzle, Mortal Coil. Adapting the parser and getting a brute force solver in place was easy, as Runaway prepared the path. But now I am stuck at level 68. I found papers on the Hamiltionian path problem, which is NP-complete and now need to think on how to divide and conquer it.
On the Hack Virtual Machine, I worked out the meaning of the two commands ^ and v and such knowledge lead to some progress: by analyzing the solution of Execution Style, I finally cracked Hello Small World! Also revision 34 of my Labyrinth program could solve the three mazes, though it required several submissions. In fact, the limit of 40'000 cycles is very tight. The version which solved the challenge could solve the reference maze I was using only in about 53'000 cycles. The Labyrinth program is based on a wall following algorithm published in Nievergelt's book about Algorithms and datatypes.
I found in the depths of Internet an almost complete map of the challenges and it is clear now that to achieve some breakthrough, I need to work out Spiral Bits. The encoding is a bit hard, but by printing out the spiral on a piece of A3 paper, it is possible to work it out with some patience. The pathfinder solver I developed follows the spiral but keeps staying on the side, so I have to think a way to balance it a bit. To debug the breadcrumbs left by the program, I use this magnifying glass.
On Khan Academy, I am close to the Copernicus Badge, but the Atlas Black Hole badge moved again out of range from 250 to 500 challenges...
Some other news
Unrelated to the previous topic, a deltasql patch to branch from existing tags for 1.4.1 is published here.
And do not forget to check the Wikileaks Spy Files!
Let me finally point at a news on the Voyager probes measuring Lyman-alpha radiation (here). Amazing piece of irons, the Voyager space probes!
emerge -av tcpdump # to monitor traffic
emerge -av openntpd # to synchronize the clock with ntpd
emerge -av tor # tor itself
torgencert # generates a certificate for your tor node
ntptime # to adjust your clock
ExitPolicy accept *:22 # ssh
ExitPolicy accept *:465 # smtps (SMTP over SSL)
ExitPolicy accept *:993 # imaps (IMAP over SSL)
ExitPolicy accept *:994 # ircs (IRC over SSL)
ExitPolicy accept *:995 # pop3s (POP3 over SSL)
ExitPolicy accept *:6660-6697 # allow irc ports, very widely
ExitPolicy reject *:* # no other exits allowed
ExitPolicy accept *:22 # ssh
ExitPolicy accept *:465 # smtps (SMTP over SSL)
ExitPolicy reject *:* # no other exits allowed
tcpdump -A | grep -i "WHATYOUAREINTERESTEDIN" > captured .txt
New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.
Keep up with the latest Advogato features by reading the Advogato status blog.
If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!