Older blog entries for brouhaha (starting at number 74)

Finally squashed a nasty bug. I thought I'd previously managed to fix the calculator state restoration bug, but it turns out that the bug only manifested on 32-bit machines and I was trying to debug it on a 64-bit machine. There was a call to strtoul() that should have been strtoull().

It's unfortunate that the C standard doesn't specify any replacement for those that is based on the actual size of the type (e.g., uint32_t, uint64_t) rather than the weakly defined ``unsigned long'' and ``unsigned long long''. I suppose the correct thing to do would either be to have a configuration test at build time to pick the right one, or to use sscanf() instead, since the C standard does provide size-appropriate format specifier strings in <inttypes.h>. Perhaps I'll add a strtouint64() function in my utility package, based on sscanf().

lcdtest release 1.01 has a new ``dots'' test pattern, and an on-screen help message.

Nonpareil

Release 0.63 of Nonpareil now simulates the HP-11C, HP-12C, and HP-15C calculators.

When I heard that Fedora Core 2 was going to include natively-compiled Eclipse, I was excited since I'd wanted to try Eclipse for quite a while but had never been able to make it work. Unfortunately the native Eclipse didn't actually make it into FC2 or even FC3, but it is part of the forthcoming FC4.

I'm running Fedora Core 4 Test 3 on an Athlon 64 3500+ with 1 GB of RAM, so I thought I'd give the native Eclipse a try, along with CDT for C development. I haven't done much with it yet, but the feature set seems quite nice. I use SCons instead of make, so I wanted to try the SConsBuilder plugin, but there isn't yet a version for Eclipse 3. I installed the Subclipse plugin for use with the Subversion configuration management system, but I haven't yet figured out how to use it.

The big problem is that Eclipse is mind-numbingly slow. Even just right-clicking on a source file takes a long time to bring up a context menu. If it's this slow when natively compiled on an Athlon 64, I can't even imagine what running a non-native version must be like. How can anyone find this useful?

Finally a new release of Nonpareil

I released version 0.61 of Nonpareil last night. Now it has support for the HP-16C "Computer Scientist" calculator, and due to bug fixes the HP-34C, HP-37E, HP-38E, and HP-38C finally work correctly.

The big news is that there is finally support for saving and restoring the calculator state, so there is finally "Continuous Memory", even for calculators that didn't actually have that as a hardware feature.

The state save uses compressed XML files thanks to libxml2.

This is also the first release that builds with SCons instead of make.

I had to solve a tricky memory corruption bug to get this release ready. One of my data structures was being corrupted. I used GDB to set a hardware watchpoint on a field of the structure, but the watchpoint never tripped, even though the field did get changed! Possibly a problem with 64-bit GDB in Fedora Core 4 test 2.

I've been fixing bugs in Nonpareil, and improving the SCons build files I use instead of Makefiles.

I've finally got the 14/15-digit display mode of the Woodstock processor (ACT) working correctly, as far as I can tell. That's used in the HP-19C, HP-67, and the HP-9x printing desktop calculators. I'm trying to get the HP-67 running.

The HP-67 normally uses the "KEYS -> A" instruction to read the keyboard, then does fairly complicated stuff to the value in A to determine where to branch. Earlier calculators just used "KEYS -> ROM", which directly replace the low eight bits of the program counter with the hardware keycode (not the same as the user-visible keycode). I was not able to figure out what the hardware keycodes should be from Tony Duell's reverse-engineered schematics, which is not his fault, but rather because I don't have enough documentation on the ACT. I tried to figure out what they should be by studying a microcode disassembly, and by trial and error, but it was very slow progress and it wasn't easy to figure out the correct codes because many incorrect codes map to normal operations. I finally decided that the best thing to do was to write a test program to run on the actual calculator. I don't currently have a means to do that, but I wrote the program and tested it on the simulator. As each key is pressed, it displays the hardware keycode as a three-digit octal number.

Bernhard Engl build a EPROM-based ROM emulator and has it interfaced to an HP-67 in place of the normal ROM chips, so I sent him my code. He burned it into EPROMs and ran it, and we were both surprised that it worked correctly the first time. He was able to determine all of the hardware keycodes and email them to me, and when I plugged them into the KML file and fixed a few minor bugs, the HP-67 started partially working in simulation. Bernhard says that he'll send me some photos of his hardware setup.

I've studied a lot of disassembled Woodstock code, but that's the first Woodstock code I've written. Just reading and studying the code others have written before wasn't enough to give me a full appreciation for how challenging it can be, due to the non-orthagonality of the registers. I'm thinking about writing a game in microcode to get more practice. Too bad most people will never be able to play it except on the simulator.

Currently the HP-67 simulation works until you do anything that requires the calculator to search for a label in program memory. The search routine enters an infinite loop. The loop is entered with P=1, and each iteration through the loop P gets incremented by 2. The maximum value of P is 13, beyond which it wraps to zero, so the next loop value after 13 is 1. However, the test for the end of the loop tests for P=0. An engineering spec for the ACT states that when P wraps on either an increment or decrement, the position "disappears" for one cycle. I suspect that this vague description is wrong, and that in this particular case P happens to have a transient value of 0 at the time the comparison is made. It's possible that this behavior is different than that of the earliest ACT chips; it is known that some revisions of the HP-67 and HP-97 microcode are not compatible with all ACT chips, and perhaps this could be why.

Typical SCons "Builders" take one source file and produce one target file. To define a rule that produces multiple target files, it is necessary to write an "Emitter". As part of the Nonpareil build process, the uasm microassembler gets built, then uasm is used to assemble microcode for several calculator models. Although it's not strictly necessary, I want the assembly to result in listing files as well as object files. I tried to write an Emitter for that months ago, based on the examples in the SCons documentation, and didn't get very far. Now that I know a tiny bit more Python, I hacked on that a bit more, and now it's basically working. The remaining problem is that I want the listing files to go into the obj subdirectory where the object files go, but it's actually creating another obj directory under that for them, so that they are in paths like obj/obj/hp45.lst.

It turns out that I'm not the only person that has been surprised that SCons doesn't come with good support for building source code distribution tarballs. Only recently this has come up on the scons-users mailing list, and Paul Davis posted a nice source tarball Builder. I tried to get it working with the Nonpareil SConstruct and SConscript files, and get an error I don't understand. So I created a simpler test case, and it works for that. Seems to be something about the way my hierarchy works. I hope to have it figured out soon, since I want to package a new source release.

This morning I was trying to figure out why Nonpareil isn't coming anywhere close to simulating the HP-67 correctly. It comes up with a single decimal point lit in the display, and the only visible responses to the keyboard are for the decimal point to move to another position or disappear.

So I captured an execution trace from the simulator, and started studying it. Almost immediately, I found that they are doing a ``DELAYED ROM SELECT'' instruction preceding a ``A -> ROM'' instruction.

On the real hardware, I don't know whether DELAYED ROM SELECT is only delayed by one instruction, or whether it is delayed until the next branch. When I wrote the Woodstock CPU code, I implemented it as being delayed until the next branch. But I forgot to handle it with the ``GOTO KEYS'' and ``A -> ROM'' instructions, which are computed branches based on the keycode and A register, respectively.

I also determined that opcode 0120 octal is almost certainly the ``KEYS -> A'' instruction.

Adding KEYS->A and fixing the DELAYED ROM SELECT did not fix the HP-67 problem. But it did fix the problems with the HP-34C, HP-37E, HP-38E, and HP-38C, all of which now seem to work correctly!

Thanks to some help from HrastProgrammer, the HP-11C and HP-16C are now fully functional in the Nonpareil simulator. The display annunciators don't work yet, but everything else seems fine.

Recently I've mostly been working on improvements to the simulation infrastructure. I've now got working code to save the state of any simulated calculator to a disk file as a gzip-compressed XML file, and to restore from such a file, so Nonpareil will finally have continuous memory. :-)

There's still some remaining work on making the save/restore work nicely with the File menu, and making it automatically save on exit and restore on startup. And after a restore, I need to force a display update.

Dealing with compressed XML using libxml2 is very easy. It's a great library.

I decoded the "hex table" of the HP-16C. There are only 243 normal user programmable operations, so there are 13 illegal codes. I haven't yet tried executing the illegal codes, but I did wedge them into a save state file to see how they would be decoded while stepping through memory in program mode. Of these, two are "WINDOW 8" and "WINDOW 9", four duplicate "FLOAT 6" through "FLOAT 9", five are "FLOAT B" through "FLOAT F", and two duplicate "STO . 0" and "0".

I also spent some time matching up math routines in the HP-11C code with the VASM listings of the HP-41C math routines (file cn6b). They are almost identical, which isn't too surprising.

I put together a crude register display window using the pseudo-reflection API I described previously. It uses a GtkTable with two columns, with GtkLabels in the left colum for the register names, and GtkEntry fields in the right column. I set the text width of the GtkEntry fields to the maximum number of characters the register can display as (i.e., 14 for full-word 56-bit registers in hexadecimal). By putting a GtkEntry directly into a table column, it forces all of them to be the same width, which I don't want.

For the stack, which is an array of four 16-bit values, I put a GtkHbox into the table, and put four GtkEntry fields into that. These entry fields still are wider than I would like, but they don't fill up the whole width of the table column. Hmmm... maybe I should put all of the entry fields into Hboxes, even when there's only a single one on the line.

Anyhow, the GtkEntry is OK for a prototype, but it's not what I want in the longer term.

I think I need to add some flags to the register description returned by pseudo-reflection, so that certain registers can be displayed in a special manner. For instance, it would be nice for the P and Q registers which point to specific digit positions in the word registers to be shown visually as pointers at the appropriate horizonal position.

Also, some registers are not significant unless execution is stopped in the middle of a multiple-cycle instruction, so they should not normally be shown (or saved).

groom: I've been thinking about the clunky SysVinit orgnaization for a while, and I think the entire current scheme should be done away with. There should be one text file per service. That text file should list what runlevels the service is supposed to run in, and what other services it depends on (requires to be running first). Then init can just do a dependency analysis (ala make) to determine what order to start things in, what things can be started concurrently (make -j), and what order to shut things down in.

Only the top-level services desired in a run level would need to be explicitly enabled, and the services they depend on would be inferred. Though there could be a setting to explicitly prohibit a service from running in a particular runlevel, which would also prevent things depending on it from running in that runlevel.

I've been wanting to prototype this for quite a while, but I haven't gotten a Round Toit yet.

I've implemented what I call ``pseudo-reflection'' in Nonpareil. Each processor architecture has a unique C structure defined which represents the architecturally visible register state of a simulated processor, and in order for general-purpose state save/restore and debugging access to be possible, the GUI thread needs to be able to ask the simulator thread what fields (registers) exist in the structure, their characteristics (name, element size, array bounds), and function calls to read and write those registers.

If I was writing this in Java, Smalltalk, or Lisp, I could use a native reflection API provided by the language to get that information. In C and C++ there is no equivalent, although there do exist some incredibly kludgy hacks to do it in C++.

Anyhow, fundamentally all that is needed is to declare an array of ``meta structures'': structures that define the characteristics of the fields of the structures of interest. I've now done almost completely for the Nut (HP-41C) architecture, and for a few registers of the Classic and Woodstock architectures. I've implemented the API that the simulator thread will use: sim_get_register_info(), sim_read_register(), and sim_write_register().

sim_get_register_info() takes a numeric index (from 0 up) and returns a pointer to a substructure containing public information about a register. Since the data is static, it can execute in any thread, and will generally execute in the GUI thread.

sim_read_register() and sim_write_register() are contending for access to the registers with the simulation thread, so to get coherenet reads and writes, there needs to be some mutual exclcusion. This could be a lock per register or a big lock for the whole structure, but either way, it would require a *very* large number of code sequences in the simulator thread to acquire and release locks, which is not a good idea. So instead, these functions send a request through the async message queue to tell the simulator thread that the GUI (or debugger) thread wants to read or write a register, and the work is done in the simulator thread. This is how a lot of other simulator interface functions already work.

The next steps will be to attach the XML state save/restore code to this API, and to build a register display window that uses it as well.

I've written a DTD for an XML Nonpareil state save file format, and code to read and write such files using libxml2. It was very easy. It's not yet integrated into the main Nonpareil program, because it will need the new memory API and pseudo-reflection. In the mean time, I wrote a utility that can convert the XML state save file format to and from the old NSIM simulator's state save file format.

XML is more verbose than the NSIM format, but because the XML files are automatically compressed on the fly, the new state files end up being 25 to 50% of the size they use in NSIM format.

Rather than developing another ugly little language to save and restore calculator state in Nonpareil, I decided that I'd try using xml, via the libxml2 library, which I've never used before. The last time I wrote any software to do anything with XML was six years ago, when I hand-coded routines to emit and parse XML. That was a pain. I was pleased to discover that libxml2 provides fairly clean, easy-to-use APIs. For parsing, you can either let it build an entire parse tree in memory (DOM style), or give it callback functions to invoke as it reads the various XML elements, attributes, entities, and user data (SAX style). For Nonpareil, SAX is clearly the right thing.

For writing XML files, the simplest API provided by libxml2 is xmlTextWriter. This particular API appears to only be suited to single-threaded use, but that's fine for Nonpareil. (Nonpareil has multiple threads, but only one will ever be reading or writing a particular XML file at any given time.)

It's also nice that there is support for doing gzip compression and decompression on the fly.

It only took me three hours to come up with a suitable DTD, study the libxml2 documentation and write and debug a mockup consisting of one program that writes a Nonpareil save state file and one that parses it.

In fact, this worked so well that I'm considering replacing the use of (modified) KML files for configuring Nonpareil with another XML file. In this case, I'd probably want the XML file to come out of a ZIP file, so I'd have to write a little bit of glue to get the libxml2 parser to read from a gsfInput stream.

lcdtest

Judging by the Freshmeat stats, lcdtest is way more popular than I ever would have guessed. There must have been some unsatisfied demand for a test pattern generator. Perhaps I'll build Windows and MacOS X binaries for it as well. There already are some similar programs for those platforms, but perhaps there's some utility in having a common cross-platform program, and anyhow, since I used SDL it should be easy.