I've recently been working on improving SBCL's build behaviour.
Well, perhaps that's not exactly a novel or surprising piece of information for this diary: but maybe this is the last major effort that has to be made. One of SBCL's raisons d'être is to be a Common Lisp compiler, written in Common Lisp, that does not have any dependencies on the host Lisp compiler used to build it (assuming that the host implementation is sufficiently conforming, but not making any unwarranted assumptions about any implementation-defined or undefined behaviour). This method for building a Lisp compiler is sufficiently tricky to wrap one's head around that I wrote and delivered a paper describing it, including diagrams, for the Workshop on Self-Sustaining Systems last year.
So what's new now? Well, given this aim, it would be reasonable for the output of the SBCL cross-compiler, running as an application in the host Lisp, to produce bitwise-identical output files independent of which host Lisp was being used. Reasonable, perhaps, but in practice the state of SBCL until a month or so ago was quite a long way away from this ideal: every output file had a header declaring when and from which compiler it had been generated, which was not going to help bitwise comparisons: and even after such straightforward issues had been dealt with, every single output file from the build process (of which there are of the order of 350) exhibited non-trivial differences when compiled with CLISP from the corresponding output files compiled with SBCL.
The differences between output files that I've observed and fixed in the last month or so – by lavish use of cmp(1), staring at emacs buffers consisting largely of control characters, and the time-old debugging method of Thinking Very Hard – can be broadly split into three categories:
- outright leaks: information being taken from the
compiler and erroneously treated as though it was applicable
target. Sometimes this was simple carelessness (the
constant-folding for symbols used symbol-value,
for things like most-positive-fixnum);
sometimes it was nastier than that (CLISP and SBCL disagree
value of (log 2d0
in the last binary digit). It's not nice to find these
showing up, but at least this set of changes should make
stick out like a sore thumb if they're ever reintroduced (or
present on other platforms...)
- traversal order: some operations have a net
deterministic, but the order in which sub-operations was
collecting a bunch of definitions by iterating over a hash
or a set of
storage classes with union, is going
correct no matter the order, but different between different
implementations of those functions. Also in this class,
strictly speaking an issue of traversal order, was the
the host's interpretation strategy: whether forms were
or not, which influences the number of times gensym is
hence the value of *gensym-counter*.
- idiosyncrasies: working towards eliminating the
differences of interpretation in the standard. Most of these
differences of opinion are legitimate: CLISP macroexpands
SBCL does because it has an interpreter on by default, for
similarly, CLISP's compiler doesn't aggressively coalesce
whereas SBCL's does. Some of them are less so; CLISP, for
prints 'foo as "'FOO" whereas SBCL prints
"(QUOTE FOO)" when *print-pretty*
is nil. I think the standard mandates SBCL's
behaviour, but for these purposes we have to work
In the process, I also fixed a terribly embarrassing bug in genesis, the Lisp application responsible for taking these output files from the cross-compiler and constructing a Lisp memory image file ready to start up. As has been discussed here in the past, the standard-mandated requirement on arrays in Common Lisp is not sufficient for us to be able to construct the image file in memory as an array of bytes: conforming implementations are permitted to impose a maximum array size as low as 1024 elements. Fortunately it is straightforward to implement a suitable data structure without this constraint; unfortunately, the implementation did not take sufficient care to zero-fill newly-allocated memory, and while most of the time Lisp implementations perform that zeroing, there are circumstances in which they don't.
All of this has now been merged to SBCL's CVS, and is awaiting the 1.0.28 release due on Thursday 30th April. The practical upshot? Well, apart from a certain amount of increased confidence in the implementation strategy, and perhaps soothing the nerves of extremely paranoid distribution package maintainers, these changes should make compilation of SBCL on new platforms more straightforward, as there are now two implementations capable of building SBCL which are themselves buildable starting just from gcc and system libraries: CLISP and Peter Graves' XCL.