Stevey: We're sorry about the lack of documentation - lack of resources mean that documentation has taken a bit of a back seat. We will have documentation of the fileformat for 1.0 though.
If you don't mind looking at code - take a look at vorbis/lib/info.c: _vorbis_unpack_comment - while it isn't documentation, it should explain the format of the comment header (header number 2 - which is packet number 2) pretty clearly. The ogg layer is fully documented. We're happy to help out if you ask on the list or find us in #vorbis on OPN. Please do so.
For an example of parsing a number of important things from vorbis files without using libvorbis (it DOES use a significantly cut down version of libogg, but you could reimplement the bits of that needed seperately if you wanted to), you could look at the vorbis support in gnapster (it's possibly a bit out of date, I haven't looked at it since writing it, but it should be largely correct)