The Moby Commander Keen File Format Reference ============================================= Last updated: Th Mar 13 2003 TBD: Must comment on differences between explicit files and data structures... perhaps give info on how structures can be recognised within files? and give known offsets (relative to exe header). Structure diagrams (created for readability instead of rigor): [ block | block | ... ] represents a sequence of blocks that make up a file or another block. (block) represents an optional data block . {block} represents a block that can be repeated one or more times {block * n} represents a block that is repeated n times {block * n-m} represents a block that is repeated from n to m times {block (comment)} represents a block with a comment indicating that the block needs to be preprocessed in some way before it can be processed. TBD: Consider how to specify which versions the block applies to. ===================================================================================================================================== :GAMEMAPS: (4, 5, 6) [ GAMEMAPS Header | {Level Data} ] GAMEMAPS stores the maps for all the levels in Commander Keen, including the world map and the high scores level. The data is the complete contents of the GAMEMAPS.CK? file. :GAMEMAPS Header: [ Signature | Junk ] Signature: Byte[8] Gives the version of TED5 used to create the file. Always "TED5v1.0". Keen does not check for this. Junk: Byte[] Zero or more junk bytes (see below). The presence of Junk is due to a bug in TED5. Here is a simple equivalent this bug: char mapidstr[8] = "TED5v1.0"; ... fwrite( mapidstr, strlen( mapidstr ), 1, file ); As you can see, there is no terminating null in the array, so TED5 just keeps writing junk until it finds one in memory. :Level Data: [ {Compressed Plane Data * 3} | Level Header ] :Level Header: [ Plane Offsets | Plane Lengths | Width | Height | Name | Signature ] Plane Offsets: Long[3] Offset within GAMEMAPS to the start of the plane. The first offset is for the background plane, the second for the foreground plane, and the third for the info plane (see below). Plane Lengths: Word[3] Length (in bytes) of the compressed plane data. The first length is for the background plane, the second for the foreground plane, and the third for the info plane (see below). Width: Word Level width (in tiles). Height: Word Level height (in tiles). Together with Width, this can be used to calculate the uncompressed size of plane data, by multiplying Width by Height and multiplying the result by sizeof(Word). Name: Byte[16] Null-terminated string specifying the name of the level. This name is used only by TED5, not by Keen. Signature: Byte[4] Marks the end of the Level Header. Always "!ID!". Levels use three planes: background, foreground, and info. The background plane contains the tile numbers used for the background of the level. The foreground plane contains the tile numbers for the foreground of the level, which the player interacts with. The info plane contains sprite numbers; some goodies; links for switches and doors; platform blocking and directing tiles; and other special information. :Compressed Plane Data: [ Length | RLE Plane Data (compressed) ] Length: Word Size (in bytes) of RLE Plane Data block when it is uncompressed. The RLE Plane Data block is compressed with the so-called Carmack compression algorithm. The Carmack algorithm removes repeated patterns of words, and replaces each with a reference to the first instance of the pattern. It has two reference types, near and far. A near reference consists of three bytes: the number of bytes in the pattern, xA7, and the relative offset (in words) of the start of the first pattern instance from the start of the reference, counting backwards. Thus the reference x03 xA7 x06 effectively means "repeat the first 3 of the last 6 words". A far reference consists of two bytes and a word, and is used to reference patterns that are more than 255 words from the reference. The first byte is again the length of the pattern. It is followed by xA8, and a word specifying the absolute offset of first pattern instance from the start of the uncompressed data. If a word in the uncompressed input stream has a high byte of xA7 or xA8, it is be replaced in the compressed data by a near or far reference (as appropriate) with the pattern length being zero, and the offset byte being the low byte of the input word. (TBD: Check the following: Note that a zero-length far reference will only have a byte, not a word, following xA8) :RLE Plane Data: [ Length | Plane Data (compressed) ] Length: Word Size (in bytes) of Plane Data block when it is uncompressed. The Plane Data block is compressed with a word-based RLE algorithm. The algorithm takes any runs of 4 or more identical words (treating the Plane Data as a single sqeuence of words) and replaces them with three words: a magic word to identify a run, the number of words in the run, and the word that the run consisted of. The magic word used is given in :MAPHEAD:. If the magic word is encountered in the input stream, it is replaced with a one-word- long run of that word. :Plane Data: [ {Row} ] Row: Word[] A single row of tiles in the plane. The size of the row is given by Width in the Level Header, and the number of rows is specified by Height in the Level Header. The tiles in the row are stored leftmost-first, and rows are stored topmost-first. ===================================================================================================================================== :MAPHEAD: (4, 5, 6, D) [ Magic Word | {Level Offset * 100} ] Magic Word: Word The magic word use by the RLE level compression algorithm that identifies a run. See :RLE Plane Data:. Level Offset: Long The absolute offset of the level header within :GAMEMAPS:, or zero if this level number is unused. The structure of MAPHEAD allows for up to 100 levels in the game, though in practice the number used is around 20. Any level number that is unused will have an offset of 0 stored in MAPHEAD. MAPHEAD is located within the EXE file. ===================================================================================================================================== :EGAGRAPH: (4, 5, 6) [ {Image Chunk} ] Image Chunk: Byte[] Huffman-encoded data: usually image data for a picture, tile, or sprite (see below). The EGAGRAPH block comprises the whole of EGAGRAPH.CK?. Each Image Chunk is compressed individually using Huffman encoding, but a single dictionary is used for all chunks. The chunks come in two flavours: all chunks except those which contain a single tile's data have as the first Long the length of the uncompressed data; tile chunks consist of compressed data only. Offsets to the chunks are stored in :EGAHEAD:. Note that there is no distinction in EGAGRAPH or :EGAHEAD: between tile chunks and non-tile chunks; that is hard-coded into the program itself. Apart from image data, EGAGRAPH also stores help text, recorded demo data, and a few other bits and pieces. Chunk 0 is the Picture Table, which stores two Words (Width\8, Height) for each picture (115 for Keen 4) Chunk 1 is the Masked Picture Table, which stores two Words (Width\8, Height) for each masked picture (3 for Keen 4) Chunk 2 is the Sprite Table, which stores the following data for each sprite (397 for Keen 4): w/8, h, x offset, y offset, clipr_l, clipr_t, clipr_r, clipr_b, flags (all words) Chunks 3, 4, and 5 are fonts. The font format is: Word h, Word[256] Offset, Byte[256]Width, data Chunks 6-120 are pics Chunks 121-123 are masked pics Chunks 124-520 are sprites chunks 523-1818 are 16x16 tiles Chunks 1819-4734 are 16x16 masked tiles end_ansi 4735 ?? 4736 ?? 4737 nomem_ansi 4738 wrist_help 4739 story_help 4740 game_help 4741 about_id_help 4742 end_game_help 4743 whoops_help 4744 order_help 4745 demos 4746 - 4750 ?? 4751 end of file 4752 ===================================================================================================================================== :EGAHEAD: (4, 5, 6) [ {Chunk Offset} ] Chunk Offset: Byte[3] Offset to an image chunk within :EGAGRAPH: (see below). Chunk offsets are stored as 24-bit integers, in normal Intel byte order. An offset of xFFFFFF indicates an empty chunk (e.g. for unused tile slots). The last chunk offset in EGAHEAD does not point to an actual chunk, but is equal to the length of :EGAGRAPH:. EGAHEAD is located within the EXE file. :EGAHEAD: (D) [ {Chunk Offset} ] Chunk Offset: Byte[4] Offset to an image chunk within :EGAGRAPH: (see below). The EGAHEAD file in Keen Dreams is identical to that used for 4, 5, 6 except that the chunk offsets are stored as Longs, and an offset of xFFFFFFFF indicates an empty chunk. =====================================================================================================================================