P64 Weight Container Standard

Document version: 0.2
Container version: 4
Date: 2026-07-10
Status: Internal Draft Standard (living — update when layout/decode gaps are found)
Canonical extension: .p64
Normative implementation: crates/qualia-core-db/src/q42/p64_weight.rs
Upgrade plan: docs/plans/p64-decode-upgrade-plan.md

Living draft rule. If implementers find container or decode-profile improvements, update this standard and the upgrade plan in the same change set. Do not leave gaps only in chat.

Abstract

P64 is QualiaDB’s little-endian, cache-line-oriented container for local model weights. It stores a model’s hyperparameters, tensor descriptors, names, 10-dimensional manifold coordinates, tokenizer data, checksums, and opaque tensor blobs in one memory-mappable artifact.

P64 is a sibling of, not a profile of, the semantic Q42 graph format. A P64 file begins with p64\0; a Q42 volume begins with Q42\0. P64 contains no NQuin records and makes no truth or provenance claims by itself.

This document specifies the byte layout emitted and accepted by container version 3 of the QualiaDB implementation.

1. Conformance language and scope

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as normative requirements.

This standard covers:

It does not standardize:

2. Design boundary

Artifact Magic Primary responsibility
Q42 semantic volume Q42\0 Quins, assertions, provenance, indexes, and graph history
P64 weight container p64\0 Model metadata and page-aligned mathematical weight blobs

A system MAY keep Q42 provenance and P64 weights co-resident, but it MUST identify each artifact by its own magic. The historical Q42W weight format is superseded and is not conformant with this standard.

The names compile_gguf_to_q42*, from_q42, and Q42TensorIndex remain in the implementation as compatibility aliases. They read or write P64 v3 bytes and do not change the on-disk magic or format identity.

3. Primitive encoding rules

  1. Every integer and floating-point field is little-endian.
  2. All file offsets are unsigned 32-bit byte offsets from the first byte of the P64 file.
  3. A conforming v3 file therefore MUST be smaller than 2^32 bytes.
  4. Fixed records are 64 bytes. Reserved bytes and alignment padding emitted by a conforming writer MUST be zero.
  5. Tensor payload interpretation is selected by the tensor entry’s dtype. Payload bytes are otherwise opaque to the container.
  6. Arithmetic used to validate offsets, lengths, counts, and alignments MUST be checked for overflow.

4. File organization

The canonical v3 writer emits sections in this order:

+-------------------------------+ 0
| P64WeightHeader        64 B   |
+-------------------------------+ hparams_offset
| P64HParams             64 B   |
+-------------------------------+ tensor_table_offset
| P64TensorEntry[tensor_count]  | 64 B each
+-------------------------------+ string_table_offset
| NUL-terminated string table   |
+-------------------------------+ align(64)
| ManifoldCoordinate10D[]       | 64 B each; n_layer + 1 entries
+-------------------------------+ tokenizer_offset
| optional Q42T tokenizer       |
+-------------------------------+ align(64)
| CRC-32C table                 | 4 * (tensor_count + 1) B
+-------------------------------+ align(page_size)
| tensor blob 0                 | start is page-aligned
| padding                       |
| tensor blob 1                 | start is page-aligned
| ...                           |
+-------------------------------+ align(64)

align(N) means rounding the next section offset upward to a multiple of N. The default page_size is 16,384 bytes. A 4,096-byte page profile is also used by tests and constrained deployments.

The section offsets in the header are authoritative. Readers MUST NOT rely only on the canonical ordering diagram.

5. P64WeightHeader

The header occupies exactly 64 bytes.

Byte range Type Field v3 requirement
0..3 u8[4] magic MUST equal 70 36 34 00 (p64\0)
4..5 u16 version MUST equal 4 (historical readers may accept 3 as a soft alias)
6..7 u16 flags Format flags; bit 0 MUST be set
8..11 u32 role_table_offset MUST be zero in the current v3 profile
12..15 u32 tensor_table_offset Start of 64-byte tensor entries
16..19 u32 tokenizer_offset Start of optional tokenizer section
20..23 u32 hparams_offset Start of the 64-byte hyperparameter record
24..27 u32 string_table_offset Start of the string pool
28..31 u32 checksum_offset Start of the CRC-32C table
32..35 u32 manifold_table_offset Start of 64-byte manifold records
36..39 u32 tensor_count Number of tensor entries and blobs
40..43 u32 page_size Power-of-two blob alignment, at least 256
44..63 u8[20] reserved MUST be zero

5.1 Header flags

Bit Constant Meaning
0 P64_FLAG_LITTLE_ENDIAN Required. Numeric fields use little-endian encoding.
1 FORMAT_FLAG_RAW_TRANSCODE Source was streamed from a high-fidelity tensor container without a complete GGUF model profile.
2 FORMAT_FLAG_TERNARY One or more tensors use P64’s BitNet-1.58b ternary encoding.
3 P64_FLAG_Q4K_SOA At least one 2-D weight matrix uses dtype = 112 (Q4_K_SOA).
4 P64_FLAG_LAYER_MAJOR Known-role tensor blobs are stored in layer-major table order (see §7.3).
5 P64_FLAG_LAYER_PACK Page-align only at layer boundaries; 256-byte align within a layer (decode/CUDA residency).
6 P64_FLAG_LAYER_SCHEDULE role_table_offset points at P64LayerScheduleEntry[n_layer] (64 B each).
7..15 reserved Writers MUST clear these bits until allocated in a later draft.

The raw-transcode flag permits zero-valued model hyperparameters and an empty tokenizer section. The ternary and Q4_K_SOA flags are container-level hints; each tensor’s dtype remains authoritative. Readers that care about residency order SHOULD prefer files with P64_FLAG_LAYER_MAJOR set.

6. P64HParams

The hyperparameter record occupies exactly 64 bytes.

Byte range Type Field Meaning
0..3 u32 n_layer Transformer layer count
4..7 u32 n_embd Embedding width
8..11 u32 n_head Attention head count
12..15 u32 n_kv_head Key/value head count
16..19 u32 vocab_size Token vocabulary size
20..23 f32 rope_freq_base RoPE frequency base
24..27 f32 rope_scale RoPE scale
28..63 u8[36] reserved MUST be zero

For a raw Safetensors transcode, fields unavailable from the source metadata MAY be zero. A runtime that requires those fields MUST reject or supplement such a container before inference.

7. P64TensorEntry

Each tensor descriptor occupies exactly 64 bytes.

Byte range Type Field Meaning
0..3 u32 name_offset Byte offset into the string table
4..5 u16 role_id Semantic engine role
6..7 u16 dtype GGML-compatible element type or P64 extension
8..11 u32 manifold_idx Index into the manifold table
12..15 u32 rank Tensor rank; MUST be in 1..=4
16..31 u32[4] dimensions Shape; unused trailing dimensions are zero
32..35 u32 blob_offset File-relative payload offset
36..39 u32 blob_size Payload size in bytes
40..47 u64 source_offset Original tensor offset in the source data region
48..55 u64 source_name_hash Canonical hash of the original tensor name
56..63 u8[8] reserved MUST be zero

blob_offset MUST be a multiple of page_size. Entries MUST describe blobs in non-decreasing file order, and their byte ranges MUST NOT overlap.

source_name_hash preserves the hash convention of the producer profile. A GGUF conversion uses full-width FNV-1a 64-bit with offset basis 0xcbf29ce484222325 and prime 0x100000001b3. The Safetensors profile uses QualiaDB q_hash, which applies the same FNV-1a calculation and then masks the result with 0x0fff_ffff_ffff_ffff. Source-parity validators MUST compare using the convention of the corresponding producer profile.

7.1 Tensor roles

Value Constant Role
0 P64_ROLE_ATTN_K Attention key projection
1 P64_ROLE_ATTN_V Attention value projection
2 P64_ROLE_ATTN_Q Attention query projection
3 P64_ROLE_ATTN_OUTPUT Attention output projection
4 P64_ROLE_FFN_GATE Feed-forward gate projection
5 P64_ROLE_FFN_UP Feed-forward up projection
6 P64_ROLE_FFN_DOWN Feed-forward down projection
7 P64_ROLE_ATTN_NORM Attention/input normalization
8 P64_ROLE_FFN_NORM Post-attention/FFN normalization
9 P64_ROLE_TOKEN_EMBD Global token embedding
10 P64_ROLE_OUTPUT Global output or language-model head
11 P64_ROLE_OUTPUT_NORM Global final normalization
0xfffe P64_ROLE_UNKNOWN Preserved tensor with no recognized engine role

Unknown source tensors MUST be retained when performing a byte-preserving GGUF conversion. A runtime MAY ignore an unknown role for execution, but validators MUST still validate its descriptor, alignment, bounds, and checksum.

Global tensors use manifold index n_layer. The Rust constant P64_LAYER_GLOBAL = 0xffff is an ingest-side role-mapping sentinel; it is not written to the 32-bit manifold_idx field.

7.2 Tensor data types

For source-preserving tensors, dtype carries the source GGML type code. The current engine explicitly supports these common codes:

Value Encoding
0 F32
1 F16
2 Q4_0
6 Q5_0
8 Q8_0
12 Q4_K
14 Q6_K
30 BF16
112 Q4_K_SOA (Qualia decode-profile layout; not a stock GGML type)
1158 P64 BitNet-1.58b ternary

Other source type codes MAY be preserved when the producer and consumer both support them. Consumers MUST NOT infer a tensor encoding solely from the container-level ternary or SoA flags.

7.2.1 Q4_K_SOA (dtype = 112)

Produced by convert layout P64ConvertLayout::Q4kSoa from source Q4_K matrices. Each 256-element superblock is 160 bytes:

qs[128]           // nibble payload (same element order as stock Q4_K)
d_sub[8] × f16    // pre-expanded (d * sub_scale) for 8 groups
m_sub[8] × f16    // pre-expanded (dmin * sub_min) for 8 groups

Purpose: GEMV kernels read scales without re-decoding packed 6-bit headers. When any tensor uses this dtype, writers MUST set P64_FLAG_Q4K_SOA.

7.3 Layer-major blob order

When P64_FLAG_LAYER_MAJOR is set, the tensor table and blob region for known roles MUST be ordered as:

for layer in 0..n_layer:
  attn_norm, attn_q, attn_k, attn_v, attn_output,
  ffn_norm, ffn_gate, ffn_up, ffn_down
then globals: token_embd, output, output_norm
then P64_ROLE_UNKNOWN tensors (stable by source_offset)

Writers MUST NOT re-sort known-role entries by the original GGUF source offset. Decode residency and CUDA multi-weight fill SHOULD walk the table in order.

7.4 Layer-pack alignment and schedule table

When P64_FLAG_LAYER_PACK is set:

When P64_FLAG_LAYER_SCHEDULE is set, role_table_offset is the start of n_layer consecutive P64LayerScheduleEntry records (exactly 64 bytes each):

Byte range Type Field
0..3 u32 layer
4..7 u32 blob_begin (inclusive file offset of first blob)
8..11 u32 blob_end (exclusive end of last blob)
12..13 u16 tensor_count
14..15 u16 roles_mask (bit i ⇒ role_id i present)
16..63 u8[48] reserved, zero

Runtimes MAY mmap or bulk-upload [blob_begin, blob_end) as one residency unit per layer.

The ternary payload is:

scale             f32 little-endian
packed trits      ceil(element_count / 5) bytes

Five trits are packed in each byte as base-3 digits:

digit = trit + 1                    # -1, 0, +1 -> 0, 1, 2
byte  = d0 + 3*d1 + 9*d2 + 27*d3 + 81*d4

The tensor element count is the product of the first rank dimensions.

8. String table

The string table begins at string_table_offset and ends at manifold_table_offset. Each tensor entry’s name_offset is relative to the start of this table.

The canonical writer places a zero byte at relative offset 0, then stores each UTF-8 tensor name followed by a NUL byte. A reader MUST confirm that every referenced name starts inside the string table and has a terminating NUL before the manifold table.

Known GGUF roles use canonical names such as:

blk.3.attn_q.weight
blk.3.ffn_down.weight
token_embd.weight
output.weight
output_norm.weight

An unknown GGUF tensor MAY be represented as tensor.<source_name_hash-as-16-lowercase-hex-digits> when its source spelling is not retained by the conversion index.

9. 10D manifold table

The manifold table:

Each record stores ten little-endian f32 values followed by 24 zero bytes:

Byte range Dimension
0..3 scale
4..7 attention_depth
8..11 epistemic_weight
12..15 topological_spin
16..19 temporal_decay
20..23 entropy_bias
24..27 spatial_phase
28..31 recurrence_frequency
32..35 density_threshold
36..39 manifold_curvature
40..63 reserved, zero

Every coordinate value MUST be finite. Each tensor’s manifold_idx MUST be less than n_layer + 1.

The canonical sequential-layer projection is:

depth                  = layer / max(total_layers, 1)
scale                  = depth
attention_depth        = 1 - depth
epistemic_weight       = 1
topological_spin       = sin(depth * pi)
temporal_decay         = 0.1
entropy_bias           = 0.5
spatial_phase          = cos(depth * tau)
recurrence_frequency   = 1
density_threshold      = 0.8
manifold_curvature     = 0

10. Embedded tokenizer section

A full GGUF conversion embeds a version-1 Q42T tokenizer section at tokenizer_offset. Its logical payload has this layout:

Field Encoding
magic Q42T
section version u16, currently 1
flags u16, currently 0
BOS token id u32
EOS token id u32
add-BOS policy u8, zero=false, nonzero=true
padding 3 zero bytes
pre-tokenizer type length-prefixed UTF-8
vocabulary count u32
vocabulary entries repeated length-prefixed UTF-8
merge-pair count u32
merge pairs repeated left and right length-prefixed UTF-8 strings

A length-prefixed string is a little-endian u32 byte length followed by that many UTF-8 bytes. The implemented reader limits the vocabulary to 1,000,000 entries and merge pairs to 5,000,000.

The tokenizer section ends at checksum_offset; zero alignment padding MAY follow its logical payload. A raw-transcode container MAY have tokenizer_offset == checksum_offset, indicating no embedded tokenizer.

11. Integrity table

The checksum table begins at checksum_offset and contains tensor_count + 1 little-endian u32 values:

entry 0       CRC-32C(file bytes [0, checksum_offset))
entry 1       CRC-32C(tensor blob 0)
entry 2       CRC-32C(tensor blob 1)
...
entry N       CRC-32C(tensor blob N - 1)

P64 uses CRC-32C Castagnoli in reflected form with polynomial 0x82f63b78, initial value 0xffffffff, and final bitwise complement.

The metadata checksum covers the header, hyperparameters, tensor table, string table, manifold table, tokenizer, and all alignment padding before the checksum table. It does not cover the checksum table itself.

CRC-32C detects accidental corruption and common tampering, but is not a cryptographic signature. Authenticity and provenance MUST be supplied by a separate trusted mechanism, such as the associated Q42 graph or signed release metadata.

12. Tensor blob region

The blob floor is:

align(checksum_offset + 4 * (tensor_count + 1), page_size)

Every blob_offset MUST be at or above that floor and aligned to page_size. The end of every blob MUST be within the file. A reader MUST validate blobs in descriptor order and reject overlap or backward ordering.

The canonical writer pads the complete file to a 64-byte boundary. Padding between sections and blobs is not part of any tensor payload.

P64 preserves the source tensor’s native dimension order. GPU or SIMD implementations MAY construct a transposed, structure-of-arrays, paired-u32, or otherwise specialized execution view, but MUST NOT describe that derived view as the P64 disk layout.

13. Producer profiles

13.1 Byte-preserving GGUF profile

A byte-preserving GGUF producer:

P64TensorIndex::validate_against_gguf additionally proves tensor-count, metadata, and byte equality against the source GGUF. That source-parity check is stronger than ordinary standalone P64 validation.

13.2 Quantized GGUF profile

A quantizing producer MAY replace selected tensor blobs and update their dtype, blob_size, offsets, checksums, and relevant flags. QualiaDB’s policy profile permits ternary or Q4_0 conversion of FFN gate/up/down projections while retaining attention, norms, embeddings, output tensors, and unknown tensors at higher fidelity.

13.3 Raw Safetensors profile

A streaming Safetensors producer sets FORMAT_FLAG_RAW_TRANSCODE. It MAY omit the tokenizer and use zero for unavailable hyperparameters. Recognized tensor names receive engine roles; unrecognized names receive P64_ROLE_UNKNOWN. Its source_offset is relative to the Safetensors data region.

14. Required reader validation

Before exposing any tensor blob, a conforming reader MUST:

  1. require at least 64 bytes for the header;
  2. validate magic, version, and little-endian flag;
  3. require a power-of-two page_size of at least 256;
  4. bounds-check the 64-byte hyperparameter record;
  5. calculate every section extent with checked arithmetic;
  6. reject overlapping or out-of-order metadata sections;
  7. require a 64-byte-aligned manifold table;
  8. verify the metadata CRC-32C;
  9. require every tensor rank to be in 1..=4;
  10. validate every manifold index;
  11. validate every referenced NUL-terminated tensor name;
  12. require every blob to be page-aligned, ordered, non-overlapping, and in-bounds; and
  13. verify every tensor CRC-32C.

A reader SHOULD also validate tokenizer syntax, finite manifold values, shape and payload-size consistency for recognized dtype values, and any model-specific required-role set before beginning inference.

Validation is fail-closed: a failed check means no tensor from that container is trusted for execution.

15. Versioning and extension rules

The current implementation accepts container version 3 only. An incompatible layout change MUST increment the header version.

Within v3:

Readers MAY preserve unsupported tensors without executing them, but MUST NOT guess their encoding.

16. Security considerations

P64 files are untrusted binary input. Implementations must defend against integer overflow, excessive counts, malformed names and tokenizer strings, invalid floating-point values, overlapping ranges, checksum confusion, and resource exhaustion.

Memory mapping does not make a file safe. All validation in Section 14 occurs before offsets are used for zero-copy GPU upload or inference.

CRC-32C is not collision-resistant. Where a P64 is distributed outside a trusted local cache, deployments SHOULD verify a cryptographic digest or signature bound to the model identity, source artifact, conversion policy, and P64 version.

17. Implementation references