Module safetensor 

Phase 6 / task #12 — safetensor (+ MLX) source parsing + dtype gate for the streaming transcoder (p64_weight::transcode_safetensor_to_p64).

This module only parses + validates a source; the streaming emit to the P64 container lives in p64_weight (it owns the container’s private serializers). The split keeps the format writer encapsulated.

Scope (honest)

• safetensor — the on-disk layout is parsed here: an 8-byte little-endian header length, then a JSON header { name: { dtype, shape, data_offsets:[begin,end] }, … }, then the raw tensor bytes. The JSON header is small (KBs); the tensor bytes are never read here — only their offsets — so a multi-GB file is not loaded to plan the transcode.
• MLX — Apple MLX exports are safetensor-format (often with __metadata__.format = "mlx"); they parse through this path. MLX .npz archives are deferred (a different container).
• high-fidelity only — is_high_fidelity_ggml accepts F32 / F16 / BF16 / Q8_0 and rejects Q4_* and other low-precision quant types (the “Q4 rejected” rail).

Structs

SafeTensorEntry

One tensor declared in a safetensor header.

SafeTensorPlan

The parsed plan: every tensor’s metadata + the absolute offset where tensor data begins.

Enums

SourceFormat

A detected model source container.

Constants

GGML_BF16

GGML_F16

GGML_F32

GGML element type codes used here (mirrors gguf_sharder): 0=F32, 1=F16, 8=Q8_0, 30=BF16; low-precision quants are Q4_0=2, Q4_1=3, Q4_K=12, ….

GGML_Q8_0

Functions

detect_format

Sniff the container format from the first bytes. GGUF starts with the ASCII magic GGUF; safetensor starts with an 8-byte LE header length immediately followed by a { (the JSON).

ggml_elem_bytes

Bytes per element for the dtypes this path accepts (used to validate declared tensor sizes).

is_high_fidelity_ggml

Whether a GGML element type is a high-fidelity source this versioned path accepts. Accepts F32 / F16 / BF16 / Q8_0; rejects Q4_* and every other low-precision quant — the “ingest high-fidelity sources only; Q4 rejected/warned” rail.

parse_safetensor_header

Parse a safetensor header (the small JSON prefix only — never the tensor bytes). Validates that every declared byte range lies within src.

safetensor_dtype_to_ggml

Map a safetensor dtype string to a GGML element type. None for anything not a supported high-fidelity weight dtype (so unknown / low-precision safetensor dtypes are rejected upstream).