Skip to main content

recommend_convert_layout

Function recommend_convert_layout 

Source
pub fn recommend_convert_layout(
    source_bytes: u64,
    vram_budget_bytes: u64,
) -> P64ConvertLayout
Expand description

Recommend convert layout from source size (bytes on disk) and a VRAM headroom budget.

Heuristic:

  1. F16Expand when Q4→f16 estimate (source * 4) fits under vram * 0.55 and the 4 GiB p64 offset cap — best decode path for small/mid models.
  2. Else Q4kSoa for mid/large Q4 containers (SoA is ~11% larger than Q4_K; always under the f16 budget). Decode GEMV drops scale-unpack barriers.
  3. Else Verbatim.