Skip to main content

decode_with_metrics

qualia_core_db::inference::inference_bench

Function decode_with_metrics 

Source 
pub fn decode_with_metrics(
    model_path: &str,
    prompt: &str,
    decode_tokens: u32,
) -> Result<(String, f64), String>
Expand description

A1b: mount a model (auto-detecting P64 vs GGUF by magic) and run ONE decode of prompt for decode_tokens, returning (text, decode_tok_s). For a ternary .q42 the FFN routing follows the global set_ternary_ffn toggle, so a caller can measure GPU-ON vs CPU-OFF on identical weights. Caller sets the toggle before invoking. (Use the _blocking wrapper from sync code.)