Skip to main content

perplexity_eval_blocking

Function perplexity_eval_blocking 

Source
pub fn perplexity_eval_blocking(
    model_path: &str,
    max_tok: usize,
) -> Result<(f64, usize), String>
Expand description

W1 — teacher-forced perplexity of model_path over the eval corpus, run through Qualia’s native engine (never an external runtime). For each corpus passage: reset_kv_cache, then per position embed → dispatch_transformer_forwardapply_output_norm_inplacedispatch_output_logits_into → NLL of the true next token; PPL = exp(ΣNLL / Σtokens). max_tok = 0 scores the whole passage,

0 caps it (to bound the slow F16-on-CPU path for big models). Returns (perplexity, tokens_scored). Runs on a dedicated thread with a current-thread tokio runtime (mirrors the decode path) so the engine’s GPU readback works. Handles both GGUF and .q42 containers.