Your device · Your weights · No cloud required

Edge LLM Hub

GGUF models mapped via memmap2 directly into the OS page cache. Phase 8 bifurcated compute via wgpu. Zero cloud egress.

Honest: decode tok/s on consumer GPUs is still below hand-tuned engines. For rights-grade grounding use QUALIA_RIGHTS_MODE=1. See 0.0.28 progress.

Daemon not detected on localhost:4242 — Demo mode active. Run qualia-cli daemon --dev to enable live inference via the WebSocket bridge.

Model Catalog

Select a model to chat. Click to copy the CLI load command.

Telemetry

ws://127.0.0.1:9090
RSS
— MB
Tokens/s
—
Hot blocks
—
Context
—
Live RAM feed from TelemetryPayload on port 9090. Connect the daemon to see real data.
?
Select a model
—
Tokens: 0
T/s: —
Elapsed: —
RAM: —
Phase 8 bifurcated

The qualia-bridge WebSocket is the single persistent connection from any browser client to the native daemon. LLM inference multiplexes over the same socket using the llm_* frame family.

Handshake (server → client)
← { "type": "HANDSHAKE_SUCCESS", "payload": { "mode": "NATIVE", "version": "0.0.30" } }
Start inference (client → server)
→ { "type": "llm_infer", "id": 1, "model": "gemma-3-1b-it-q4_0", "prompt": "Explain NQuin layout", "max_tokens": 256, "temperature": 0.7 }
Streaming token (server → client)
← { "type": "llm_token", "id": 1, "token": " Hello", "tps": 14.2, "rss_mb": 2841 } ← { "type": "llm_token", "id": 1, "token": " world", … }
Completion (server → client)
← { "type": "llm_done", "id": 1, "total_tokens": 128, "elapsed_ms": 9043, "tps": 14.1, "stop_reason": "eos_token" }
Cancel (client → server)
→ { "type": "llm_cancel", "id": 1 }
Status query (client → server)
→ { "type": "llm_status" } ← { "type": "llm_status_reply", "loaded_model": "gemma-3-1b…", "lifecycle": "Resident", "rss_mb": 2841, "thermal": "Normal" }
The llm_* frame family is planned infrastructure. Today, send llm_infer and the daemon returns { "code": "unknown_type" } — use qualia-cli llm eval for actual inference.
List models in vault
qualia-cli llm list qualia-cli llm list --vault-path C:\llmmodels
Download a model
qualia-cli resources download gemma-3-1b-it-q4_0 qualia-cli resources download qwen2.5-1.5b-instruct-q4km
Load into memory
qualia-cli llm load gemma-3-1b-it-q4_0 # Maps GGUF via memmap2 → wgpu pipeline (Vulkan default) # Prints profile_id + resident bytes
Run inference
qualia-cli llm eval "Explain NQuin layout" qualia-cli llm eval "Explain NQuin" --orchestrated # --orchestrated routes through Webizen VM pre/post checks
Check status & lifecycle
qualia-cli llm status # Shows: lifecycle state, resident bytes, KV cache MB, # thermal state, scrubbing lock
Evict (free GPU/RAM)
qualia-cli llm evict gemma-3-1b-it-q4_0
Inference pipeline summary
GgufSharder → parses GGUF header → NQuin pointer map (byte offsets in upper bits, modality flag 0b1001)
GgufBridge → mmap weights into OS page cache via memmap2 → zero heap allocation
fused_tensor_contraction.wgsl → 64 threads/workgroup, 4096 FMA ops/thread → Vulkan/DX12/Metal
LocalLlmAgent → Phase 8 bifurcated compute: LLM Engine ↔ Webizen Sentinel via SPSC ring buffers
Orchestrator → N3Logic rights pre-flight → infer → provenance NQuin citation check
LLM Engine Thread
  • Autoregressive token loop
  • GPU dispatch via wgpu
  • Pushes logit vectors → LogitStream
  • Reads control signals ← ControlStream
  • On DenyRollback: recalculates mid-generation
Webizen Sentinel Thread
  • Reads logit vectors in real time
  • Detects anomalies (e.g., 0x99 byte signature)
  • Injects DenyRollback into ControlStream
  • Writes conduct violation Quin → WAL (ed25519)
  • Governance is mid-generation, not post-hoc
Orchestrator Gates
  • validate_intent() — N3Logic rights pre-flight
  • Deny writes signed Quin to WAL; model never invoked
  • agent.infer() — actual GPU inference
  • validate_output() — output must cite ≥1 NQuin provenance
  • Ungrounded output rejected — no hallucination egress
AgentBackend variants: Local — GGUF on disk → wgpu → in-process. No outbound traffic. 128 MB RAM cap. Remote — API call → Nym mixnet → ILP metered. Requires signed VC from Principal. Hybrid — Local-first. Falls back to Remote only with explicit Principal consent. // Never add an Ollama backend. Model it on LocalLlmAgent in llm_agent.rs.