Interface Control Plane — Interactivity Loop Plan

Also: Phone Console (primary remote client). Filename retained for searchability.

Date: 2026-06-17
Branch: 0.0.17-dev
Status: IN PROGRESS — PR-I1–I5 shipped (2026-06-17); health ingest + voice (I6–I8) next Companion: wasm-viewport-migration-plan.md Track I, qualia-wasm-portal.md, AUDIO_PROJECT_STATUS.md

Executive summary

The Interface Control Plane (ICP) is the normalized input layer for QualiaPortal: every human interface device emits the same fixed-size command envelopes, which the viewport applies through one dispatch path (set_camera, navigate_to_node, set_standpoint, …).

The phone console is the flagship remote ICP client — a sovereign companion PWA, not a second “everything app.” It runs the same WASM artifact in an input-heavy, render-light role while a primary portal owns T2 phenomenal rendering.

Hard rule: JavaScript is glue only. ICP semantics compile to fixed-size envelopes (64-bit PortalControlCommand or 48-byte NQuin), not ad-hoc JSON scene graphs. No Three.js, no Oxigraph-in-browser, no Ollama/llama.cpp HTTP sidecar.

Field observations (2026-06-17)

Empirical testing on a physical phone shows Qualia WASM can run faster than the same build on a desktop machine that is plugged into power. Do not architect the phone as a permanently degraded T0-only client.

Plan implication: Phone console UI should probe and display tier (portal.tier()), not hard-code “remote control only.” A phone may legitimately run local phenomenal preview, personal graph queries, or Anatomy overlays when vault data is on-device.

Primary user journey (v0.1 — locked intent)

Single origin entry; no app-store distribution for now.

1. User opens Qualia site on desktop (e.g. spatial.html, anatomy playground)
2. User taps **Install mobile companion** → PWA manifest → phone-console.html added to home screen
3. Desktop shows **Link phone** — renders QR (session + auth material) on screen
4. Phone PWA opens camera → scans QR → completes Ed25519 / session handshake
5. Linked: phone = remote + secondary lists; desktop = primary phenomenal viewport
6. (Optional) Phone: **Choose local folder** → sovereign personal vault root
7. (Optional) Phone: import Samsung Health export (CSV/JSON) → cold ingest → personal graph
8. Desktop Anatomy (or spatial) view reflects ingested health observations when user navigates there

sequenceDiagram
    participant User
    participant Desktop as Desktop browser
    participant Phone as Phone PWA

    User->>Desktop: Open spatial.html / anatomy
    User->>Phone: Install PWA from same origin
    Desktop->>Desktop: createConsoleSession() + render QR
    User->>Phone: Open camera, scan QR
    Phone->>Desktop: Auth hello (session_id, pubkey, exp)
    Desktop->>Phone: Accept + ContextFrame
    opt Sovereign folder
        User->>Phone: showDirectoryPicker (or OPFS default)
        Phone->>Phone: Cold ingest → OPFS SuperBlocks
    end
    User->>Phone: Navigate / select health metric
    Phone->>Desktop: PortalControlCommand
    Desktop->>Desktop: Anatomy overlay / tensor navigate

Desktop: QR payload (normative sketch)

{
  "v": 1,
  "origin": "https://example.github.io/qualiaDB",
  "relay": "https://<lan-host>:4242",
  "session_id": "hex",
  "desktop_pubkey": "hex",
  "exp_unix": 1718650000,
  "capabilities": ["remote", "context_push", "vault_sync"]
}

Phone stores pairing in sessionStorage + durable OPFS record (qualia-pairing.v1). Re-scan invalidates prior session.

Phone: install surface

Personal vault & local folder (optional sovereign path)

User may decline cloud/daemon and keep data on the phone only.

Standpoint promotion: Folder or OPFS vault unlocks identifier (class 2) then vault (class 3) standpoint on phone-local graph slices — no full SOA leaves the device unless user explicitly links desktop with consent (Prolog Sentinel / deontic lane).

Personal graph: Cold path only on phone — N3/CSV → NQuin bake → OPFS blocks → bounded visit_*_into queries in WASM. Desktop receives hashes + observation summaries via ContextFrame, not raw export files.

Health ingest trail — Samsung Health → Anatomy

Samsung Health “export data” produces CSV (and sometimes JSON) under com.samsung.health.* / com.samsung.shealth.* prefixes. Qualia already parses several of these in wellfare-core:

All carry prov:wasDerivedFrom <urn:health:source:samsung-health-export> in wellfare-core/src/rdf.rs.

Display path (Anatomy QApp)

Phone folder pick or file input
    → wellfare-core parser (cold, heap OK once)
    → Turtle / NQuin compile (SHACL: bundled/qapps/Anatomy/Knowledge/shapes/radlex-anatomy.shacl.ttl)
    → OPFS resident graph on phone
    → ContextPush summaries to desktop (metric ids, date ranges, organ system hashes)
    → docs/playground/anatomy.html + bundled/qapps/Anatomy overlay
         (dicom-overlay.js, knowledge-parser.js, anatomy_context.rs on native)

UX on phone: After vault init, Import health export walks selected folder for *.csv, shows parse preview (row counts, date span), user confirms ingest.

UX on desktop: Anatomy view highlights systems with fresh observations (e.g. cardiovascular ← heart rate, musculoskeletal ← weight). Full 3D holograph stays on desktop; phone shows tabular/list companion view.

This is Phase I7 (below) — depends on I5 vault, not on daemon.

What Qualia already has (do not re-invent)

Primary portal (desktop / large screen)

Storage (phone-local personal data)

Daemon graph engine (optional co-located host)

Port 4242 is the semantic graph daemon, not an LLM server:

LLM inference stays in-process on native hosts via LocalLlmAgent / Phase 8 bifurcated compute. WASM on Pages does not load GGUF; phone companion must not assume on-device LLM unless a future tiny portal feature gate is explicitly added.

Architecture correction (vs. generic “multi-WASM agent” drafts)

┌──────────────────── PRIMARY (T1–T2) ────────────────────┐
│  spatial.html / webizen-browser / Flutter WebView        │
│  QualiaPortal: WebGPU projector + ambient + bloom        │
│  Daemon :4242 (optional): tensor slice + chat relay        │
│  Native host (optional): full GGUF + Sentinel thread     │
└────────────────────────▲──────────────────────────────────┘
                         │  PortalRemoteFrame (fixed-size)
                         │  + optional ContextPush (bounded)
┌────────────────────────┴──────────────────────────────────┐
│  PHONE CONSOLE PWA (tier-probed — T1/T2 observed in field)   │
│  Same qualia_bg.wasm — lists, menus, sensors, local vault    │
│  OPFS: personal SuperBlocks + last context cache          │
│  U3: MessagePort path (no SAB required on iOS)              │
│  Input: DeviceOrientation, touch, mic → Sonic Tokens        │
└─────────────────────────────────────────────────────────────┘

Interface Control Plane (ICP) — architecture

One dispatch core in qualia-core-db; multiple HID adapters in JS shells. Remote clients use the same opcodes as local ones — only the ingress path differs.

┌─────────────────────────────────────────────────────────────────┐
│                     VIEWPORT (QualiaPortal)                      │
│  drain_control_commands() → apply → render → push_context()     │
└────────────▲───────────────────────────────────────▲────────────┘
             │ local queue (SPSC)                    │ remote queue
┌────────────┴────────────┐              ┌───────────┴──────────────┐
│  Local ICP adapters     │              │  Remote ICP transport     │
│  · pointer (mouse/touch)│              │  · daemon chat relay      │
│  · keyboard hotkeys     │              │  · WebRTC DataChannel     │
│  · wheel / trackpad     │              │  · native IPC (C10)       │
└────────────▲────────────┘              └───────────▲──────────────┘
             │                                        │
   ┌─────────┴─────────┬──────────────┬────────────────┴────────┐
   │ Desktop HID       │ Touch screen │ Phone / tablet companion │
   │ mouse, keyboard   │ on primary   │ gyro, mic, Graph Lens    │
   └───────────────────┴──────────────┴──────────────────────────┘

HID capability matrix

Capability negotiation: Each client registers { client_id, hid_class, caps[] } at connect. Viewport enables UI affordances in ContextFrame accordingly (e.g. omit TILT_FRAME hints for mouse-only clients).

Local vs remote ingress

Both call portal.push_control_command(raw_u64) (local) or transport equivalent (remote). Viewport drain_control_commands() runs once per tick — zero heap, fixed ring (same pattern as Sonic Token SPSC).

Keyboard / mouse on primary (ordinary HID)

No phone required. Primary viewport mounts local ICP by default:

Touch-screen on the primary device reuses the same bindings as phone Deck and Graph Lens shells when responsive rules select touch chrome — one codebase (qualia-phone-deck.js shared as qualia-icp-touch.js).

App rules — device fallbacks & responsive design

ICP shells MUST probe capabilities, pick sensible defaults for the device class, and degrade gracefully — never show controls that cannot work, never require a phone when mouse/keyboard suffice.

Principles (normative)

1. Feature detection, not user-agent sniffing — use matchMedia, PointerEvent, DeviceOrientationEvent, getCapabilities()-style probes; UA only as last-resort hint.
2. Basic expectations per device — each class gets the interaction model users already expect (see table below); Qualia adds Deck/Graph/Control only when width or pairing warrants it.
3. Responsive layout — one HTML shell; CSS grid/flex reflow; canvas remains primary on large screens; chrome expands on narrow screens.
4. Explicit override — user can force interface mode in Settings (e.g. “Desktop layout on tablet”, “Pointer mode on touch laptop”); stored in localStorage / OPFS prefs.
5. Fail open to local ICP — if relay pairing fails, primary viewport still fully usable with mouse/keyboard/touch defaults.

Device profile (IcpDeviceProfile)

Computed at boot by qualia-icp-profile.js; recomputed on resize, orientationchange, and matchMedia updates.

// Illustrative — not shipped API
{
  form_factor: 'desktop' | 'tablet' | 'phone',      // from breakpoints + UA hints
  pointer_primary: 'fine' | 'coarse' | 'none',      // (pointer: fine), (pointer: coarse)
  hover_available: boolean,                         // (hover: hover)
  keyboard_available: boolean,                      // heuristic: focusable + no virtual keyboard-only
  orientation_sensor: boolean,                      // permission + DeviceOrientationEvent
  motion_sensor: boolean,
  voice_capable: boolean,                           // speechSynthesis / mic permission path
  paired_remote: boolean,                           // active ICP relay session
  wasm_tier: 0 | 1 | 2,                             // portal.tier()
  prefers_reduced_motion: boolean,                  // (prefers-reduced-motion: reduce)
}

Responsive breakpoints (layout)

CSS contract: shells use clamp() typography, dvh for viewport height, env(safe-area-inset-*) on notched phones. No horizontal scroll on control surfaces; graph lens scales with aspect-ratio: 1.

Default interface by profile

Capability fallback ladder

When a sensor or API is unavailable, fall back to the next simpler control — never block the loop.

UI visibility rules (shouldShow)

Shell components consult profile + caps — do not render disabled affordances (saves layout noise and avoids false expectations).

ContextFrame hints (viewport → client)

Desktop pushes recommended UI mode so remote phone does not fight local layout:

{
  "icp_hints": {
    "default_interface": "deck",
    "show_graph_lens": true,
    "show_tilt": false,
    "focus_index": 7,
    "reduce_motion": false
  }
}

Honour prefers-reduced-motion on both sides — disable fly-to animation smoothing; use step transitions.

Implementation modules (Track I)

PR-I1g: device profile + responsive rules + icp_hints in ContextFrame; tests in phone-console-verify.mjs (synthetic profiles via query ?icp_profile=tablet).

Acceptance checks

• Desktop 1920×1080: canvas orbit via mouse; no swipe pad unless touch override.
• Narrow 390×844: interface bar visible; 44px buttons; canvas usable but not required for navigate.
• Phone remote without gyro: Deck works swipe-only; tilt toggle hidden.
• Keyboard-only: Tab through Control menus; fly-to via Enter on list row.
• prefers-reduced-motion: no inertial camera smoothing.
• Pairing failed: banner + full local pointer ICP on primary.

The interactivity loop (normative)

Closed loop any ICP client completes with the viewport:

sequenceDiagram
    participant Client as ICP client (local or remote)
    participant Ingress as Local queue / Relay
    participant Viewport as QualiaPortal
    participant Daemon as Daemon :4242

    opt Remote only
        Client->>Ingress: Pair (session_id, Ed25519, hid_class)
        Ingress->>Client: ContextFrame + GraphLensFrame
    end

    loop Each interaction
        Client->>Ingress: PortalControlCommand (64-bit)
        Ingress->>Viewport: deliver (direct or drain)
        Viewport->>Viewport: apply set_camera / navigate / standpoint
        Viewport->>Daemon: optional tensor slice refresh
        Daemon-->>Viewport: SSE revision
        Viewport->>Ingress: ContextPush (delta)
        Ingress->>Client: update UI highlight / menus / graph lens
    end

Viewport dispatch (map 1:1 to existing APIs)

Context outputs (bounded, all clients)

No unbounded graph serializations. Full Tensor10D SOA stays on viewport host or daemon; remote clients receive indices and hashes only.

Rename note: PortalRemoteCommand in earlier drafts → PortalControlCommand (same layout; REMOTE_MAGIC bit means “arrived via relay” for provenance logging only).

Linked mode — multi-interface controller

When paired, the phone is a control surface family — several interchangeable UIs that all emit the same PortalControlCommand envelopes. The desktop stays the sole phenomenal viewport (T2 WebGPU); the phone never tries to mirror it pixel-for-pixel.

┌─────────────────────────────────────────────────────────────┐
│  Linked · Interface bar:  [ Deck ] [ Control ] [ Graph ]    │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│   (one of three surfaces — see below)                       │
│                                                             │
└─────────────────────────────────────────────────────────────┘

User can switch interfaces anytime; last choice persisted in OPFS. Voice and Tilt toggles apply across interfaces where relevant.

App states

Interface 1 — Remote Deck

Single-purpose motion controller. Large touch targets, minimal chrome, haptics where available.

Remote Deck layout (wireframe)

┌─────────────────────────────────────┐
│  ● Linked   sleep · cardiovascular  │  ← Context strip (from ContextPush)
├─────────────────────────────────────┤
│                                     │
│         SWIPE PAD (primary)         │  ← 2D drag + flick gestures
│    swipe → pan / orbit desktop      │
│                                     │
├──────────┬──────────┬───────────────┤
│  ◀ Back  │  ⊕ Select│  ▶ Next       │  ← Button rail (3–6 actions)
├──────────┴──────────┴───────────────┤
│  ⌖ Home  │  ♫ Sonify │  ≡ List      │
├─────────────────────────────────────┤
│  [ Tilt ● ]  [ Voice ○ ]            │  ← Mode toggles (gyro / TTS feedback)
└─────────────────────────────────────┘

Design rules

• One dominant gesture surface — the swipe pad occupies ≥50% vertical space; thumbs can flick without precision aiming.
• Buttons send commands, not URLs — each maps to a PortalControlCommand opcode (see table below).
• Context strip is read-only text from desktop (ContextPush); long labels truncated; tap opens List view.
• No live video mirror of the desktop — bandwidth + sovereignty; user looks at the big screen.

Input → desktop action map

Tilt fusion: DeviceOrientationEvent + DeviceMotionEvent → complementary filter in WASM (stack buffers, no heap). Emit at ≤30 Hz when tilt mode active; desktop applies exponential smoothing to avoid jitter.

Swipe pad implementation: Pointer events on a single <canvas> or <div>; velocity at pointerup classifies flick vs drag; dead zone in center for accidental touches.

Interface 2 — Control Surface

Structured menus and control rails for discrete jumps — when the user knows what they want (a facet, time window, anatomy system) rather than where to steer the camera.

Layout (wireframe)

┌─────────────────────────────────────┐
│  ● Linked          [ Deck Graph ]   │
├─────────────────────────────────────┤
│  ▼ Explore                          │
│     Biometrics · Civics · Graph     │  ← top-level menu (from ContextFrame)
│  ▼ Health vault                     │
│     Sleep · Heart · Weight          │
├─────────────────────────────────────┤
│  t_slice  ═══════●══════  0.62      │  ← scrubbers → SET_STANDPOINT_SCALAR
│  t_window ═══●══════════  0.15      │
│  epistemic_q ═════●══════  0.80     │
├─────────────────────────────────────┤
│  Display: [ Manifold ▾ ] [ σ color ]│
│  [ Browse nodes… ]  [ Vault… ]      │
└─────────────────────────────────────┘

Menu → desktop mapping

Menus are not free-form HTML navigation. Desktop pushes a bounded tree in ContextFrame:

{
  "menus": [
    { "id": 1, "label_hash": "0x…", "label": "Health vault", "parent": 0 },
    { "id": 2, "label_hash": "0x…", "label": "Sleep", "parent": 1, "action": "facet:sleep" }
  ]
}

Anatomy / health context: Menu leaves like Sleep or Cardiovascular carry action tokens that desktop maps to Wellfair graph filters + Anatomy QApp organ highlight (anatomy_context.rs on native; JS bridge on playground).

Interface 3 — Graph Lens

A touchable projection of the desktop’s current neighborhood — nodes as pointers, edges as faint links. Press a node → desktop flies to it and pivots the phenomenal view.

What the phone renders (bounded cold payload)

Desktop pushes GraphLensFrame (≤ 8 KiB) alongside ContextFrame:

{
  "revision": 42,
  "proj_dims": ["x", "y"],
  "nodes": [
    { "index": 3, "x": 0.12, "y": -0.44, "sigma": 0.71, "q": 0.9, "selected": false },
    { "index": 7, "x": 0.55, "y": 0.22, "sigma": 0.33, "q": 1.0, "selected": true }
  ],
  "edges": [
    { "a": 3, "b": 7 }
  ],
  "focus_index": 7
}

• Positions are normalized 2D (desktop chooses projection: PCA pair, (x,y) slice, or manual dimension picker exposed in Control Surface).
• Max 64 nodes, 96 edges per frame — enough for local neighborhood, not full graph export.
• Labels optional: label_hash only; phone resolves short strings from last ContextFrame lexicon slice.

Graph Lens layout (wireframe)

┌─────────────────────────────────────┐
│  ● Linked   rev 42    [ Deck Ctrl ] │
├─────────────────────────────────────┤
│      ○───○                          │
│       \ │                           │
│        ○●7  ← selected (larger)     │
│           \                         │
│            ○                        │
│   pinch: zoom lens (local only)     │
│   tap node: fly desktop to index    │
├─────────────────────────────────────┤
│  Proj: x×y  [ Change ▾ ]            │
└─────────────────────────────────────┘

Pointer press → desktop pivot

Phone graph draw path: canvas2d or lightweight WebGPU point sprites — cold JS only, no second QualiaPortal phenomenal loop required. Optional: reuse portal CPU pick projection math from cpu_pick_node_at for consistency.

Highlight sync: After desktop processes navigation, next ContextPush marks selected: true on the active index so phone lens stays aligned.

Context payloads (summary)

Companion Voice — phone-local TTS (and optional STT)

Reads aloud context from the phone (selection confirm, list row, health import summary). Synthesis runs on the phone; audio plays on the phone speaker/earpiece only.

Separation from U3 AcousticPlane

U0 still never emits conversational PCM into the graph. Companion Voice is a PWA shell service (like notification sounds), gated by user toggle Voice on the Deck.

Model strategy (small audio LM on phone)

Field testing shows phone WASM/WebGPU can be faster than desktop — a compact on-device voice stack is plausible when tier ≥ T1 and storage quota allows.

Cold load once: model bytes in OPFS (companion-voice/ prefix); ThermalGovernor / phone battery API downgrades to Web Speech or disables under low power.

What gets spoken: Short strings from ContextPush — e.g. “Linked to desktop”, “Showing cardiovascular”, “Imported 142 sleep rows”. Never speak full vault contents without explicit user tap on a list row.

What does not happen: TTS PCM is not streamed to the desktop over the relay. Desktop phenomenal audio remains U3. If desktop must speak, that is a separate native/host feature — out of scope for phone Companion Voice.

Voice toggle on Remote Deck

[ Tilt ● ]  [ Voice ● ]

With Voice on:

1. Each successful command → brief confirm earcon (optional parametric blip via Web Audio, not U3).
2. Context strip changes → TTS speaks truncated label.
3. Long-press ⊕ Select → speak full list row detail.

Mic + STT (I8c) reuses the same Voice permission gate; recognized phrase → MENU_ACTION hash (e.g. q_hash("show sleep")) → desktop navigates.

PortalControlCommand sketch (zero-heap)

Reuse the Sonic Token bit discipline (sonic_token.rs) with a distinct magic flag range. Implemented in portal_control.rs (planned; was portal_remote.rs in early drafts):

Bits [0..7]   opcode
              0x60 = SET_CAMERA_DELTA
              0x61 = NAVIGATE_INDEX
              0x62 = COLLAPSE_Q
              0x63 = SET_STANDPOINT_SCALAR
              0x64 = MENU_ACTION
              0x65 = SONIC_TOKEN_FORWARD (raw token in bits 16..63)
              0x66 = SWIPE_GESTURE (dir: L/R/U/D, velocity tier)
              0x67 = BUTTON_ACTION (deck button id)
              0x68 = TILT_FRAME (yaw/pitch packed as i16)
Bits [8..15]  channel / session lane
Bits [16..31] tensor_index or menu_id (16-bit)
Bits [32..47] param A (e.g. yaw × 1000 as i16)
Bits [48..55] param B
Bits [56..62] reserved
Bit  [63]     ICP_MAGIC (0x1) — control plane (not Sonic Token 0x53); set when ingress is relayed

Implement in portal_control.rs, validate in portal_phenomenal_contract.rs:

• push_control_command(raw: u64) — local + remote ingress
• drain_control_commands(max: u32) — viewport tick drain
• Relay adapter copies bytes unchanged — no opcode translation

Transport phases

Phase I1 — Daemon chat relay (LAN / localhost)

Why first: Already shipped (/chat/publish, /chat/pull); Lamport ordering matches CRDT story; no new server binary.

Pairing UX: Desktop shows QR encoding {v:1, relay:"http://<host>:4242", session_id, pubkey, exp}. Phone scans → stores session in sessionStorage + OPFS pairing record.

Phase I2 — WebRTC DataChannel (offline LAN, no daemon)

For Pages-only or field scenarios without daemon:

• Use browser RTCPeerConnection + manual QR offer/answer (out-of-band paste or QR chunk).
• Same binary frames as I1 — only transport changes.
• Signaling: optional tiny same-origin /pair static page, not a cloud service.

Phase I3 — Native host bridge (PR-C10 alignment)

webizen-browser / Flutter embed forwards relay or IPC to in-process daemon. Phone talks to desktop native listener; desktop portal receives commands via postMessage from host shell.

Phone PWA feature matrix

LLM and compute (scoped honestly)

Full GGUF text LLM on phone remains out of scope for Track I. Small audio models (TTS/STT) are in scope as Companion Voice — separate from U3, cold-loaded, OPFS-cached, thermal-gated.

Security & sovereignty

PR stack (Track I)

Dependency: PR-C10 (desktop host parity) for native embed path — not blocking I1–I4 on browser-only. I7 requires I5 vault + Anatomy QApp. I3b depends on I3 link. I8 parallel after I4 (needs context strings to speak).

Suggested implementation order

I1 → I2 → I3 (pair) → I3b (Deck) → I4 (Context + GraphLens) → I4b (Control + Graph UI)
     → I8a (Web Speech) → I5 (vault) → I7 (health) → I8b (ONNX TTS) → I6 → I8c (STT)

Verification

cargo test -p qualia-core-db portal_control --lib
cargo test -p qualia-core-db phenomenal_contract --lib
node docs/tests/phone-console-verify.mjs   # after PR-I1

Manual matrix:

Out of scope (this track)

• iOS / Android store apps (PWA only)
• Full graph editor on phone
• Cloud pairing or account system
• Neural PCM generation on phone
• Screen mirroring / remote desktop

Progress log

References

• docs/manuals/qualia-wasm-portal.md — portal API
• docs/manuals/adr/0007-u3-acoustic-plane-symbolic-audio.md — U3 on phone
• crates/qualia-core-db/src/daemon_tensor.rs — standpoint signing
• docs/js/qualia-shell.js — daemon link + badge
• AI_INSTRUCTIONS.md — port 4242 surface
• crates/wellfare-core/src/parser.rs — Samsung Health CSV parsers
• crates/qualia-mobile-harness/ — QR scan + showDirectoryPicker scaffold
• bundled/qapps/Anatomy/ — anatomy overlay + SHACL shapes
• docs/playground/anatomy.html — browser anatomy demo