Expand description
Qualia Audio — native auditory intelligence.
Architecture: docs/plans/native-auditory-language-and-music-intelligence.md
Swarm: docs/plans/native-auditory-swarm-delivery.md
§Rules
- Hot path caller-buffered; no Python.
- Model outputs are epistemic proposals.
- Frames authoritative; PCM never in NQuins.
- Oral languages need no orthography.
Re-exports§
pub use aed_weights::AedWeightBundle;pub use aed_weights::WeightedAedModel;pub use capture::CapturePurpose;pub use capture::CaptureSession;pub use convert::mono_f32_to_i16_le;pub use convert::to_mono_f32;pub use cross_modal::events_overlapping_window;pub use cross_modal::frames_to_media_ms;pub use cross_modal::propose_temporal_correlations;pub use cross_modal::AvCorrelationProposal;pub use cross_modal::TimeIntervalMs;pub use events::ReferenceEventModel;pub use events::CLASS_NOISE;pub use events::CLASS_SILENCE;pub use events::CLASS_SPEECH_LIKE;pub use events::CLASS_TONAL;pub use features::forward_cqt_mono;pub use features::frame_energy;pub use features::frame_zcr;pub use features::log_mel_from_mono;pub use features::magnitude_stft_chunk;pub use features::StreamingStft;pub use generation::separate_two_stems_reference;pub use generation::synthesize_reference_tone;pub use generation::StemReceipt;pub use generation::SynthReceipt;pub use generation::VoiceConsent;pub use hash::media_digest;pub use hash::q_hash;pub use hash::MediaDigest;pub use language::AccessClass;pub use language::AnnotationTier;pub use language::LanguageResourceBundle;pub use media_store::guard_duration_bytes;pub use media_store::synth_silence;pub use media_store::synth_tone_f32;pub use media_store::AudioMediaRecord;pub use media_store::AudioMediaStore;pub use media_store::RetentionClass;pub use music::chroma12_from_mel;pub use music::detect_onsets;pub use music::estimate_f0_hz;pub use music::estimate_tempo_from_onsets;pub use music::propose_structure_segments;pub use music::track_pitch;pub use music::MusicAssumptions;pub use music::OnsetEvent;pub use music::PitchEstimate;pub use music::StructureSegment;pub use music::TempoEstimate;pub use pipeline::run_ears_demo;pub use pipeline::run_ears_on_wav_file;pub use pipeline::run_ears_weighted;pub use pipeline::section18_smoke;pub use pipeline::sonify_demo_to_wav;pub use pipeline::speech_phone_demo;pub use pipeline::EarsDemoResult;pub use production::apply_comp_sample;pub use production::apply_eq_sample;pub use production::DelayLine;pub use production::ProcessPlan;pub use production::StatefulMixer;pub use production::TrackState;pub use production::MAX_BLOCK;pub use production::MAX_DELAY_SAMPLES;pub use production::MAX_MIX_DELAY;pub use production::MAX_TRACKS;pub use resample::resample_linear_mono;pub use semantic::compile_auditory_quins;pub use semantic::human_correct_quin;pub use semantic::human_reject_quin;pub use semantic::AudioQuin;pub use semantic::CTX_AUDIO;pub use semantic::P_AUDITORY_OBSERVATION;pub use semantic::P_PROPOSES_SOUND_CLASS;pub use session_history::AutomationLane;pub use session_history::AutomationPoint;pub use session_history::OpKind;pub use session_history::SessionHistory;pub use session_history::SessionOp;pub use session_history::MAX_AUTO_POINTS;pub use session_history::MAX_HISTORY;pub use sonify::class_to_hz;pub use sonify::sonify_events_mono;pub use speech::decode_for_language;pub use speech::greedy_phone_decode;pub use speech::SpeechEncoderWeights;pub use speech::StreamingSpeechDecoder;pub use speech::PHONES;pub use wav::decode_wav;pub use wav::encode_wav_i16_mono;pub use wav::DecodedWav;pub use types::*;
Modules§
- aed_
weights - P64-style acoustic event head (QVWT-like seed weights) — COMPLETE-WITH-GATE for real P64.
- aqa
- Audio Quality Assessment — intrusive (pure-Rust, no licence) + non-intrusive MOS (NeedsWeights). Never emit a fabricated MOS number: learned MOS fails closed until weights are supplied.
- capability_
registry - Capability registry — the honest, machine-readable status of every audio capability.
- capture
- Capture adapter — permission + session (platform device behind desktop).
- convert
- Sample format and channel conversion (caller-buffered after setup).
- cross_
modal - Swarm X — eyes + ears correlation (overlap ≠ causality).
- events
- A5 — reference VAD + energy event classifier (honest reference backend).
- features
- Streaming / block feature extraction (CPU reference path).
- fx
- Production FX — reverb, stereo. Real DSP the
productionblock delegates to. Re-exports only (AU-PROD). - generation
- Swarm G_audio — reference tone synthesis, consent revoke, simple stem separation.
- hash
- FNV-1a 64 — same family as Qualia
q_hash(vision semantic). - io
- I/O — metadata, equal-loudness loading, quality resampling. Re-exports only (AU-IO fills submodules).
- language
- Swarm L — language resource bundle scaffold (oral-first).
- media_
store - Content-addressed audio media store (mirror vision media store).
- midi
- Native MIDI engine — MIDI 1.0 + MIDI 2.0 (UMP), Standard MIDI File I/O, sequencer/transport, microtuning (Scala/MTS), and a zero-alloc synth. Symbolic authoring/production, under the Qualia ABI.
- models
- Learned audio heads — loaded via a common P64 loader that fails closed (abstains /
NeedsWeights) when weights are absent. No Candle/Burn runtime, no Python. Real weights are gated on the principal (HA1 corpus, HA6 speech model). See ADR 007 + delivery plan Waves 7–8. - music
- Swarm M — foundational music features (assumptions declared).
- pipeline
- End-to-end Ears MVP demo path (A5 product surface).
- production
- Swarm P — bounded production engine (zero-alloc process block + FX primitives).
- qac_
spatial - QAC Spatial Audio, Higher-Order Ambisonics (HOA) Bus, Wigner-D Rotation,
Binaural HRIR/RIR Rendering, ITU-R BS.1770-5 Loudness & Modal Resonance Adapters
(
qac_spatial.rs). - resample
- Linear resampler (cold/simple; declared latency 0 for linear interp).
- semantic
- Auditory observations → NQuin-compatible records (epistemic).
- session_
history - DAW session operation history (cold path — not in audio callback).
- sonify
- U3-style hear path: events → parametric mono PCM (navigate by time).
- spatial
- Zero-Heap 3D Spatial Audio & Doppler Engine (
spatial.rs) - speech
- Speech path scaffold — greedy “phone” decode over log-mel (not full ASR).
- types
- Stable auditory ABI (fixed-layout, caller-buffered where hot).
- wav
- Minimal WAV (PCM i16 / f32 LE) cold decoder — no external codec crate.
Structs§
- Spatial
Emitter - 3D Spatial Sound Emitter.
- Spatial
Listener - 3D Spatial Listener (camera or avatar ears).
- Spatial
Occlusion Bridge - Acoustic occlusion bridge parameters for SDF/BVH ray-marched scene audio graphs.
- Spatial
Occlusion Filter State - Zero-heap stereo DF2T biquad state for per-voice occlusion low-pass filtering.
- Spatial
Voice Params - Computed spatial voice parameters for audio buffer rendering.
Enums§
- Distance
Model - Distance attenuation model.
Constants§
- MAX_
SPATIAL_ VOICES - Maximum spatial sound emitters processed in a single zero-heap mix pass.
- SPEED_
OF_ SOUND_ AIR - Speed of sound in dry air at 20°C in m/s.
Functions§
- mix_
spatial_ voice_ into - Zero-heap stereo mixer mixing mono PCM audio into a stereo output buffer with spatial gains.
- mix_
spatial_ voice_ occluded_ into - Zero-heap occluded stereo mixer with DF2T biquad low-pass occlusion filtering and wet reverb send bus accumulation.