Module information 

Information theory — Shannon entropy, KL divergence, cross-entropy and mutual information over discrete distributions / samples (all in bits, log₂).

Mission note: mutual information is a principled, assumption-free relevance signal — I(X;Y) measures how much knowing X reduces uncertainty about Y with no linearity assumption — which is exactly what the 10D→5D NQuin relevance router needs to choose its projection.

Functions

cross_entropy

Cross-entropy H(p, q) = −Σ pᵢ·log₂ qᵢ (bits). None like kl_divergence.

entropy

Shannon entropy H(p) = −Σ pᵢ·log₂ pᵢ (bits) of a probability vector. Zero probabilities contribute 0. None if empty or the masses don’t form a positive distribution.

entropy_from_counts

Entropy from integer counts (normalized internally).

kl_divergence

Kullback–Leibler divergence D(p‖q) = Σ pᵢ·log₂(pᵢ/qᵢ) (bits). Both inputs are normalized internally. None on a length mismatch, empty input, or if qᵢ = 0 where pᵢ > 0 (the divergence is then infinite — refuse rather than fabricate).

mutual_information_discrete

Mutual information I(X;Y) = H(X) + H(Y) − H(X,Y) (bits), estimated from paired discrete samples (small non-negative integer labels). None on length mismatch or empty input. I ≥ 0, and I = 0 iff X ⟂ Y in the sample.