feat(adr-118): feature decorrelation + multi-node extractor

Audit on 6-node training data (151,329 frames) found 21 multicollinear
pairs (|r|>0.85), one dead feature (amp_min constant 0), and only node[0]
used in 8 of 15 features. Top per-feature F-stat = 15,497 but accuracy
stuck at 44.4% — classifier couldn't extract the signal that physical
sensors were already capturing.

Refactor:
- Drop 8 dead/redundant features (amp_min, amp_range, breath_bp,
  spec_pow, motion_bp, amp_mean, amp_max, amp_iqr, amp_kurt).
- Keep 4 globals: variance, mean_rssi, dom_hz, change_pts.
- Add per-node features × all 6 nodes: amp_std, amp_skew, amp_entropy.
- New N_FEATURES = 22 (was 15). Z-score normalisation kept.

API change: features_from_runtime now takes &[(u8, &[f64])] — caller
must supply per-node amplitudes. New helper current_per_node_amps()
reads AMP_HIST.nbvi_history.back() for all live nodes.

Old data/adaptive_model.json removed (incompatible 15-feature schema).

Retrain result on same 151k frames:
  44.4% → 49.58% accuracy (+5.2 pts)
Total improvement vs 2-node baseline (40.4%): +9.2 pts.

Live confidence distribution now meaningful (0.30-0.85) vs pre-fix
near-uniform 0.04-0.10. Sensor placement matters: n6 (near door, far
from AP) sep_ratio=0.60 best; n1/n5 (near AP) ~0.01-0.06 nearly dead.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
arsen 2026-05-18 00:35:08 +07:00
parent 6ce25cec79
commit e86f650681
4 changed files with 308 additions and 75 deletions

View File

@ -0,0 +1,193 @@
# ADR-118 — Feature Decorrelation + Multi-node Extractor (Adaptive Classifier)
**Status**: Accepted
**Date**: 2026-05-18
**Scope**: `v2/crates/wifi-densepose-sensing-server/src/adaptive_classifier.rs`
(`N_FEATURES`, `features_from_frame`, `features_from_runtime`), call sites in
`main.rs::adaptive_override`, `main.rs:~6200` per-node loop, and
`csi.rs::adaptive_override`.
## Context
After ADR-117 the adaptive_classifier produced **40.4% accuracy** on a
2-node, 7-class training set (52,857 frames). Adding 4 more sensors and
recording the same 7 classes at 6 nodes increased the set to **151,329 frames
(2.9× more data)** but accuracy only moved to **44.4%** (+4 pts).
Diagnostic Python audit (run against both datasets) found three architectural
defects in the feature pipeline, not the data:
| Defect | 2-node set | 6-node set |
|---|---|---|
| Constant feature (`amp_min = 0.00` across all frames — HT20 null subcarrier) | ✗ dead | ✗ dead |
| Multicollinear pairs `|r| > 0.85` | 17 pairs | 21 pairs |
| Top F-stat vs accuracy | F=1,516, acc 40.4% | F=15,497, acc 44.4% |
The 10× higher F-stat on 6-node data confirmed the **signal was getting
stronger** but the classifier couldn't extract it. Root cause:
`features_from_frame` used only `nodes.first()` — 5 of 6 sensors carried
**zero weight** in the feature vector. Adding nodes physically helped, but
only via the small contribution to the 7 aggregated server-level features.
Within a single node, the 8 subcarrier scalars were 90-99% correlated with
each other (mean ≈ std ≈ max ≈ p25/75/90 — they all measure "amplitude
level"). And the 4 energy features (variance, motion_band_power,
breathing_band_power, spectral_power) were 87-99% correlated. The 15-feature
space had effective rank ≈ 5.
## Decisions
### D1 — Drop the dead and redundant features
* **Dropped**: `amp_min` (constant 0), `amp_range = max min ≡ max`
(collinear), `motion_band_power`/`breathing_band_power`/`spectral_power`
(all r > 0.95 with `variance`), `amp_mean`/`amp_max`/`amp_iqr`/`amp_kurt`
(all r > 0.90 with `amp_std`).
* **Kept (globally)**: `variance`, `mean_rssi`, `dominant_freq_hz`,
`change_points` — the 4 server-level features that retained marginal
independence.
### D2 — Per-node features × all 6 nodes
For each node id `N ∈ {1..6}`, extract 3 features:
* `amp_std` — multipath spread (motion-sensitive)
* `amp_skew` — distribution asymmetry (sensitive to dominant scatterer
position relative to this sensor)
* `amp_entropy` — spectral diversity (normalised to [0, 1])
Total: `4 + 6 × 3 = 22 features`. Each node's contribution lives at a fixed
offset (`base = 4 + (node_id - 1) × 3`) so 5 of 6 sensors are no longer
discarded.
Missing-node features are zero-padded; z-score normalisation (already in
the model from ADR-117 era) treats them consistently across train and
classify.
### D3 — `features_from_runtime` signature change
Old:
```rust
pub fn features_from_runtime(feat: &Value, amps: &[f64]) -> [f64; 15]
```
New:
```rust
pub fn features_from_runtime(
feat: &Value,
per_node_amps: &[(u8, &[f64])],
) -> [f64; 22]
```
Three call sites updated:
1. `main.rs::adaptive_override` (global state path) — new helper
`current_per_node_amps()` reads `AMP_HIST.nbvi_history.back()` for each
active node, then passes the slice.
2. `main.rs:~6200` (per-node loop in the broadcast tick task) — same
helper, called once per tick.
3. `csi.rs::adaptive_override` (legacy, no live callers) — degraded to
single-node fallback with `[(1u8, amps)]`; documented as emergency only.
### D4 — Old 15-feature model file is incompatible
`AdaptiveModel` serializes `[f64; N_FEATURES]` arrays. Loading a 15-array
into a 22-slot field fails. `data/adaptive_model.json` removed at deploy
time; first start re-runs `train_from_recordings` over the existing 7 train
files.
## Files Touched
```
v2/crates/wifi-densepose-sensing-server/src/adaptive_classifier.rs:
* N_FEATURES: 15 → 22
* New constants N_GLOBAL_FEATURES=4, N_PER_NODE_FEATURES=3, MAX_NODES=6
* features_from_frame rewritten — multi-node + decorrelated
* features_from_runtime signature changed
* per_node_stats helper (3 scalars: std/skew/entropy)
* Old subcarrier_stats removed
v2/crates/wifi-densepose-sensing-server/src/main.rs:
+ current_per_node_amps() helper (snapshots AMP_HIST.nbvi_history.back())
+ 2 call sites updated to pass &[(u8, &[f64])] instead of &[f64]
v2/crates/wifi-densepose-sensing-server/src/csi.rs:
+ adaptive_override updated to new signature (dead code path, kept for ABI)
data/adaptive_model.json: removed (15-feature incompatible)
docs/adr/ADR-118-feature-decorrelation-multinode.md (this)
```
## Verified Acceptance
Re-ran `POST /api/v1/adaptive/train` against the same 151,329-frame 6-node
recording set:
```
2-node, 15 features: 40.4%
6-node, 15 features: 44.4% (+4.0 from more data)
6-node, 22 features: 49.58% (+5.2 from feature engineering)
```
Total improvement: **+9.2 percentage points** from the baseline, on the
same hardware in the same room.
Live confidence distribution (10s samples post-retrain):
```
absent: conf 0.30-0.85 (was 0.04-0.10 pre-ADR-118)
present_still: conf 0.40-0.85
present_moving: conf 0.30-0.50
active: conf 0.27-0.45
transition: conf 0.84-0.86 (high — model has clear signal for this)
waving: conf — class not active during sample window
```
Confidence is now meaningful (model has separation), whereas pre-ADR-118 the
near-uniform 0.04-0.10 indicated the classifier was essentially flipping a
coin.
### Per-feature class separability (post-train, sep_ratio = between-class
spread / within-class std):
| Feature | sep_ratio | Verdict |
|---|---|---|
| `n6_std` | 0.60 ★ | best — node 6 near door catches both motion + door state |
| `n2_std` | 0.35 | second — node 2 far from AP, high modulation |
| `n6_skew` | 0.25 | useful |
| `n3_skew` | 0.26 | useful |
| `n2_skew` | 0.18 | marginal |
| `n4_std` | 0.14 | marginal |
| `n1_*` | 0.01-0.06 | near AP — almost no class signal |
| `n5_*` | 0.01-0.05 | similar to n1 |
| all `entropy` features | 0.01-0.02 | **dead** — distribution shape doesn't vary by activity |
| `variance` (global) | 0.11 | weak |
| `mean_rssi` (global) | 0.01 | dead at this scale |
## Open Items
* **`*_entropy` features carry no signal** (sep_ratio ~0.01 across all 6
nodes). Could be dropped: 22 → 16 features. Marginal expected gain (~1%),
not worth a follow-up ADR right now.
* **Aggregated server features all sub-0.11**`mean_rssi` / `dom_hz` /
`change_pts` could go too. Would reduce to 12-13 truly useful features.
* **Logistic regression ceiling**`n6_std` alone has sep_ratio 0.60 but
a linear classifier can't fully exploit non-linear class boundaries.
Next big lever is replacing the LogReg with a small MLP or random forest.
Out of scope here.
* **`standing` and `sitting` recordings collapse to one class** — file
naming maps both to `present_still`. They're physically distinct
signatures (different RF profile) but the trainer treats them as one.
Separating them in `classify_recording_name` would add a class but might
lower accuracy due to inherent confusability — TBD via experiment.
* **Sensor placement matters more than algorithm tweaks** — n1/n5 (near AP)
carry almost no class signal. Reposition them away from the AP if
possible (closer to walking zone, farther from the line-of-sight to AP).
## References
* ADR-101 — raw amplitude classifier (the runtime classifier this adaptive
model can override)
* ADR-117 — process hygiene + previous training infrastructure
* `data/recordings/archive_2node_2026-05-17/` — earlier 2-node training
set, kept for comparison; not used by trainer (outside `recordings/`
root scope)

View File

@ -21,94 +21,112 @@ use std::path::{Path, PathBuf};
// ── Feature vector ───────────────────────────────────────────────────────────
/// Extended feature vector: 7 server features + 8 subcarrier-derived features = 15.
const N_FEATURES: usize = 15;
/// ADR-118: feature vector redesigned for multi-node use + multicollinearity
/// reduction. Audit on 7-class training set showed:
/// * 17-21 multicollinear pairs (|r|>0.85) — energy features and amplitude
/// scalars were highly redundant.
/// * `amp_min` constant 0.0 across all frames (null subcarrier of HT20),
/// making `amp_range = amp_max - 0` fully redundant with `amp_max`.
/// * On 6-node data F-stat 10× higher than 2-node, but classifier accuracy
/// barely budged (40→44%) because the prior 15-feature pipeline used only
/// `nodes.first()` — 5 of 6 sensors carried zero weight.
///
/// New 22-feature layout:
/// [0..4] global signal features:
/// variance, mean_rssi, dominant_freq_hz, change_points
/// [4..22] per-node features (6 nodes × 3 features each):
/// per node id N∈{1..6}, base = 4 + (N-1)*3:
/// base+0: amp_std — motion / multipath spread
/// base+1: amp_skew — distribution asymmetry (where strong scatterers are)
/// base+2: amp_entropy — spectral diversity (normalised)
/// Total: 22 features.
const N_GLOBAL_FEATURES: usize = 4;
const N_PER_NODE_FEATURES: usize = 3;
const MAX_NODES: usize = 6;
const N_FEATURES: usize = N_GLOBAL_FEATURES + MAX_NODES * N_PER_NODE_FEATURES;
/// Default class names for backward compatibility with old saved models.
const DEFAULT_CLASSES: &[&str] = &["absent", "present_still", "present_moving", "active"];
/// Extract extended feature vector from a JSONL frame (features + raw amplitudes).
/// Extract extended feature vector from a JSONL frame (features + per-node amplitudes).
/// Missing-node features are zero-padded; z-score normalisation later treats
/// them consistently.
pub fn features_from_frame(frame: &serde_json::Value) -> [f64; N_FEATURES] {
let feat = frame.get("features").cloned().unwrap_or(serde_json::Value::Null);
let nodes = frame.get("nodes").and_then(|n| n.as_array());
let amps: Vec<f64> = nodes
.and_then(|ns| ns.first())
.and_then(|n| n.get("amplitude"))
.and_then(|a| a.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_f64()).collect())
.unwrap_or_default();
let mut out = [0.0f64; N_FEATURES];
// Server-computed features (0-6).
let variance = feat.get("variance").and_then(|v| v.as_f64()).unwrap_or(0.0);
let mbp = feat.get("motion_band_power").and_then(|v| v.as_f64()).unwrap_or(0.0);
let bbp = feat.get("breathing_band_power").and_then(|v| v.as_f64()).unwrap_or(0.0);
let sp = feat.get("spectral_power").and_then(|v| v.as_f64()).unwrap_or(0.0);
let df = feat.get("dominant_freq_hz").and_then(|v| v.as_f64()).unwrap_or(0.0);
let cp = feat.get("change_points").and_then(|v| v.as_f64()).unwrap_or(0.0);
let rssi = feat.get("mean_rssi").and_then(|v| v.as_f64()).unwrap_or(0.0);
// ── Global signal features (0..4) ──
out[0] = feat.get("variance").and_then(|v| v.as_f64()).unwrap_or(0.0);
out[1] = feat.get("mean_rssi").and_then(|v| v.as_f64()).unwrap_or(0.0);
out[2] = feat.get("dominant_freq_hz").and_then(|v| v.as_f64()).unwrap_or(0.0);
out[3] = feat.get("change_points").and_then(|v| v.as_f64()).unwrap_or(0.0);
// Subcarrier-derived features (7-14).
let (amp_mean, amp_std, amp_skew, amp_kurt, amp_iqr, amp_entropy, amp_max, amp_range) =
subcarrier_stats(&amps);
[
variance, mbp, bbp, sp, df, cp, rssi,
amp_mean, amp_std, amp_skew, amp_kurt, amp_iqr, amp_entropy, amp_max, amp_range,
]
// ── Per-node features (4..22) ──
if let Some(nodes) = frame.get("nodes").and_then(|n| n.as_array()) {
for node_obj in nodes {
let nid = node_obj.get("node_id").and_then(|v| v.as_u64()).unwrap_or(0) as usize;
if nid == 0 || nid > MAX_NODES { continue; }
let amps: Vec<f64> = node_obj.get("amplitude")
.or_else(|| node_obj.get("amplitudes"))
.and_then(|a| a.as_array())
.map(|arr| arr.iter().filter_map(|v| v.as_f64()).collect())
.unwrap_or_default();
let (std_a, skew_a, entropy_a) = per_node_stats(&amps);
let base = N_GLOBAL_FEATURES + (nid - 1) * N_PER_NODE_FEATURES;
out[base] = std_a;
out[base + 1] = skew_a;
out[base + 2] = entropy_a;
}
}
out
}
/// Also keep a simpler version for runtime (no JSONL, just FeatureInfo + amps).
pub fn features_from_runtime(feat: &serde_json::Value, amps: &[f64]) -> [f64; N_FEATURES] {
let variance = feat.get("variance").and_then(|v| v.as_f64()).unwrap_or(0.0);
let mbp = feat.get("motion_band_power").and_then(|v| v.as_f64()).unwrap_or(0.0);
let bbp = feat.get("breathing_band_power").and_then(|v| v.as_f64()).unwrap_or(0.0);
let sp = feat.get("spectral_power").and_then(|v| v.as_f64()).unwrap_or(0.0);
let df = feat.get("dominant_freq_hz").and_then(|v| v.as_f64()).unwrap_or(0.0);
let cp = feat.get("change_points").and_then(|v| v.as_f64()).unwrap_or(0.0);
let rssi = feat.get("mean_rssi").and_then(|v| v.as_f64()).unwrap_or(0.0);
let (amp_mean, amp_std, amp_skew, amp_kurt, amp_iqr, amp_entropy, amp_max, amp_range) =
subcarrier_stats(amps);
[
variance, mbp, bbp, sp, df, cp, rssi,
amp_mean, amp_std, amp_skew, amp_kurt, amp_iqr, amp_entropy, amp_max, amp_range,
]
/// Runtime variant: callers pass the already-aggregated feature struct and a
/// slice of (node_id, &amplitudes) pairs. Compatible with the broadcast tick
/// task which has access to all live nodes simultaneously.
pub fn features_from_runtime(
feat: &serde_json::Value,
per_node_amps: &[(u8, &[f64])],
) -> [f64; N_FEATURES] {
let mut out = [0.0f64; N_FEATURES];
out[0] = feat.get("variance").and_then(|v| v.as_f64()).unwrap_or(0.0);
out[1] = feat.get("mean_rssi").and_then(|v| v.as_f64()).unwrap_or(0.0);
out[2] = feat.get("dominant_freq_hz").and_then(|v| v.as_f64()).unwrap_or(0.0);
out[3] = feat.get("change_points").and_then(|v| v.as_f64()).unwrap_or(0.0);
for (nid, amps) in per_node_amps {
let nid = *nid as usize;
if nid == 0 || nid > MAX_NODES { continue; }
let (std_a, skew_a, entropy_a) = per_node_stats(amps);
let base = N_GLOBAL_FEATURES + (nid - 1) * N_PER_NODE_FEATURES;
out[base] = std_a;
out[base + 1] = skew_a;
out[base + 2] = entropy_a;
}
out
}
/// Compute statistical features from raw subcarrier amplitudes.
fn subcarrier_stats(amps: &[f64]) -> (f64, f64, f64, f64, f64, f64, f64, f64) {
/// Compute the 3 per-node statistics used in the new feature vector:
/// std (motion / multipath spread), skew (distribution asymmetry),
/// entropy (spectral diversity, normalised to [0, 1]).
fn per_node_stats(amps: &[f64]) -> (f64, f64, f64) {
if amps.is_empty() {
return (0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0);
return (0.0, 0.0, 0.0);
}
let n = amps.len() as f64;
let mean = amps.iter().sum::<f64>() / n;
let var = amps.iter().map(|a| (a - mean).powi(2)).sum::<f64>() / n;
let std = var.sqrt().max(1e-9);
// Skewness (asymmetry).
let skew = amps.iter().map(|a| ((a - mean) / std).powi(3)).sum::<f64>() / n;
// Kurtosis (peakedness).
let kurt = amps.iter().map(|a| ((a - mean) / std).powi(4)).sum::<f64>() / n - 3.0;
// IQR (inter-quartile range).
let mut sorted = amps.to_vec();
sorted.sort_by(|a, b| a.partial_cmp(b).unwrap());
let q1 = sorted[sorted.len() / 4];
let q3 = sorted[3 * sorted.len() / 4];
let iqr = q3 - q1;
// Spectral entropy (normalised).
let total_power: f64 = amps.iter().map(|a| a * a).sum::<f64>().max(1e-9);
let entropy: f64 = amps.iter()
.map(|a| {
let p = (a * a) / total_power;
if p > 1e-12 { -p * p.ln() } else { 0.0 }
})
.sum::<f64>() / n.ln().max(1e-9); // normalise to [0,1]
let max_val = sorted.last().copied().unwrap_or(0.0);
let range = max_val - sorted.first().copied().unwrap_or(0.0);
(mean, std, skew, kurt, iqr, entropy, max_val, range)
.sum::<f64>() / n.ln().max(1e-9);
(std, skew, entropy)
}
// ── Per-class statistics ─────────────────────────────────────────────────────

View File

@ -481,9 +481,16 @@ pub fn smooth_and_classify_node(ns: &mut NodeState, raw: &mut ClassificationInfo
raw.confidence = (0.4 + sm * 0.6).clamp(0.0, 1.0);
}
/// ADR-118: legacy single-node override variant kept for API compatibility.
/// New callers should query per-node amps from AMP_HIST and pass the full
/// `&[(u8, &[f64])]` slice. This variant degrades to "node 1 only" which
/// produces a feature vector with 5 zero-padded node slots — usable for
/// emergency fallback but the trained model expects the full multi-node
/// vector.
pub fn adaptive_override(state: &AppStateInner, features: &FeatureInfo, classification: &mut ClassificationInfo) {
if let Some(ref model) = state.adaptive_model {
let amps = state.frame_history.back().map(|v| v.as_slice()).unwrap_or(&[]);
let amps_owned: Vec<f64> = state.frame_history.back().cloned().unwrap_or_default();
let per_node_refs: Vec<(u8, &[f64])> = vec![(1u8, amps_owned.as_slice())];
let feat_arr = adaptive_classifier::features_from_runtime(
&serde_json::json!({
"variance": features.variance,
@ -494,7 +501,7 @@ pub fn adaptive_override(state: &AppStateInner, features: &FeatureInfo, classifi
"change_points": features.change_points,
"mean_rssi": features.mean_rssi,
}),
amps,
&per_node_refs,
);
let (label, conf) = model.classify(&feat_arr);
classification.motion_level = label.to_string();

View File

@ -2645,14 +2645,27 @@ fn smooth_and_classify_node(ns: &mut NodeState, raw: &mut ClassificationInfo, ra
raw.confidence = (0.4 + sm * 0.6).clamp(0.0, 1.0);
}
/// ADR-118: collect the latest amplitude vector per node from `AMP_HIST`.
/// The adaptive classifier's new 22-feature pipeline reads 3 features per
/// node × 6 nodes; calling code at the override sites no longer has access
/// to a single global "amps" vector — it needs the per-node breakdown.
fn current_per_node_amps() -> Vec<(u8, Vec<f64>)> {
let map = amp_hist_init().lock().unwrap();
map.iter()
.filter_map(|(nid, st)| {
st.nbvi_history.back().cloned().map(|amps| (*nid, amps))
})
.collect()
}
/// If an adaptive model is loaded, override the classification with the
/// model's prediction. Uses the full 15-feature vector for higher accuracy.
/// model's prediction. Uses the 22-feature multi-node vector (ADR-118)
/// for higher accuracy than the legacy 15-feature single-node vector.
fn adaptive_override(state: &AppStateInner, features: &FeatureInfo, classification: &mut ClassificationInfo) {
if let Some(ref model) = state.adaptive_model {
// Get current frame amplitudes from the latest history entry.
let amps = state.frame_history.back()
.map(|v| v.as_slice())
.unwrap_or(&[]);
let per_node_owned = current_per_node_amps();
let per_node_refs: Vec<(u8, &[f64])> = per_node_owned.iter()
.map(|(n, a)| (*n, a.as_slice())).collect();
let feat_arr = adaptive_classifier::features_from_runtime(
&serde_json::json!({
"variance": features.variance,
@ -2663,7 +2676,7 @@ fn adaptive_override(state: &AppStateInner, features: &FeatureInfo, classificati
"change_points": features.change_points,
"mean_rssi": features.mean_rssi,
}),
amps,
&per_node_refs,
);
let (label, conf) = model.classify(&feat_arr);
classification.motion_level = label.to_string();
@ -6179,10 +6192,12 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
smooth_and_classify_node(ns, &mut classification, raw_motion);
// Adaptive override using cloned model (safe, no raw pointers).
// ADR-118: full multi-node feature vector — pull all 6
// nodes' latest amps from AMP_HIST, not just this node's.
if let Some(ref model) = adaptive_model_clone {
let amps = ns.frame_history.back()
.map(|v| v.as_slice())
.unwrap_or(&[]);
let per_node_owned = current_per_node_amps();
let per_node_refs: Vec<(u8, &[f64])> = per_node_owned.iter()
.map(|(n, a)| (*n, a.as_slice())).collect();
let feat_arr = adaptive_classifier::features_from_runtime(
&serde_json::json!({
"variance": features.variance,
@ -6193,7 +6208,7 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
"change_points": features.change_points,
"mean_rssi": features.mean_rssi,
}),
amps,
&per_node_refs,
);
let (label, conf) = model.classify(&feat_arr);
classification.motion_level = label.to_string();