This commit is contained in:
ruvnet 2026-06-12 12:23:43 +00:00
parent 64a8e2eed8
commit 0817de143e
2 changed files with 132 additions and 3 deletions

View File

@ -182,9 +182,15 @@ label or behavior change, consistent with leaving their claim surface intact.)
sign-language claim requires labelled clinical/affective/ASL data and reference
standards that do not exist in this repo. The disclaimers + feature gate are the
honest stand-in. Nothing is claimed that is not measured.
- **Criterion benches for `process_frame` budget claims****ACCEPTED-FUTURE**.
`tests/budget_compliance.rs` asserts L/S/H tier wall-clock budgets (25 tests,
passing), but a regression-grade criterion bench is not yet wired.
- **Criterion benches for `process_frame` budget claims** — **DONE (host)**
(ADR-163, 2026-06-12). `benches/process_frame_bench.rs` benches the heaviest
hot paths (`exo_time_crystal` 256×128 autocorrelation, `exo_ghost_hunter`
periodicity, `sec_weapon_detect` per-subcarrier Welford, `med_seizure_detect`
clonic rhythm) and reports committed **host** medians
(`benchmarks/edge-latency/RESULTS.md`). `tests/budget_compliance.rs` continues
to assert the L/S/H tier wall-clock budgets (25 tests, passing). **ESP32-on-
hardware (Xtensa/WASM3) latency remains PENDING** — the host bench is an
upper-bound algorithm-cost proxy, NOT the ESP32 figure (needs hardware).
- **`wasm32-unknown-unknown` `static_mut_refs` confirmation** — **ACCEPTED-FUTURE**
(toolchain): the source pattern is eliminated; a CI job on the wasm target should
assert zero `static_mut_refs` once the target is added to the build image.

View File

@ -0,0 +1,123 @@
# ADR-163: Edge-Latency Measurement — CLAIMED budgets → MEASURED-on-host
- **Status**: accepted
- **Date**: 2026-06-12
- **Deciders**: ruv
- **Tags**: edge-latency, wasm-edge, esp32, cog-inference, criterion, prove-everything, measurement-debt
- **Amends**: ADR-160 (deferred "criterion benches for process_frame budget claims" line now DONE-on-host); ADR-159 (cog inference latency)
## Context — Milestone 9 of the beyond-SOTA sweep
Prior milestones (M5/M6, ADR-159/ADR-160) flagged **measurement debt**: edge
latency budgets asserted in doc-comments and manifests but **never reproduced by
a committed benchmark**. Specifically:
- Many `wifi-densepose-wasm-edge` skill modules document a timing budget *"on
ESP32-S3 WASM3"* (e.g. `exo_time_crystal`: "H (heavy, <10 ms)"). These were
**CLAIMED**, not benchmarked. ADR-160's deferred backlog named exactly this:
*"Criterion benches for `process_frame` budget claims — ACCEPTED-FUTURE."*
- `cog-pose-estimation`'s manifest cites `cold_start_ms_avg: 5.4`, but neither
cog had a `benches/` directory or any committed inference-latency number.
Under the project's **prove-everything / anti-"AI-slop"** directive, a CLAIMED
latency budget that a skeptic cannot reproduce is debt. M9 pays it down — benches
and docs only, **no production-code behavior change** (so nothing republishes).
## Headline
**Converted the CLAIMED edge-latency budgets into MEASURED-on-host numbers, with
the honest host-vs-ESP32 caveat stated everywhere.** Added committed criterion
benches over the heaviest hot paths and a results file a skeptic can re-run. The
ESP32-on-hardware figure remains explicitly **UNMEASURED** — this milestone does
not pretend a laptop reproduces an Xtensa/WASM3 budget.
## Decision — benches landed
### T1 — wasm-edge `process_frame` budget benches
`v2/crates/wifi-densepose-wasm-edge/benches/process_frame_bench.rs` (criterion,
`harness = false`, `required-features = ["std"]`). The crate is **excluded from
the v2 workspace**, so it runs from the crate dir. Benches the M6-audit-named
heaviest hot paths over a **fixed synthetic CSI frame**, each driven through the
public `process_frame` after warming the relevant ring/phase buffers so the
expensive path actually executes:
- `exo_time_crystal::process_frame` — full 256-pt × 128-lag autocorrelation.
- `exo_ghost_hunter::process_frame` — empty-room periodicity / hidden-breathing.
- `sec_weapon_detect::process_frame` — per-subcarrier (MAX_SC=32) Welford.
- `med_seizure_detect::process_frame` — clonic-rhythm path (`#[cfg(feature =
"medical-experimental")]`, only built/run with that gate).
The lib's `bench = false` was set so the libtest harness does not intercept
criterion CLI flags; the `ghost_hunter` bin is already `standalone-bin`-gated and
not built under `--features std`.
**Measured host medians** (Intel Core Ultra 9 285H, native `--release`):
`exo_time_crystal` **17.3 µs** · `exo_ghost_hunter` **1.44 µs** ·
`sec_weapon_detect` **0.42 µs** · `med_seizure_detect` **0.10 µs**.
### T2 — cog inference latency benches
`v2/crates/cog-person-count/benches/infer_bench.rs` and
`v2/crates/cog-pose-estimation/benches/infer_bench.rs` (criterion,
`harness = false`). Each loads the **real** shipped weights from the in-repo
`cog/artifacts/`, asserts the Candle CPU backend (so the stub can never be
silently benched), warms one forward, then times steady-state
`InferenceEngine::infer` over a fixed CSI window on `Device::Cpu`.
**Measured host medians:** cog-person-count **305 µs** · cog-pose-estimation
**305 µs** (steady-state, CPU, real weights).
### T3 — results file
`benchmarks/edge-latency/RESULTS.md`, in the `benchmarks/wiflow-std/RESULTS.md`
style: each number with its exact reproduce command, the machine, the
MEASURED-on-host grade, and the honest caveat.
## The honest caveat (recorded, non-negotiable)
1. **Host ≠ ESP32.** The wasm-edge benches run native x86_64, not Xtensa/WASM3.
A host median is an **upper bound on algorithm work**, not the ESP32 number;
WASM3 interpretation on a ~240 MHz core is 12 orders of magnitude slower than
native `-O`. A host median under budget does **not** prove the ESP32 meets it.
**The ESP32 figure is NOT reproduced here — it needs hardware.**
2. **Bench ≠ the doc-claimed measurement.** The cogs' manifest cites a
**cold-start** number (weight-load included); these benches measure
**steady-state** per-frame `infer`. We report both, labelled, and do not
conflate them. Empirically, pose steady-state (305 µs host) is ~18× under the
5.4 ms cold-start — the expected shape, and exactly why conflating would lie.
## Deferred / still-pending (nothing dropped)
- **ESP32-on-hardware `process_frame` latency****PENDING (hardware)**. Needs
the `wasm32-unknown-unknown` target built + flashed to an ESP32-S3 and timed
under WASM3. The host bench is the algorithm-cost proxy until then.
- **Per-skill *accuracy*** remains **DATA-GATED** (unchanged from ADR-160) —
this ADR measures latency only, never claims detection accuracy.
## Reproduction (MEASURED)
```bash
# T1 — wasm-edge (workspace-excluded → run from the crate dir)
cd v2/crates/wifi-densepose-wasm-edge
cargo bench --features std -- --warm-up-time 1 --measurement-time 2
cargo bench --features std,medical-experimental -- --warm-up-time 1 --measurement-time 2 med_seizure
# T2 — cogs (workspace members)
cd v2
cargo bench -p cog-person-count --no-default-features --bench infer_bench
cargo bench -p cog-pose-estimation --no-default-features --bench infer_bench
# existing tests still green (behavior unchanged)
cargo test -p cog-person-count -p cog-pose-estimation --no-default-features
```
## Consequences
- ADR-160's deferred *"Criterion benches for `process_frame` budget claims"* line
is now **DONE (host)**; the ESP32-on-hardware confirmation is explicitly the
one remaining pending item.
- The cogs now ship committed, reproducible steady-state inference-latency
numbers, cleanly distinguished from the manifest's cold-start claim.
- No runtime behavior changed; no crate republishes. `PROOF.md`'s performance
table and `scripts/prove.sh`'s gated section reference the new benches.