100 lines
4.9 KiB
Markdown
100 lines
4.9 KiB
Markdown
# We audited a state-of-the-art WiFi pose model. Here's what broke, what reproduced, and the 30× smaller model that nearly matches it.
|
||
|
||
*RuView team, June 2026. All numbers measured; full scripts and forensics in the
|
||
[RuView repo](https://github.com/ruvnet/RuView/tree/main/benchmarks/wiflow-std).*
|
||
|
||
## The setup
|
||
|
||
WiFi sensing is having a moment: a 2026 preprint ("WiFlow", arXiv 2602.08661)
|
||
claims **97.25% pose-estimation accuracy (PCK@20) from WiFi signals alone**,
|
||
with a tiny 2.23M-parameter model — and unlike most papers, it ships
|
||
everything: code, trained weights, and a 360,000-sample dataset.
|
||
|
||
We build WiFi sensing systems, so before adopting any external number we run
|
||
it through a simple rule: **a claim is "CLAIMED" until we reproduce it, then
|
||
it's "MEASURED."** Here's what happened when we tried.
|
||
|
||
## Day 1: nothing works
|
||
|
||
- **The code doesn't run.** The package imports a class that doesn't exist.
|
||
(One-line fix.)
|
||
- **The released model scores 0.08%, not 97.25%.** The shipped checkpoint was
|
||
trained under a different data normalization than the shipped dataset —
|
||
it's a real trained model, just not *this* pipeline's model. Even letting it
|
||
cheat with a fitted per-keypoint correction only reaches 72%.
|
||
- **The dataset is corrupted.** Its last 13 files contain garbage values up to
|
||
3.4×10³⁸ (float32's maximum). Subtle consequence: the training loop uses
|
||
fp16 mixed precision with no guards, so the first corrupted batch overflows
|
||
and **permanently poisons the model's BatchNorm statistics**. Training from
|
||
the public download produces NaN from epoch 1, every time.
|
||
- The training script also crashes before its own test phase ever runs
|
||
(calls an undefined function), and ignores its `--data_dir` flag.
|
||
|
||
At this point a less patient reader concludes "fraud." That would be wrong.
|
||
|
||
## Day 1, later: actually, the science is real
|
||
|
||
We repaired the artifacts — fixed the import, zeroed the 9,072 corrupted
|
||
windows, retrained from scratch with the authors' own code and
|
||
hyperparameters on one GPU (~50 minutes):
|
||
|
||
| Metric | Published | Our retrain |
|
||
|---|---|---|
|
||
| PCK@20 | 97.25% | **96.1–96.6%** |
|
||
| PCK@50 | 99.48% | 99.0–99.1% |
|
||
| Params | 2.23M | 2,225,042 (exact) |
|
||
|
||
**The claims reproduce.** What didn't survive contact was the *packaging*:
|
||
wrong checkpoint, corrupted upload, broken glue code. This distinction —
|
||
**artifact rot vs. bad science** — is the single most useful thing a
|
||
reproduction can establish, and you can't establish it without actually
|
||
running the thing.
|
||
|
||
(We filed all six defects upstream with fixes:
|
||
[issue #3](https://github.com/DY2434/WiFlow-WiFi-Pose-Estimation-with-Spatio-Temporal-Decoupling/issues/3).
|
||
And to be clear: the authors released more than 90% of papers do. That's the
|
||
only reason this audit was possible.)
|
||
|
||
## Day 2: the model is also 2.6× too big
|
||
|
||
Once we could train, we asked: does the architecture need 2.23M parameters?
|
||
|
||
| Variant | Params | Accuracy (PCK@20) | Size on disk |
|
||
|---|---|---|---|
|
||
| Original | 2,225,042 | 96.61% | 8.97 MB |
|
||
| **Half** | **843,834** | **96.62%** ✨ | — |
|
||
| Quarter | 338,600 | 96.05% | — |
|
||
| **Tiny** | **56,290** | **94.11%** | **295 KB** |
|
||
|
||
The half-width model **matches the original exactly** (and converges faster).
|
||
The tiny one — 1/39th the parameters — gives up 2.5 points and runs at
|
||
**0.66 ms per inference on a laptop CPU** (~1,500 poses/second) as a 295 KB
|
||
ONNX file. For edge devices, that's the interesting end of the curve.
|
||
|
||
Quantization footnote: the paper's "~2.2 MB int8" estimate is reachable
|
||
(we measured 2.44–2.53 MB) but only via conv-capable toolchains — PyTorch's
|
||
one-line dynamic quantization converts *literally nothing* on this model
|
||
(it has no Linear layers), a trap worth knowing about.
|
||
|
||
## What we took away
|
||
|
||
1. **Run the artifact, not the README.** Every number in a paper is one
|
||
`git clone` away from being either confirmed or understood. Both outcomes
|
||
are valuable; only one is publishable by the original authors.
|
||
2. **fp16 + unvalidated data = silent model death.** Mixed-precision training
|
||
with no NaN/inf guards doesn't fail loudly — it corrupts BatchNorm buffers
|
||
and ships a broken model with a green progress bar. Validate inputs, or
|
||
train in fp32, or guard the autocast.
|
||
3. **Evidence-grade your own claims too.** Mid-audit, the same forensics
|
||
tooling caught one of *our own* published accuracy numbers resting on a
|
||
degenerate evaluation (a constant-output model scored with a flawed
|
||
metric). We retracted it the same day. The rule has to cut both ways or
|
||
it's marketing, not measurement.
|
||
4. **Over-parameterization hides in SOTA tables.** Nobody publishes the
|
||
half-size ablation that matches their headline model. Run it yourself;
|
||
it's an hour of GPU time and sometimes it *is* the result.
|
||
|
||
*Reproduction scripts, corruption masks, the efficiency-sweep configs, and a
|
||
numerically parity-proven Rust port (max divergence 1.2e-7) are all in
|
||
[`benchmarks/wiflow-std/`](https://github.com/ruvnet/RuView/tree/main/benchmarks/wiflow-std).*
|