feat(beyond-sota): ADR-156 M2 — RaBitQ unbiased distance estimator (rigorous published negative on strict-K) (#1056)
* feat(ruvector): RaBitQ unbiased distance estimator (ADR-156 M2)
Implement the real Gao & Long (SIGMOD 2024) RaBitQ contribution on top of
the existing Pass-2 rotation: an unbiased estimator of the inner product /
squared distance recovered from the 1-bit code plus 8 B/vec per-vector side
info (residual_norm + x_dot_o), used to rerank the candidate set instead of
raw Hamming.
- src/estimator.rs (new): EstimatorSketch, SideInfo, EstimatorQuery,
DistanceEstimator (estimate_inner_product / estimate_sq_distance /
ranking_key / cosine_ranking_key), EstimatorBank (topk_estimated[_cosine],
with_centroid). Zero-centroid simplification documented; paper-faithful
centroid path also built.
- src/rotation.rs: extract apply_padded() (full padded FHT frame the code
lives in); apply() now truncates apply_padded(). No behaviour change.
- lib.rs: export estimator types.
Additive + backward-compatible: Pass-1 Sketch / Pass-2 SketchBank / WireSketch
wire format unchanged; all external callers use Pass-1 and are unaffected.
Co-Authored-By: claude-flow <ruv@ruv.net>
* test(ruvector): estimator strict-K coverage harness (ADR-156 M2)
Add measure_estimator (cosine rerank) + measure_estimator_euclidean to the
coverage harness, on the BIT-IDENTICAL fixture / cluster centres / query
stream / cosine ground truth as measure_pass1/measure_pass2 — apples-to-apples
sign-Hamming vs unbiased-estimator-rerank.
Regression tests:
- estimator_rerank_not_worse_than_sign (>= sign-only Pass-2 on a fixed fixture)
- estimator_coverage_is_deterministic
- estimator_coverage_report (--nocapture prints the strict-K table)
MEASURED strict-K (candidate_k=K=8): Pass-1 36.13% -> Pass-2-sign 46.39% ->
estimator-cosine 49.71%. Still short of the ADR-084 90% strict bar; estimator
reaches 95.12% at candidate_k=24 (vs sign 91.60%). Published negative.
Co-Authored-By: claude-flow <ruv@ruv.net>
* docs(ruvector): record RaBitQ estimator measured negative (ADR-156 §11, ADR-084)
- sketch_bench: estimator cosine/euclid columns in the coverage table.
- ADR-156 §11 (new): estimator formula + zero-centroid simplification stated
honestly; strict-K coverage table; RESOLVED-NEGATIVE verdict (49.71% strict,
short of 90%); pinning test names. §5 #2 + §10.5 updated.
- ADR-084 'Pass 2b' (new): estimator landed + measured strict-K vs the bar.
- CHANGELOG [Unreleased]: ADR-156 §11 Milestone-2 entry.
Co-Authored-By: claude-flow <ruv@ruv.net>
91248536bc