Update README + user guide with dense point cloud features

Added pointcloud section to README (quick start, CLI, performance).
Added comprehensive user guide section: setup, sensors, commands,
pipeline components, API endpoints, training, output formats,
deep room scan, ESP32 provisioning.

Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
ruv 2026-04-19 21:25:19 -04:00
parent ae792aad0d
commit 4ab69359ef
2 changed files with 135 additions and 0 deletions

View File

@ -96,6 +96,42 @@ node scripts/mincut-person-counter.js --port 5006 # Correct person counting
> >
--- ---
### Real-Time Dense Point Cloud (NEW)
RuView now generates **real-time 3D point clouds** by fusing camera depth + WiFi CSI + mmWave radar. All sensors stream simultaneously into a unified spatial model.
| Sensor | Data | Integration |
|--------|------|-------------|
| **Camera** | MiDaS monocular depth (GPU) | 640×480 → 19,200+ depth points per frame |
| **ESP32 CSI** | ADR-018 binary frames (UDP) | RF tomography → 8×8×4 occupancy grid |
| **WiFlow Pose** | 17 COCO keypoints from CSI | Skeleton overlay on point cloud |
| **Vital Signs** | Breathing rate from CSI phase | Stored in ruOS brain every 60s |
| **Motion** | CSI amplitude variance | Adaptive capture rate (skip depth when still) |
**Quick start:**
```bash
cd rust-port/wifi-densepose-rs
cargo build --release -p wifi-densepose-pointcloud
./target/release/ruview-pointcloud serve --port 9880
# Open http://localhost:9880 for live 3D viewer
```
**CLI commands:**
```bash
ruview-pointcloud demo # synthetic demo
ruview-pointcloud serve --port 9880 # live server + Three.js viewer
ruview-pointcloud capture --output room.ply # capture to PLY
ruview-pointcloud train # depth calibration + DPO pairs
ruview-pointcloud cameras # list available cameras
ruview-pointcloud csi-test --count 100 # send test CSI frames
```
**Performance:** 22ms pipeline, 905 req/s API, 40K voxel room model from 20 frames.
**Brain integration:** Spatial observations (motion, vitals, skeleton, occupancy) sync to the ruOS brain every 60 seconds for agent reasoning.
See [PR #405](https://github.com/ruvnet/RuView/pull/405) for full details.
### What's New in v0.7.0 ### What's New in v0.7.0
<details> <details>

View File

@ -536,6 +536,105 @@ Both UIs update in real-time via WebSocket and auto-detect the sensing server on
--- ---
## Dense Point Cloud (Camera + WiFi CSI Fusion)
RuView can generate real-time 3D point clouds by fusing camera depth estimation with WiFi CSI spatial sensing. This creates a spatial model of the environment that updates in real-time.
### Setup
```bash
# Build the pointcloud binary
cd rust-port/wifi-densepose-rs
cargo build --release -p wifi-densepose-pointcloud
# Start the server (auto-detects camera + CSI)
./target/release/ruview-pointcloud serve --port 9880
```
Open `http://localhost:9880` for the interactive Three.js 3D viewer.
### Sensors
| Sensor | Auto-detected | Data |
|--------|--------------|------|
| Camera (`/dev/video0`) | Yes (Linux UVC) | RGB frames → MiDaS depth → 3D points |
| ESP32 CSI (UDP:3333) | Yes (if provisioned) | ADR-018 binary → occupancy + pose + vitals |
| MiDaS depth server (port 9885) | Optional | GPU-accelerated neural depth estimation |
### Commands
| Command | Description |
|---------|-------------|
| `ruview-pointcloud serve --port 9880` | Start HTTP server + Three.js viewer |
| `ruview-pointcloud demo` | Generate synthetic point cloud (no hardware needed) |
| `ruview-pointcloud capture --output room.ply` | Capture single frame to PLY file |
| `ruview-pointcloud cameras` | List available cameras |
| `ruview-pointcloud train --data-dir ./data` | Depth calibration + occupancy training |
| `ruview-pointcloud csi-test --count 100` | Send test CSI frames (no ESP32 needed) |
### Pipeline Components
1. **ADR-018 Parser** — Decodes ESP32 CSI binary frames from UDP, extracts I/Q subcarrier amplitudes and phases
2. **WiFlow Pose** — 17 COCO keypoint estimation from CSI (loads `wiflow-v1.json`, 186K params)
3. **Vital Signs** — Breathing rate from CSI phase analysis (peak counting on stable subcarrier)
4. **Motion Detection** — CSI amplitude variance over 20 frames, triggers adaptive capture
5. **RF Tomography** — Backprojection from per-node RSSI to 8×8×4 occupancy grid
6. **Camera Depth** — MiDaS monocular depth (GPU) with luminance+edge fallback
7. **Sensor Fusion** — Voxel-grid merging of camera depth + CSI occupancy
8. **Brain Bridge** — Stores spatial observations in the ruOS brain every 60 seconds
### API Endpoints
| Endpoint | Method | Returns |
|----------|--------|---------|
| `/health` | GET | `{"status": "ok"}` |
| `/api/status` | GET | Camera, CSI, pipeline state, vitals, motion |
| `/api/cloud` | GET | Point cloud (up to 1000 points) + pipeline data |
| `/api/splats` | GET | Gaussian splats for Three.js rendering |
| `/` | GET | Interactive Three.js 3D viewer |
### Training
The training pipeline calibrates depth estimation and occupancy detection:
```bash
ruview-pointcloud train --data-dir ~/.local/share/ruview/training --brain http://127.0.0.1:9876
```
This captures frames, runs depth calibration (grid search over scale/offset/gamma), trains occupancy thresholds, exports DPO preference pairs, and submits results to the ruOS brain.
### Output Formats
- **PLY** — Standard 3D point cloud (ASCII, with RGB color)
- **Gaussian Splats** — JSON format for Three.js rendering
- **Brain Memories** — Spatial observations stored as `spatial-observation`, `spatial-motion`, `spatial-vitals`
### Deep Room Scan
Capture a high-quality 3D model of the room:
```bash
# Stop the live server first (frees the camera)
# Then capture 20 frames and process with MiDaS
ruview-pointcloud capture --frames 20 --output room_model.ply
```
Result: 40,000+ voxels at 5cm resolution, 12,000+ Gaussian splats.
### ESP32 Provisioning for CSI
To send CSI data to the pointcloud server:
```bash
python3 firmware/esp32-csi-node/provision.py \
--port /dev/ttyACM0 \
--ssid "YourWiFi" --password "YourPassword" \
--target-ip 192.168.1.123 --target-port 3333 \
--node-id 1
```
---
## Vital Sign Detection ## Vital Sign Detection
The system extracts breathing rate and heart rate from CSI signal fluctuations using FFT peak detection. The system extracts breathing rate and heart rate from CSI signal fluctuations using FFT peak detection.