Update README + user guide with dense point cloud features
Added pointcloud section to README (quick start, CLI, performance). Added comprehensive user guide section: setup, sensors, commands, pipeline components, API endpoints, training, output formats, deep room scan, ESP32 provisioning. Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
ruv
parent
ae792aad0d
commit
4ab69359ef
36
README.md
36
README.md
|
|
@ -96,6 +96,42 @@ node scripts/mincut-person-counter.js --port 5006 # Correct person counting
|
|||
>
|
||||
---
|
||||
|
||||
### Real-Time Dense Point Cloud (NEW)
|
||||
|
||||
RuView now generates **real-time 3D point clouds** by fusing camera depth + WiFi CSI + mmWave radar. All sensors stream simultaneously into a unified spatial model.
|
||||
|
||||
| Sensor | Data | Integration |
|
||||
|--------|------|-------------|
|
||||
| **Camera** | MiDaS monocular depth (GPU) | 640×480 → 19,200+ depth points per frame |
|
||||
| **ESP32 CSI** | ADR-018 binary frames (UDP) | RF tomography → 8×8×4 occupancy grid |
|
||||
| **WiFlow Pose** | 17 COCO keypoints from CSI | Skeleton overlay on point cloud |
|
||||
| **Vital Signs** | Breathing rate from CSI phase | Stored in ruOS brain every 60s |
|
||||
| **Motion** | CSI amplitude variance | Adaptive capture rate (skip depth when still) |
|
||||
|
||||
**Quick start:**
|
||||
```bash
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo build --release -p wifi-densepose-pointcloud
|
||||
./target/release/ruview-pointcloud serve --port 9880
|
||||
# Open http://localhost:9880 for live 3D viewer
|
||||
```
|
||||
|
||||
**CLI commands:**
|
||||
```bash
|
||||
ruview-pointcloud demo # synthetic demo
|
||||
ruview-pointcloud serve --port 9880 # live server + Three.js viewer
|
||||
ruview-pointcloud capture --output room.ply # capture to PLY
|
||||
ruview-pointcloud train # depth calibration + DPO pairs
|
||||
ruview-pointcloud cameras # list available cameras
|
||||
ruview-pointcloud csi-test --count 100 # send test CSI frames
|
||||
```
|
||||
|
||||
**Performance:** 22ms pipeline, 905 req/s API, 40K voxel room model from 20 frames.
|
||||
|
||||
**Brain integration:** Spatial observations (motion, vitals, skeleton, occupancy) sync to the ruOS brain every 60 seconds for agent reasoning.
|
||||
|
||||
See [PR #405](https://github.com/ruvnet/RuView/pull/405) for full details.
|
||||
|
||||
### What's New in v0.7.0
|
||||
|
||||
<details>
|
||||
|
|
|
|||
|
|
@ -536,6 +536,105 @@ Both UIs update in real-time via WebSocket and auto-detect the sensing server on
|
|||
|
||||
---
|
||||
|
||||
## Dense Point Cloud (Camera + WiFi CSI Fusion)
|
||||
|
||||
RuView can generate real-time 3D point clouds by fusing camera depth estimation with WiFi CSI spatial sensing. This creates a spatial model of the environment that updates in real-time.
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Build the pointcloud binary
|
||||
cd rust-port/wifi-densepose-rs
|
||||
cargo build --release -p wifi-densepose-pointcloud
|
||||
|
||||
# Start the server (auto-detects camera + CSI)
|
||||
./target/release/ruview-pointcloud serve --port 9880
|
||||
```
|
||||
|
||||
Open `http://localhost:9880` for the interactive Three.js 3D viewer.
|
||||
|
||||
### Sensors
|
||||
|
||||
| Sensor | Auto-detected | Data |
|
||||
|--------|--------------|------|
|
||||
| Camera (`/dev/video0`) | Yes (Linux UVC) | RGB frames → MiDaS depth → 3D points |
|
||||
| ESP32 CSI (UDP:3333) | Yes (if provisioned) | ADR-018 binary → occupancy + pose + vitals |
|
||||
| MiDaS depth server (port 9885) | Optional | GPU-accelerated neural depth estimation |
|
||||
|
||||
### Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `ruview-pointcloud serve --port 9880` | Start HTTP server + Three.js viewer |
|
||||
| `ruview-pointcloud demo` | Generate synthetic point cloud (no hardware needed) |
|
||||
| `ruview-pointcloud capture --output room.ply` | Capture single frame to PLY file |
|
||||
| `ruview-pointcloud cameras` | List available cameras |
|
||||
| `ruview-pointcloud train --data-dir ./data` | Depth calibration + occupancy training |
|
||||
| `ruview-pointcloud csi-test --count 100` | Send test CSI frames (no ESP32 needed) |
|
||||
|
||||
### Pipeline Components
|
||||
|
||||
1. **ADR-018 Parser** — Decodes ESP32 CSI binary frames from UDP, extracts I/Q subcarrier amplitudes and phases
|
||||
2. **WiFlow Pose** — 17 COCO keypoint estimation from CSI (loads `wiflow-v1.json`, 186K params)
|
||||
3. **Vital Signs** — Breathing rate from CSI phase analysis (peak counting on stable subcarrier)
|
||||
4. **Motion Detection** — CSI amplitude variance over 20 frames, triggers adaptive capture
|
||||
5. **RF Tomography** — Backprojection from per-node RSSI to 8×8×4 occupancy grid
|
||||
6. **Camera Depth** — MiDaS monocular depth (GPU) with luminance+edge fallback
|
||||
7. **Sensor Fusion** — Voxel-grid merging of camera depth + CSI occupancy
|
||||
8. **Brain Bridge** — Stores spatial observations in the ruOS brain every 60 seconds
|
||||
|
||||
### API Endpoints
|
||||
|
||||
| Endpoint | Method | Returns |
|
||||
|----------|--------|---------|
|
||||
| `/health` | GET | `{"status": "ok"}` |
|
||||
| `/api/status` | GET | Camera, CSI, pipeline state, vitals, motion |
|
||||
| `/api/cloud` | GET | Point cloud (up to 1000 points) + pipeline data |
|
||||
| `/api/splats` | GET | Gaussian splats for Three.js rendering |
|
||||
| `/` | GET | Interactive Three.js 3D viewer |
|
||||
|
||||
### Training
|
||||
|
||||
The training pipeline calibrates depth estimation and occupancy detection:
|
||||
|
||||
```bash
|
||||
ruview-pointcloud train --data-dir ~/.local/share/ruview/training --brain http://127.0.0.1:9876
|
||||
```
|
||||
|
||||
This captures frames, runs depth calibration (grid search over scale/offset/gamma), trains occupancy thresholds, exports DPO preference pairs, and submits results to the ruOS brain.
|
||||
|
||||
### Output Formats
|
||||
|
||||
- **PLY** — Standard 3D point cloud (ASCII, with RGB color)
|
||||
- **Gaussian Splats** — JSON format for Three.js rendering
|
||||
- **Brain Memories** — Spatial observations stored as `spatial-observation`, `spatial-motion`, `spatial-vitals`
|
||||
|
||||
### Deep Room Scan
|
||||
|
||||
Capture a high-quality 3D model of the room:
|
||||
|
||||
```bash
|
||||
# Stop the live server first (frees the camera)
|
||||
# Then capture 20 frames and process with MiDaS
|
||||
ruview-pointcloud capture --frames 20 --output room_model.ply
|
||||
```
|
||||
|
||||
Result: 40,000+ voxels at 5cm resolution, 12,000+ Gaussian splats.
|
||||
|
||||
### ESP32 Provisioning for CSI
|
||||
|
||||
To send CSI data to the pointcloud server:
|
||||
|
||||
```bash
|
||||
python3 firmware/esp32-csi-node/provision.py \
|
||||
--port /dev/ttyACM0 \
|
||||
--ssid "YourWiFi" --password "YourPassword" \
|
||||
--target-ip 192.168.1.123 --target-port 3333 \
|
||||
--node-id 1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Vital Sign Detection
|
||||
|
||||
The system extracts breathing rate and heart rate from CSI signal fluctuations using FFT peak detection.
|
||||
|
|
|
|||
Loading…
Reference in New Issue