252 lines
8.6 KiB
Markdown
252 lines
8.6 KiB
Markdown
# ADR-056: RuView Desktop Complete Capabilities Reference
|
|
|
|
## Status
|
|
Accepted
|
|
|
|
## Context
|
|
RuView Desktop is a comprehensive WiFi-based sensing platform that combines hardware management, real-time signal processing, neural network inference, and intelligent monitoring. This ADR documents all integrated capabilities across the desktop application and underlying crates.
|
|
|
|
## Decision
|
|
The RuView Desktop application consolidates all WiFi-DensePose functionality into a single, unified interface with the following capabilities.
|
|
|
|
---
|
|
|
|
## 1. Hardware Management
|
|
|
|
### 1.1 Node Discovery
|
|
- **mDNS discovery**: Automatic detection of ESP32 nodes via Bonjour/Avahi
|
|
- **UDP probe**: Direct UDP broadcast discovery on port 5005
|
|
- **HTTP sweep**: Sequential IP scanning with health checks
|
|
- **Manual registration**: User-defined node configuration
|
|
|
|
### 1.2 Firmware Flashing
|
|
- **Serial flashing**: Direct USB flash via espflash integration
|
|
- **Chip detection**: Automatic ESP32/S2/S3/C3/C6 identification
|
|
- **Progress monitoring**: Real-time progress with speed metrics
|
|
- **Verification**: Post-flash integrity verification
|
|
|
|
### 1.3 OTA Updates
|
|
- **Single-node OTA**: HTTP-based firmware push to individual nodes
|
|
- **Batch OTA**: Coordinated multi-node updates with strategies:
|
|
- `sequential`: One node at a time
|
|
- `tdm_safe`: Respects TDM slot timing
|
|
- `parallel`: Concurrent updates with throttling
|
|
- **Rollback support**: Automatic rollback on verification failure
|
|
- **Version tracking**: Pre/post version comparison
|
|
|
|
### 1.4 Node Configuration
|
|
- **NVS provisioning**: WiFi credentials, node ID, TDM slot assignment
|
|
- **Mesh configuration**: Coordinator/node/aggregator role assignment
|
|
- **TDM scheduling**: Time-division multiplexing slot allocation
|
|
|
|
---
|
|
|
|
## 2. Sensing Server
|
|
|
|
### 2.1 Data Sources
|
|
- **ESP32 CSI**: Real UDP frames from ESP32 hardware (port 5005)
|
|
- **Windows WiFi**: Native Windows RSSI monitoring via netsh
|
|
- **Simulation**: Synthetic data generation for demo/testing
|
|
- **Auto**: Automatic source detection based on available hardware
|
|
|
|
### 2.2 Real-Time Processing
|
|
- **CSI pipeline**: 56-subcarrier amplitude/phase extraction
|
|
- **FFT analysis**: Spectral decomposition for motion detection
|
|
- **Vital signs**: Breathing rate (0.1-0.5 Hz), heart rate (0.8-2.0 Hz)
|
|
- **Motion classification**: still/walking/running/exercising
|
|
- **Presence detection**: Binary presence with confidence score
|
|
|
|
### 2.3 WebSocket Streaming
|
|
- **Sensing endpoint**: `ws://localhost:8765/ws/sensing`
|
|
- **Pose endpoint**: `ws://localhost:8765/ws/pose`
|
|
- **Real-time broadcast**: 10-100 Hz update rate
|
|
- **Multi-client support**: Concurrent WebSocket connections
|
|
|
|
### 2.4 REST API
|
|
- **Health check**: `GET /health`
|
|
- **Status**: `GET /api/status`
|
|
- **Recording control**: `POST /api/recording/start|stop`
|
|
- **Model management**: `GET/POST /api/models`
|
|
|
|
---
|
|
|
|
## 3. Neural Network Inference
|
|
|
|
### 3.1 Model Formats
|
|
- **RVF (RuVector Format)**: Proprietary binary container with:
|
|
- Model weights (quantized f32/f16/i8)
|
|
- Vital sign configuration
|
|
- SONA environment profiles
|
|
- Training provenance
|
|
- Cryptographic attestation
|
|
|
|
### 3.2 Inference Capabilities
|
|
- **Pose estimation**: 17 COCO keypoints from WiFi CSI
|
|
- **Activity recognition**: Multi-class classification
|
|
- **Vital signs**: Breathing and heart rate extraction
|
|
- **Multi-person detection**: Up to 3 simultaneous subjects
|
|
|
|
### 3.3 Self-Learning (SONA)
|
|
- **Environment adaptation**: LoRA-based fine-tuning to room geometry
|
|
- **Profile switching**: Multiple learned environment profiles
|
|
- **Online learning**: Continuous adaptation during runtime
|
|
- **Transfer learning**: Profile export/import between deployments
|
|
|
|
---
|
|
|
|
## 4. WASM Edge Modules
|
|
|
|
### 4.1 Module Management
|
|
- **Upload**: Deploy WASM modules to ESP32 nodes
|
|
- **Start/Stop**: Runtime control of edge processing
|
|
- **Status monitoring**: CPU, memory, execution count
|
|
- **Hot reload**: Update modules without node reboot
|
|
|
|
### 4.2 Supported Operations
|
|
- **Local filtering**: On-device noise reduction
|
|
- **Feature extraction**: Pre-compute features at edge
|
|
- **Compression**: Reduce data before transmission
|
|
- **Custom logic**: User-defined processing pipelines
|
|
|
|
---
|
|
|
|
## 5. Mesh Visualization
|
|
|
|
### 5.1 Network Topology
|
|
- **Live mesh view**: Real-time node connectivity graph
|
|
- **Signal quality**: RSSI/SNR visualization per link
|
|
- **Latency monitoring**: Round-trip time measurement
|
|
- **Packet loss**: Delivery success rate tracking
|
|
|
|
### 5.2 CSI Visualization
|
|
- **Amplitude heatmap**: Per-subcarrier amplitude display
|
|
- **Phase unwrapping**: Continuous phase visualization
|
|
- **Spectrogram**: Time-frequency representation
|
|
- **Signal field**: 3D voxel grid of RF perturbations
|
|
|
|
---
|
|
|
|
## 6. Training & Export
|
|
|
|
### 6.1 Dataset Management
|
|
- **Recording**: Capture CSI frames with annotations
|
|
- **Labeling**: Activity and pose ground truth
|
|
- **Augmentation**: Synthetic data generation
|
|
- **Export**: Standard formats (JSON, CSV, NumPy)
|
|
|
|
### 6.2 Training Pipeline (ADR-023)
|
|
- **Contrastive pretraining**: Self-supervised feature learning
|
|
- **Supervised fine-tuning**: Labeled pose estimation
|
|
- **SONA adaptation**: Environment-specific tuning
|
|
- **Validation**: Cross-environment testing
|
|
|
|
### 6.3 Export Formats
|
|
- **RVF container**: Production deployment format
|
|
- **ONNX**: Interoperability with external tools
|
|
- **PyTorch**: Research and experimentation
|
|
- **Candle**: Rust-native inference
|
|
|
|
---
|
|
|
|
## 7. Security Features
|
|
|
|
### 7.1 Network Security
|
|
- **OTA PSK**: Pre-shared key for firmware updates
|
|
- **Node authentication**: MAC-based node verification
|
|
- **Encrypted transport**: Optional TLS for API endpoints
|
|
|
|
### 7.2 Code Signing
|
|
- **Firmware verification**: Hash-based integrity checks
|
|
- **WASM attestation**: Module signature validation
|
|
- **Model provenance**: Training lineage tracking
|
|
|
|
---
|
|
|
|
## 8. Configuration & Settings
|
|
|
|
### 8.1 Server Configuration
|
|
- **Ports**: HTTP (8080), WebSocket (8765), UDP (5005)
|
|
- **Bind address**: Localhost or network-wide
|
|
- **Data source**: auto/wifi/esp32/simulate
|
|
- **Log level**: debug/info/warn/error
|
|
|
|
### 8.2 Application Settings
|
|
- **Theme**: Dark/light mode
|
|
- **Auto-discovery**: Periodic node scanning
|
|
- **Discovery interval**: Configurable scan frequency
|
|
- **UI customization**: Responsive layout options
|
|
|
|
---
|
|
|
|
## 9. Crate Architecture
|
|
|
|
| Crate | Capabilities |
|
|
|-------|-------------|
|
|
| `wifi-densepose-core` | CSI frame primitives, traits, error types |
|
|
| `wifi-densepose-signal` | FFT, phase unwrapping, vital signs, RuvSense |
|
|
| `wifi-densepose-nn` | ONNX/PyTorch/Candle inference backends |
|
|
| `wifi-densepose-train` | Training pipeline, dataset, metrics |
|
|
| `wifi-densepose-mat` | Mass casualty assessment tool |
|
|
| `wifi-densepose-hardware` | ESP32 protocol, TDM, channel hopping |
|
|
| `wifi-densepose-ruvector` | Cross-viewpoint fusion, attention |
|
|
| `wifi-densepose-api` | REST API (Axum) |
|
|
| `wifi-densepose-db` | Postgres/SQLite/Redis persistence |
|
|
| `wifi-densepose-config` | Configuration management |
|
|
| `wifi-densepose-wasm` | Browser WASM bindings |
|
|
| `wifi-densepose-cli` | Command-line interface |
|
|
| `wifi-densepose-sensing-server` | Real-time sensing server |
|
|
| `wifi-densepose-wifiscan` | Multi-BSSID scanning |
|
|
| `wifi-densepose-vitals` | Vital sign extraction |
|
|
| `wifi-densepose-desktop` | Tauri desktop application |
|
|
|
|
---
|
|
|
|
## 10. UI Design System (ADR-053)
|
|
|
|
### 10.1 Pages
|
|
- **Dashboard**: Overview, node status, quick actions
|
|
- **Discovery**: Network scanning interface
|
|
- **Nodes**: Node management and configuration
|
|
- **Flash**: Serial firmware flashing
|
|
- **OTA**: Over-the-air update management
|
|
- **Edge Modules**: WASM deployment
|
|
- **Sensing**: Real-time monitoring with server control
|
|
- **Mesh View**: Network topology visualization
|
|
- **Settings**: Application configuration
|
|
|
|
### 10.2 Components
|
|
- **StatusBadge**: Health indicator
|
|
- **NodeCard**: Node information display
|
|
- **LogViewer**: Real-time log streaming
|
|
- **ActivityFeed**: Sensing data visualization
|
|
- **ProgressBar**: Operation progress
|
|
- **ConfigForm**: Settings input
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
- **Unified interface**: All capabilities in one application
|
|
- **Bundled deployment**: Single package with server included
|
|
- **Real-time feedback**: WebSocket-based live updates
|
|
- **Cross-platform**: macOS, Windows, Linux support
|
|
- **Extensible**: WASM modules, custom models, API access
|
|
|
|
### Negative
|
|
- **Larger bundle**: ~6MB app + ~2.6MB server
|
|
- **Complexity**: Many features require learning curve
|
|
- **Hardware dependency**: Full functionality requires ESP32 nodes
|
|
|
|
### Neutral
|
|
- Documentation required for all features
|
|
- Training materials needed for advanced capabilities
|
|
- Community contributions welcome
|
|
|
|
## Related ADRs
|
|
- ADR-053: UI Design System
|
|
- ADR-054: Desktop Full Implementation
|
|
- ADR-055: Integrated Sensing Server
|
|
- ADR-023: 8-Phase Training Pipeline
|
|
- ADR-016: RuVector Integration
|