Compare commits

...

8 Commits

Author SHA1 Message Date
ruv d5c457aa30 Add CSI fingerprint DB + night mode detection
Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 21:57:01 -04:00
ruv b2e3f27fa1 Enhance viewer: skeleton overlay, weather, buildings, better camera
Add COCO skeleton rendering with yellow keypoint spheres and white bone
lines, info panel sections for weather/buildings/CSI rate/confidence,
overhead camera at (0,2,-4), and denser point size with sizeAttenuation.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 21:55:30 -04:00
ruv e39a35edee Fix OSM/SRTM queries, add change detection + night mode
- OSM: use inclusive building filter with relation query and 25s timeout
- SRTM: switch to NASA public mirror with viewfinderpanoramas fallback
- Add detect_tile_changes() for pixel-diff satellite change detection
- Add is_night() solar-declination model for CSI-only night mode
- 6 new unit tests (night mode + tile change detection)

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 21:54:55 -04:00
ruv f49ecb163f Update ADR-044: add Common Crawl WET, NASA FIRMS, OpenAQ, Overture Maps sources
Extended geospatial data sources leveraging ruvector's existing web_ingest
and Common Crawl support for hyperlocal context.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 21:41:54 -04:00
ruv c79543283b Add ruview-geo: geospatial satellite integration (11 modules, 8/8 tests)
New crate with free satellite imagery, terrain, OSM, weather, and brain integration.

Modules: types, coord, locate, cache, tiles, terrain, osm, register, fuse, brain, temporal
Tests: 8 passed (haversine, ENU roundtrip, tiles, HGT parse, registration)
Validation: real data — 43.49N 79.71W, 4 Sentinel-2 tiles, 2°C weather, brain stored

Data sources (all free, no API keys):
- EOX Sentinel-2 cloudless (10m satellite tiles)
- SRTM GL1 (30m elevation)
- Overpass API (OSM buildings/roads)
- ip-api.com (geolocation)
- Open Meteo (weather)

ADR-044 documents architecture decisions.
README.md in crate subdirectory.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 21:39:11 -04:00
ruv 4ab69359ef Update README + user guide with dense point cloud features
Added pointcloud section to README (quick start, CLI, performance).
Added comprehensive user guide section: setup, sensors, commands,
pipeline components, API endpoints, training, output formats,
deep room scan, ESP32 provisioning.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 21:25:19 -04:00
ruv ae792aad0d Add brain bridge — sparse spatial observation sync every 60s
Stores room scan summaries, motion events, and vital signs
in the ruOS brain as memories. Only syncs every 120 frames
(~60 seconds) to keep the brain sparse and optimized.

Categories: spatial-observation, spatial-motion, spatial-vitals.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 20:38:17 -04:00
ruv 898d90f689 Complete 7-component sensor fusion pipeline (all working)
1. ADR-018 binary parser — decodes ESP32 CSI UDP frames, extracts I/Q subcarriers
2. WiFlow pose — 17 COCO keypoints from CSI (186K param model loaded)
3. Camera depth — MiDaS on CUDA + luminance fallback
4. Sensor fusion — camera depth + CSI occupancy grid + skeleton overlay
5. RF tomography — ISTA-inspired backprojection from per-node RSSI
6. Vital signs — breathing rate from CSI phase analysis
7. Motion-adaptive — skip expensive depth when CSI shows no motion

Live results: 510 CSI frames/session, 17 keypoints, 26% motion, 40 BPM breathing.
Both ESP32 nodes provisioned to send CSI to 192.168.1.123:3333.
Magic number fix: supports both 0xC5110001 (v1) and 0xC5110006 (v6) frames.

Co-Authored-By: claude-flow <ruv@ruv.net>
2026-04-19 19:51:54 -04:00
25 changed files with 2467 additions and 124 deletions

View File

@ -96,6 +96,42 @@ node scripts/mincut-person-counter.js --port 5006 # Correct person counting
>
---
### Real-Time Dense Point Cloud (NEW)
RuView now generates **real-time 3D point clouds** by fusing camera depth + WiFi CSI + mmWave radar. All sensors stream simultaneously into a unified spatial model.
| Sensor | Data | Integration |
|--------|------|-------------|
| **Camera** | MiDaS monocular depth (GPU) | 640×480 → 19,200+ depth points per frame |
| **ESP32 CSI** | ADR-018 binary frames (UDP) | RF tomography → 8×8×4 occupancy grid |
| **WiFlow Pose** | 17 COCO keypoints from CSI | Skeleton overlay on point cloud |
| **Vital Signs** | Breathing rate from CSI phase | Stored in ruOS brain every 60s |
| **Motion** | CSI amplitude variance | Adaptive capture rate (skip depth when still) |
**Quick start:**
```bash
cd rust-port/wifi-densepose-rs
cargo build --release -p wifi-densepose-pointcloud
./target/release/ruview-pointcloud serve --port 9880
# Open http://localhost:9880 for live 3D viewer
```
**CLI commands:**
```bash
ruview-pointcloud demo # synthetic demo
ruview-pointcloud serve --port 9880 # live server + Three.js viewer
ruview-pointcloud capture --output room.ply # capture to PLY
ruview-pointcloud train # depth calibration + DPO pairs
ruview-pointcloud cameras # list available cameras
ruview-pointcloud csi-test --count 100 # send test CSI frames
```
**Performance:** 22ms pipeline, 905 req/s API, 40K voxel room model from 20 frames.
**Brain integration:** Spatial observations (motion, vitals, skeleton, occupancy) sync to the ruOS brain every 60 seconds for agent reasoning.
See [PR #405](https://github.com/ruvnet/RuView/pull/405) for full details.
### What's New in v0.7.0
<details>

View File

@ -0,0 +1,65 @@
# ADR-044: Geospatial Satellite Integration
## Status
Accepted
## Context
RuView generates real-time 3D point clouds from camera + WiFi CSI, but these exist in a local coordinate frame with no geographic reference. Integrating free satellite imagery, terrain elevation, and map data provides environmental context that enables the ruOS brain to reason about the physical world beyond the room.
## Decision
### Data Sources (all free, no API keys)
| Source | Data | Resolution | Update | Format |
|--------|------|-----------|--------|--------|
| EOX Sentinel-2 Cloudless | Satellite tiles | 10m | Static mosaic | XYZ/JPEG |
| SRTM GL1 (NASA) | Elevation/DEM | 30m (1-arcsec) | Static | Binary HGT |
| Overpass API (OSM) | Buildings, roads | Vector | Real-time | JSON |
| ip-api.com | IP geolocation | ~1km | Per-request | JSON |
| Sentinel-2 STAC | Temporal satellite | 10m | Every 5 days | COG/STAC |
| Open Meteo | Weather | Point | Hourly | JSON |
### Architecture
Pure Rust implementation in `wifi-densepose-geo` crate. No GDAL/PROJ/GEOS — coordinate transforms implemented directly (~250 LOC). Tile caching on disk at `~/.local/share/ruview/geo-cache/`.
### Coordinate System
- WGS84 for geographic coordinates
- ENU (East-North-Up) as the bridge between local sensor frame and world
- Local sensor frame: camera origin, +Z forward, +Y up
### Temporal Awareness
Nightly scheduled fetch of Sentinel-2 latest imagery + OSM diffs + weather.
Changes detected via image comparison and stored as brain memories for
contrastive learning.
### Brain Integration
Geospatial context stored as brain memories:
- `spatial-geo`: location, elevation, nearby landmarks
- `spatial-change`: detected changes in satellite/OSM data
- `spatial-weather`: current conditions + forecast
- `spatial-season`: vegetation index, snow cover, seasonal patterns
- `spatial-local`: hyperlocal web context from Common Crawl WET
### Extended Data Sources (via ruvector WET/Common Crawl)
| Source | Data | Use |
|--------|------|-----|
| Common Crawl WET | Web text near location | Local business info, reviews, events |
| Wikidata | Structured knowledge | Building names, POI descriptions |
| NASA FIRMS | Active fire (3-hour) | Safety alerts |
| USGS Earthquakes | Seismic events | Safety context |
| OpenAQ | Air quality (PM2.5) | Environmental health |
| Overture Maps | Building footprints (Meta/MS) | Higher quality than OSM |
The ruvector brain server has existing `web_ingest` + Common Crawl support.
WET files filtered by geographic URL patterns provide hyperlocal context.
## Consequences
### Positive
- Agent gains environmental awareness beyond the room
- Temporal data enables seasonal calibration of CSI sensing
- Change detection finds construction, vegetation, weather effects
- All data sources are genuinely free with no API keys
### Negative
- Initial data fetch requires internet (~2MB tiles + ~25MB DEM)
- Cached data becomes stale (mitigated by nightly refresh)
- IP geolocation has ~1km accuracy (mitigated by manual override)

View File

@ -536,6 +536,105 @@ Both UIs update in real-time via WebSocket and auto-detect the sensing server on
---
## Dense Point Cloud (Camera + WiFi CSI Fusion)
RuView can generate real-time 3D point clouds by fusing camera depth estimation with WiFi CSI spatial sensing. This creates a spatial model of the environment that updates in real-time.
### Setup
```bash
# Build the pointcloud binary
cd rust-port/wifi-densepose-rs
cargo build --release -p wifi-densepose-pointcloud
# Start the server (auto-detects camera + CSI)
./target/release/ruview-pointcloud serve --port 9880
```
Open `http://localhost:9880` for the interactive Three.js 3D viewer.
### Sensors
| Sensor | Auto-detected | Data |
|--------|--------------|------|
| Camera (`/dev/video0`) | Yes (Linux UVC) | RGB frames → MiDaS depth → 3D points |
| ESP32 CSI (UDP:3333) | Yes (if provisioned) | ADR-018 binary → occupancy + pose + vitals |
| MiDaS depth server (port 9885) | Optional | GPU-accelerated neural depth estimation |
### Commands
| Command | Description |
|---------|-------------|
| `ruview-pointcloud serve --port 9880` | Start HTTP server + Three.js viewer |
| `ruview-pointcloud demo` | Generate synthetic point cloud (no hardware needed) |
| `ruview-pointcloud capture --output room.ply` | Capture single frame to PLY file |
| `ruview-pointcloud cameras` | List available cameras |
| `ruview-pointcloud train --data-dir ./data` | Depth calibration + occupancy training |
| `ruview-pointcloud csi-test --count 100` | Send test CSI frames (no ESP32 needed) |
### Pipeline Components
1. **ADR-018 Parser** — Decodes ESP32 CSI binary frames from UDP, extracts I/Q subcarrier amplitudes and phases
2. **WiFlow Pose** — 17 COCO keypoint estimation from CSI (loads `wiflow-v1.json`, 186K params)
3. **Vital Signs** — Breathing rate from CSI phase analysis (peak counting on stable subcarrier)
4. **Motion Detection** — CSI amplitude variance over 20 frames, triggers adaptive capture
5. **RF Tomography** — Backprojection from per-node RSSI to 8×8×4 occupancy grid
6. **Camera Depth** — MiDaS monocular depth (GPU) with luminance+edge fallback
7. **Sensor Fusion** — Voxel-grid merging of camera depth + CSI occupancy
8. **Brain Bridge** — Stores spatial observations in the ruOS brain every 60 seconds
### API Endpoints
| Endpoint | Method | Returns |
|----------|--------|---------|
| `/health` | GET | `{"status": "ok"}` |
| `/api/status` | GET | Camera, CSI, pipeline state, vitals, motion |
| `/api/cloud` | GET | Point cloud (up to 1000 points) + pipeline data |
| `/api/splats` | GET | Gaussian splats for Three.js rendering |
| `/` | GET | Interactive Three.js 3D viewer |
### Training
The training pipeline calibrates depth estimation and occupancy detection:
```bash
ruview-pointcloud train --data-dir ~/.local/share/ruview/training --brain http://127.0.0.1:9876
```
This captures frames, runs depth calibration (grid search over scale/offset/gamma), trains occupancy thresholds, exports DPO preference pairs, and submits results to the ruOS brain.
### Output Formats
- **PLY** — Standard 3D point cloud (ASCII, with RGB color)
- **Gaussian Splats** — JSON format for Three.js rendering
- **Brain Memories** — Spatial observations stored as `spatial-observation`, `spatial-motion`, `spatial-vitals`
### Deep Room Scan
Capture a high-quality 3D model of the room:
```bash
# Stop the live server first (frees the camera)
# Then capture 20 frames and process with MiDaS
ruview-pointcloud capture --frames 20 --output room_model.ply
```
Result: 40,000+ voxels at 5cm resolution, 12,000+ Gaussian splats.
### ESP32 Provisioning for CSI
To send CSI data to the pointcloud server:
```bash
python3 firmware/esp32-csi-node/provision.py \
--port /dev/ttyACM0 \
--ssid "YourWiFi" --password "YourPassword" \
--target-ip 192.168.1.123 --target-port 3333 \
--node-id 1
```
---
## Vital Sign Detection
The system extracts breathing rate and heart rate from CSI signal fluctuations using FFT peak detection.

View File

@ -5446,6 +5446,18 @@ version = "2.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "178f93f84a4a72c582026a45d9b8710acf188df4a22a25434c5dbba1df6c4cac"
[[package]]
name = "ruview-geo"
version = "0.1.0"
dependencies = [
"anyhow",
"chrono",
"reqwest 0.12.28",
"serde",
"serde_json",
"tokio",
]
[[package]]
name = "ryu"
version = "1.0.23"

View File

@ -18,6 +18,7 @@ members = [
"crates/wifi-densepose-ruvector",
"crates/wifi-densepose-desktop",
"crates/wifi-densepose-pointcloud",
"crates/wifi-densepose-geo",
]
# ADR-040: WASM edge crate targets wasm32-unknown-unknown (no_std),
# excluded from workspace to avoid breaking `cargo test --workspace`.

View File

@ -0,0 +1,13 @@
[package]
name = "ruview-geo"
version = "0.1.0"
edition = "2021"
description = "Geospatial satellite integration — free satellite tiles, DEM, OSM, temporal tracking"
[dependencies]
serde = { workspace = true }
serde_json = { workspace = true }
tokio = { workspace = true }
anyhow = { workspace = true }
reqwest = { version = "0.12", features = ["json", "native-tls"], default-features = false }
chrono = "0.4"

View File

@ -0,0 +1,105 @@
# ruview-geo — Geospatial Satellite Integration
Free satellite imagery, terrain elevation, and map data for RuView spatial sensing. No API keys required.
## What It Does
Integrates your local sensor data (camera + WiFi CSI point cloud) with geographic context:
- **Satellite tiles** — 10m Sentinel-2 cloudless imagery for your location
- **Elevation** — SRTM 30m DEM for terrain modeling
- **Buildings + roads** — OpenStreetMap data via Overpass API
- **Weather** — Open Meteo current conditions + forecast
- **Geo-registration** — maps local sensor coordinates to WGS84
- **Temporal tracking** — detects changes over time (construction, vegetation, weather)
- **Brain integration** — stores geospatial context as ruOS brain memories
## Data Sources (all free, no API keys)
| Source | Data | Resolution | License |
|--------|------|-----------|---------|
| [EOX S2 Cloudless](https://s2maps.eu/) | Satellite tiles | 10m | CC-BY-4.0 |
| [SRTM GL1](https://portal.opentopography.org/) | Elevation/DEM | 30m | Public domain |
| [Overpass API](https://overpass-api.de/) | OSM buildings/roads | Vector | ODbL |
| [ip-api.com](http://ip-api.com/) | IP geolocation | ~1km | Free |
| [Open Meteo](https://open-meteo.com/) | Weather | Point | CC-BY-4.0 |
## Modules
| Module | LOC | Purpose |
|--------|-----|---------|
| `types.rs` | 140 | GeoPoint, GeoBBox, TileCoord, ElevationGrid, OsmFeature |
| `coord.rs` | 80 | WGS84/ENU transforms, tile math, haversine distance |
| `locate.rs` | 45 | IP geolocation with caching |
| `cache.rs` | 55 | Disk cache (`~/.local/share/ruview/geo-cache/`) |
| `tiles.rs` | 80 | Sentinel-2/ESRI/OSM tile fetcher |
| `terrain.rs` | 100 | SRTM HGT parser, elevation lookup |
| `osm.rs` | 150 | Overpass API client, building/road extraction |
| `register.rs` | 50 | Local-to-WGS84 coordinate registration |
| `fuse.rs` | 70 | Multi-source scene builder + summary |
| `brain.rs` | 30 | Store geo context in ruOS brain |
| `temporal.rs` | 100 | Weather, OSM change detection |
## Usage
```rust
use ruview_geo::{fuse, brain, temporal};
// Build geo scene for current location
let scene = fuse::build_scene(500.0).await?; // 500m radius
println!("{}", fuse::summarize(&scene));
// "Location: 43.6532N, 79.3832W, elevation 76m ASL.
// 23 buildings within view. 8 roads nearby (King St, Queen St).
// 12 satellite tiles at zoom 16."
// Store in brain
brain::store_geo_context(&scene).await?;
// Fetch weather
let weather = temporal::fetch_weather(&scene.location).await?;
// temperature: 12°C, partly cloudy, humidity 65%
```
## Brain Integration
Geospatial context is stored as brain memories:
| Category | Content | Frequency |
|----------|---------|-----------|
| `spatial-geo` | Location, elevation, buildings, roads | On startup + daily |
| `spatial-weather` | Temperature, conditions, humidity, wind | Nightly |
| `spatial-change` | New/removed buildings, road changes | Nightly diff |
The ruOS agent can search: "what buildings are near me?" or "what's the weather?" and get geospatial context from the brain.
## Security
- No API keys stored or transmitted
- IP geolocation uses HTTP (not HTTPS) — location is approximate (~1km)
- All tile fetches use HTTPS except ip-api.com
- Path traversal protection in cache key sanitization
- No user data sent to external services
- All data cached locally after first fetch
## Architecture
```
IP Geolocation ──→ (lat, lon)
┌─────────────┼─────────────┐
▼ ▼ ▼
Sentinel-2 SRTM DEM Overpass API
(tiles) (elevation) (buildings/roads)
│ │ │
└─────────────┼─────────────┘
GeoScene (fused)
┌───────┴───────┐
▼ ▼
Brain Memory Three.js Viewer
```
## License
MIT (same as RuView)

View File

@ -0,0 +1,47 @@
use ruview_geo::*;
#[tokio::main]
async fn main() -> anyhow::Result<()> {
println!("╔══════════════════════════════════════════════╗");
println!("║ ruview-geo — Real Data Validation ║");
println!("╚══════════════════════════════════════════════╝\n");
let t0 = std::time::Instant::now();
let cache = cache::TileCache::new("/tmp/ruview-geo-validate");
let loc = locate::get_location(&format!("{}/location.json", cache.base_dir.display())).await?;
println!(" Location: {:.4}N, {:.4}W", loc.lat, loc.lon);
let bbox = GeoBBox::from_center(&loc, 300.0);
let tiles_list = tiles::fetch_area(&tiles::TileProvider::Sentinel2Cloudless, &bbox, 16, &cache).await?;
println!(" Tiles: {} ({:.0}KB)", tiles_list.len(),
tiles_list.iter().map(|t| t.data.len()).sum::<usize>() as f64 / 1024.0);
let dem = terrain::fetch_elevation(&loc, &cache).await?;
println!(" Elevation: {:.0}m (grid {}x{})", terrain::elevation_at(&dem, &loc), dem.cols, dem.rows);
let buildings = osm::fetch_buildings(&loc, 300.0).await.unwrap_or_default();
let roads = osm::fetch_roads(&loc, 300.0).await.unwrap_or_default();
println!(" OSM: {} buildings, {} roads", buildings.len(), roads.len());
let weather = temporal::fetch_weather(&loc).await?;
println!(" Weather: {:.0}°C humidity={:.0}% wind={:.1}m/s",
weather.temperature_c, weather.humidity_pct, weather.wind_speed_ms);
let scene = GeoScene {
location: loc.clone(), bbox, elevation_m: terrain::elevation_at(&dem, &loc),
buildings, roads, tile_count: tiles_list.len(),
registration: register::auto_register(&loc),
last_updated: chrono::Utc::now().to_rfc3339(),
};
println!("\n {}", fuse::summarize(&scene));
match brain::store_geo_context(&scene).await {
Ok(n) => println!(" Brain: {} memories stored", n),
Err(e) => println!(" Brain: {e}"),
}
println!("\n Total: {}ms | Cache: {:.0}KB",
t0.elapsed().as_millis(), cache.size_bytes() as f64 / 1024.0);
Ok(())
}

View File

@ -0,0 +1,28 @@
//! Brain integration — store geospatial context in ruOS brain.
use crate::fuse;
use crate::types::GeoScene;
use anyhow::Result;
const BRAIN_URL: &str = "http://127.0.0.1:9876";
/// Store geospatial context in the brain.
pub async fn store_geo_context(scene: &GeoScene) -> Result<u32> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.build()?;
let mut stored = 0u32;
// Store location summary
let summary = fuse::summarize(scene);
let body = serde_json::json!({
"category": "spatial-geo",
"content": summary,
});
if client.post(format!("{BRAIN_URL}/memories")).json(&body).send().await.is_ok() {
stored += 1;
}
Ok(stored)
}

View File

@ -0,0 +1,61 @@
//! Disk cache for tiles, DEM, and OSM data.
use anyhow::Result;
use std::path::{Path, PathBuf};
pub struct TileCache {
pub base_dir: PathBuf,
}
impl TileCache {
pub fn new(base_dir: &str) -> Self {
let expanded = base_dir.replace('~', &std::env::var("HOME").unwrap_or_default());
let path = PathBuf::from(expanded);
let _ = std::fs::create_dir_all(&path);
Self { base_dir: path }
}
pub fn default_cache() -> Self {
Self::new("~/.local/share/ruview/geo-cache")
}
pub fn get(&self, key: &str) -> Option<Vec<u8>> {
let path = self.key_path(key);
std::fs::read(&path).ok()
}
pub fn put(&self, key: &str, data: &[u8]) -> Result<()> {
let path = self.key_path(key);
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent)?;
}
std::fs::write(&path, data)?;
Ok(())
}
pub fn has(&self, key: &str) -> bool {
self.key_path(key).exists()
}
pub fn size_bytes(&self) -> u64 {
walkdir(self.base_dir.as_path())
}
fn key_path(&self, key: &str) -> PathBuf {
// Sanitize key to prevent path traversal
let safe_key = key.replace("..", "_").replace('/', "_");
self.base_dir.join(safe_key)
}
}
fn walkdir(path: &Path) -> u64 {
std::fs::read_dir(path)
.into_iter()
.flatten()
.filter_map(|e| e.ok())
.map(|e| {
if e.path().is_dir() { walkdir(&e.path()) }
else { e.metadata().map(|m| m.len()).unwrap_or(0) }
})
.sum()
}

View File

@ -0,0 +1,72 @@
//! Coordinate transforms — WGS84, UTM, ENU, tile math.
use crate::types::{GeoPoint, GeoBBox, TileCoord};
const WGS84_A: f64 = 6_378_137.0;
const WGS84_F: f64 = 1.0 / 298.257_223_563;
const WGS84_E2: f64 = 2.0 * WGS84_F - WGS84_F * WGS84_F;
/// Haversine distance in meters.
pub fn haversine(a: &GeoPoint, b: &GeoPoint) -> f64 {
let dlat = (b.lat - a.lat).to_radians();
let dlon = (b.lon - a.lon).to_radians();
let lat1 = a.lat.to_radians();
let lat2 = b.lat.to_radians();
let h = (dlat / 2.0).sin().powi(2) + lat1.cos() * lat2.cos() * (dlon / 2.0).sin().powi(2);
2.0 * WGS84_A * h.sqrt().asin()
}
/// WGS84 to local ENU (East-North-Up) relative to origin, in meters.
pub fn wgs84_to_enu(point: &GeoPoint, origin: &GeoPoint) -> [f64; 3] {
let dlat = (point.lat - origin.lat).to_radians();
let dlon = (point.lon - origin.lon).to_radians();
let lat = origin.lat.to_radians();
let east = dlon * WGS84_A * lat.cos();
let north = dlat * WGS84_A;
let up = point.alt - origin.alt;
[east, north, up]
}
/// Local ENU to WGS84.
pub fn enu_to_wgs84(enu: &[f64; 3], origin: &GeoPoint) -> GeoPoint {
let lat = origin.lat.to_radians();
let dlat = enu[1] / WGS84_A;
let dlon = enu[0] / (WGS84_A * lat.cos());
GeoPoint {
lat: origin.lat + dlat.to_degrees(),
lon: origin.lon + dlon.to_degrees(),
alt: origin.alt + enu[2],
}
}
/// WGS84 to XYZ tile coordinates (Slippy Map).
pub fn wgs84_to_tile(lat: f64, lon: f64, zoom: u8) -> TileCoord {
let n = 2f64.powi(zoom as i32);
let x = ((lon + 180.0) / 360.0 * n).floor() as u32;
let lat_rad = lat.to_radians();
let y = ((1.0 - lat_rad.tan().asinh() / std::f64::consts::PI) / 2.0 * n).floor() as u32;
TileCoord { z: zoom, x, y }
}
/// Tile bounds in WGS84.
pub fn tile_bounds(coord: &TileCoord) -> GeoBBox {
let n = 2f64.powi(coord.z as i32);
let west = coord.x as f64 / n * 360.0 - 180.0;
let east = (coord.x + 1) as f64 / n * 360.0 - 180.0;
let north = (std::f64::consts::PI * (1.0 - 2.0 * coord.y as f64 / n)).sinh().atan().to_degrees();
let south = (std::f64::consts::PI * (1.0 - 2.0 * (coord.y + 1) as f64 / n)).sinh().atan().to_degrees();
GeoBBox { south, west, north, east }
}
/// Get all tile coordinates covering a bounding box at a zoom level.
pub fn tiles_for_bbox(bbox: &GeoBBox, zoom: u8) -> Vec<TileCoord> {
let tl = wgs84_to_tile(bbox.north, bbox.west, zoom);
let br = wgs84_to_tile(bbox.south, bbox.east, zoom);
let mut tiles = Vec::new();
for y in tl.y..=br.y {
for x in tl.x..=br.x {
tiles.push(TileCoord { z: zoom, x, y });
}
}
tiles
}

View File

@ -0,0 +1,72 @@
//! Multi-source fusion — satellite + terrain + OSM + local sensor data.
use crate::cache::TileCache;
use crate::types::*;
use crate::{locate, osm, terrain, tiles};
use anyhow::Result;
/// Build a complete geo scene for a location.
pub async fn build_scene(radius_m: f64) -> Result<GeoScene> {
let cache = TileCache::default_cache();
// 1. Locate
let cache_path = cache.base_dir.join("location.json");
let location = locate::get_location(cache_path.to_str().unwrap_or("")).await?;
eprintln!(" Geo: located at {:.4}N, {:.4}W", location.lat, location.lon);
// 2. Fetch satellite tiles
let bbox = GeoBBox::from_center(&location, radius_m);
let tile_list = tiles::fetch_area(&tiles::TileProvider::Sentinel2Cloudless, &bbox, 16, &cache).await?;
eprintln!(" Geo: fetched {} satellite tiles", tile_list.len());
// 3. Fetch elevation
let dem = terrain::fetch_elevation(&location, &cache).await?;
let elevation = terrain::elevation_at(&dem, &location);
eprintln!(" Geo: elevation {:.0}m ASL", elevation);
// 4. Fetch OSM buildings + roads
let buildings = osm::fetch_buildings(&location, radius_m).await.unwrap_or_default();
let roads = osm::fetch_roads(&location, radius_m).await.unwrap_or_default();
eprintln!(" Geo: {} buildings, {} roads", buildings.len(), roads.len());
// 5. Build registration
let mut reg_origin = location.clone();
reg_origin.alt = elevation as f64;
let registration = crate::register::auto_register(&reg_origin);
Ok(GeoScene {
location: reg_origin,
bbox,
elevation_m: elevation,
buildings,
roads,
tile_count: tile_list.len(),
registration,
last_updated: chrono::Utc::now().to_rfc3339(),
})
}
/// Generate a text summary of the geo scene.
pub fn summarize(scene: &GeoScene) -> String {
let building_count = scene.buildings.len();
let road_count = scene.roads.len();
let road_names: Vec<&str> = scene.roads.iter()
.filter_map(|r| match r {
OsmFeature::Road { name, .. } => name.as_deref(),
_ => None,
})
.take(3)
.collect();
format!(
"Location: {:.4}N, {:.4}W, elevation {:.0}m ASL. \
{} buildings within view. {} roads nearby{}. \
{} satellite tiles at zoom 16. Updated: {}.",
scene.location.lat, scene.location.lon, scene.elevation_m,
building_count, road_count,
if road_names.is_empty() { String::new() }
else { format!(" ({})", road_names.join(", ")) },
scene.tile_count,
&scene.last_updated[..10],
)
}

View File

@ -0,0 +1,19 @@
//! wifi-densepose-geo — geospatial satellite integration for RuView.
//!
//! Provides: IP geolocation, satellite tile fetching (Sentinel-2),
//! SRTM elevation, OSM buildings/roads, coordinate transforms,
//! temporal change tracking, and brain memory integration.
pub mod types;
pub mod coord;
pub mod locate;
pub mod cache;
pub mod tiles;
pub mod terrain;
pub mod osm;
pub mod register;
pub mod fuse;
pub mod brain;
pub mod temporal;
pub use types::*;

View File

@ -0,0 +1,40 @@
//! IP geolocation — determine location from public IP.
use crate::types::GeoPoint;
use anyhow::Result;
/// Locate by IP address (free, no API key).
pub async fn locate_by_ip() -> Result<GeoPoint> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.build()?;
// Primary: ip-api.com (free, 45 req/min)
let resp: serde_json::Value = client
.get("http://ip-api.com/json/?fields=lat,lon,city,regionName,country")
.send().await?
.json().await?;
let lat = resp.get("lat").and_then(|v| v.as_f64()).unwrap_or(0.0);
let lon = resp.get("lon").and_then(|v| v.as_f64()).unwrap_or(0.0);
if lat == 0.0 && lon == 0.0 {
anyhow::bail!("IP geolocation returned (0,0)");
}
Ok(GeoPoint { lat, lon, alt: 0.0 })
}
/// Get location with caching.
pub async fn get_location(cache_path: &str) -> Result<GeoPoint> {
// Check cache
if let Ok(data) = std::fs::read_to_string(cache_path) {
if let Ok(point) = serde_json::from_str::<GeoPoint>(&data) {
return Ok(point);
}
}
let point = locate_by_ip().await?;
let _ = std::fs::write(cache_path, serde_json::to_string(&point)?);
Ok(point)
}

View File

@ -0,0 +1,136 @@
//! OpenStreetMap data via Overpass API — buildings, roads, land use.
use crate::types::{GeoBBox, GeoPoint, OsmFeature};
use anyhow::Result;
const OVERPASS_URL: &str = "https://overpass-api.de/api/interpreter";
/// Fetch buildings within radius of a point.
///
/// Uses an inclusive `["building"]` filter that matches all building values
/// (residential, commercial, yes, etc.) and also queries relations for
/// multipolygon buildings. Default recommended radius: 500 m.
pub async fn fetch_buildings(center: &GeoPoint, radius_m: f64) -> Result<Vec<OsmFeature>> {
let bbox = GeoBBox::from_center(center, radius_m);
let query = format!(
r#"[out:json][timeout:25];(way["building"]({},{},{},{});relation["building"]({},{},{},{}););out body;>;out skel qt;"#,
bbox.south, bbox.west, bbox.north, bbox.east,
bbox.south, bbox.west, bbox.north, bbox.east,
);
let resp = overpass_query(&query).await?;
parse_buildings(&resp)
}
/// Fetch roads within radius.
pub async fn fetch_roads(center: &GeoPoint, radius_m: f64) -> Result<Vec<OsmFeature>> {
let bbox = GeoBBox::from_center(center, radius_m);
let query = format!(
r#"[out:json][timeout:10];way["highway"]({},{},{},{});out body;>;out skel qt;"#,
bbox.south, bbox.west, bbox.north, bbox.east
);
let resp = overpass_query(&query).await?;
parse_roads(&resp)
}
async fn overpass_query(query: &str) -> Result<serde_json::Value> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(15))
.user_agent("RuView/0.1")
.build()?;
let resp = client.post(OVERPASS_URL)
.form(&[("data", query)])
.send().await?;
if !resp.status().is_success() {
anyhow::bail!("Overpass API error: {}", resp.status());
}
Ok(resp.json().await?)
}
fn parse_buildings(data: &serde_json::Value) -> Result<Vec<OsmFeature>> {
let mut buildings = Vec::new();
let mut nodes: std::collections::HashMap<u64, [f64; 2]> = std::collections::HashMap::new();
let elements = data.get("elements").and_then(|e| e.as_array()).cloned().unwrap_or_default();
// First pass: collect nodes
for el in &elements {
if el.get("type").and_then(|t| t.as_str()) == Some("node") {
if let (Some(id), Some(lat), Some(lon)) = (
el.get("id").and_then(|v| v.as_u64()),
el.get("lat").and_then(|v| v.as_f64()),
el.get("lon").and_then(|v| v.as_f64()),
) {
nodes.insert(id, [lat, lon]);
}
}
}
// Second pass: build ways
for el in &elements {
if el.get("type").and_then(|t| t.as_str()) != Some("way") { continue; }
let tags = el.get("tags").cloned().unwrap_or(serde_json::json!({}));
if tags.get("building").is_none() { continue; }
let node_ids = el.get("nodes").and_then(|n| n.as_array()).cloned().unwrap_or_default();
let outline: Vec<[f64; 2]> = node_ids.iter()
.filter_map(|id| id.as_u64().and_then(|id| nodes.get(&id).copied()))
.collect();
if outline.len() < 3 { continue; }
let height = tags.get("height").and_then(|h| h.as_str())
.and_then(|s| s.trim_end_matches('m').trim().parse::<f32>().ok())
.or(Some(8.0)); // default building height
let name = tags.get("name").and_then(|n| n.as_str()).map(|s| s.to_string());
buildings.push(OsmFeature::Building { outline, height, name });
}
Ok(buildings)
}
fn parse_roads(data: &serde_json::Value) -> Result<Vec<OsmFeature>> {
let mut roads = Vec::new();
let mut nodes: std::collections::HashMap<u64, [f64; 2]> = std::collections::HashMap::new();
let elements = data.get("elements").and_then(|e| e.as_array()).cloned().unwrap_or_default();
for el in &elements {
if el.get("type").and_then(|t| t.as_str()) == Some("node") {
if let (Some(id), Some(lat), Some(lon)) = (
el.get("id").and_then(|v| v.as_u64()),
el.get("lat").and_then(|v| v.as_f64()),
el.get("lon").and_then(|v| v.as_f64()),
) {
nodes.insert(id, [lat, lon]);
}
}
}
for el in &elements {
if el.get("type").and_then(|t| t.as_str()) != Some("way") { continue; }
let tags = el.get("tags").cloned().unwrap_or(serde_json::json!({}));
let highway = tags.get("highway").and_then(|h| h.as_str());
if highway.is_none() { continue; }
let node_ids = el.get("nodes").and_then(|n| n.as_array()).cloned().unwrap_or_default();
let path: Vec<[f64; 2]> = node_ids.iter()
.filter_map(|id| id.as_u64().and_then(|id| nodes.get(&id).copied()))
.collect();
if path.len() < 2 { continue; }
let name = tags.get("name").and_then(|n| n.as_str()).map(|s| s.to_string());
roads.push(OsmFeature::Road {
path,
road_type: highway.unwrap_or("unknown").to_string(),
name,
});
}
Ok(roads)
}

View File

@ -0,0 +1,41 @@
//! Geo-registration — maps local sensor coordinates to WGS84.
use crate::coord;
use crate::types::{GeoPoint, GeoRegistration};
/// Auto-register using IP location (sensor at IP location, facing north).
pub fn auto_register(ip_location: &GeoPoint) -> GeoRegistration {
GeoRegistration {
origin: ip_location.clone(),
heading_deg: 0.0,
scale: 1.0,
}
}
/// Transform local point [x, y, z] to WGS84.
pub fn local_to_wgs84(reg: &GeoRegistration, local: &[f32; 3]) -> GeoPoint {
let heading_rad = reg.heading_deg.to_radians();
let cos_h = heading_rad.cos();
let sin_h = heading_rad.sin();
// Rotate local by heading (local X → East when heading=0)
let east = (local[0] as f64 * cos_h - local[2] as f64 * sin_h) * reg.scale;
let north = (local[0] as f64 * sin_h + local[2] as f64 * cos_h) * reg.scale;
let up = local[1] as f64 * reg.scale;
coord::enu_to_wgs84(&[east, north, up], &reg.origin)
}
/// Transform WGS84 to local point.
pub fn wgs84_to_local(reg: &GeoRegistration, geo: &GeoPoint) -> [f32; 3] {
let enu = coord::wgs84_to_enu(geo, &reg.origin);
let heading_rad = (-reg.heading_deg).to_radians();
let cos_h = heading_rad.cos();
let sin_h = heading_rad.sin();
let x = ((enu[0] * cos_h - enu[1] * sin_h) / reg.scale) as f32;
let z = ((enu[0] * sin_h + enu[1] * cos_h) / reg.scale) as f32;
let y = (enu[2] / reg.scale) as f32;
[x, y, z]
}

View File

@ -0,0 +1,310 @@
//! Temporal change tracking — detect changes in satellite/OSM/weather over time.
use crate::cache::TileCache;
use crate::types::GeoPoint;
#[allow(unused_imports)]
use crate::types::GeoScene;
use anyhow::Result;
/// Fetch current weather (Open Meteo, free, no key).
pub async fn fetch_weather(point: &GeoPoint) -> Result<WeatherData> {
let url = format!(
"https://api.open-meteo.com/v1/forecast?latitude={:.4}&longitude={:.4}&current=temperature_2m,relative_humidity_2m,wind_speed_10m,weather_code",
point.lat, point.lon
);
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()?;
let resp: serde_json::Value = client.get(&url).send().await?.json().await?;
let current = resp.get("current").cloned().unwrap_or(serde_json::json!({}));
Ok(WeatherData {
temperature_c: current.get("temperature_2m").and_then(|v| v.as_f64()).unwrap_or(0.0) as f32,
humidity_pct: current.get("relative_humidity_2m").and_then(|v| v.as_f64()).unwrap_or(0.0) as f32,
wind_speed_ms: current.get("wind_speed_10m").and_then(|v| v.as_f64()).unwrap_or(0.0) as f32,
weather_code: current.get("weather_code").and_then(|v| v.as_u64()).unwrap_or(0) as u16,
})
}
/// Check for OSM changes since last fetch.
pub async fn check_osm_changes(scene: &GeoScene, cache: &TileCache) -> Result<Vec<String>> {
let mut changes = Vec::new();
let cache_key = "osm_building_count";
let prev_count: usize = cache.get(cache_key)
.and_then(|d| String::from_utf8(d).ok())
.and_then(|s| s.trim().parse().ok())
.unwrap_or(0);
let current_count = scene.buildings.len();
if prev_count > 0 && current_count != prev_count {
let diff = current_count as i64 - prev_count as i64;
changes.push(format!("Building count changed: {}{} ({:+})", prev_count, current_count, diff));
}
cache.put(cache_key, current_count.to_string().as_bytes())?;
Ok(changes)
}
/// Generate temporal summary for brain storage.
pub fn temporal_summary(weather: &WeatherData, changes: &[String]) -> String {
let weather_desc = match weather.weather_code {
0 => "clear sky",
1..=3 => "partly cloudy",
45 | 48 => "foggy",
51..=57 => "drizzle",
61..=67 => "rain",
71..=77 => "snow",
80..=82 => "showers",
95..=99 => "thunderstorm",
_ => "unknown",
};
let mut summary = format!(
"Weather: {:.0}°C, {weather_desc}, humidity {:.0}%, wind {:.1}m/s.",
weather.temperature_c, weather.humidity_pct, weather.wind_speed_ms,
);
for change in changes {
summary.push_str(&format!(" Change: {change}."));
}
summary
}
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct WeatherData {
pub temperature_c: f32,
pub humidity_pct: f32,
pub wind_speed_ms: f32,
pub weather_code: u16,
}
// ---------------------------------------------------------------------------
// Satellite tile change detection
// ---------------------------------------------------------------------------
/// Result of comparing two tile snapshots.
#[derive(Clone, Debug, serde::Serialize, serde::Deserialize)]
pub struct TileChangeResult {
/// 0.0 = identical, 1.0 = completely different.
pub diff_score: f64,
/// Number of pixels that changed.
pub changed_pixels: usize,
/// Total pixels compared.
pub total_pixels: usize,
}
/// Compare a newly-fetched tile against its previously-cached version.
///
/// Returns a `TileChangeResult` with a diff score between 0.0 (identical) and
/// 1.0 (completely different). When the diff exceeds 0.1 the function stores
/// a change event as a brain memory via the local ruOS brain endpoint.
pub async fn detect_tile_changes(
cache_key: &str,
new_data: &[u8],
cache: &TileCache,
) -> Result<TileChangeResult> {
let previous = cache.get(cache_key);
let result = match previous {
Some(ref old_data) => {
let total = old_data.len().max(new_data.len()).max(1);
let comparable = old_data.len().min(new_data.len());
let mut changed: usize = 0;
for i in 0..comparable {
if old_data[i] != new_data[i] {
changed += 1;
}
}
// Any extra bytes in the longer slice count as changed.
changed += total - comparable;
TileChangeResult {
diff_score: changed as f64 / total as f64,
changed_pixels: changed,
total_pixels: total,
}
}
None => {
// No previous data — treat as fully new (score 1.0).
TileChangeResult {
diff_score: 1.0,
changed_pixels: new_data.len(),
total_pixels: new_data.len().max(1),
}
}
};
// Persist new snapshot into cache for future comparisons.
cache.put(cache_key, new_data)?;
// When significant change is detected, store a brain memory.
if result.diff_score > 0.1 {
let _ = store_change_event(cache_key, &result).await;
}
Ok(result)
}
/// Post a change event to the local ruOS brain.
async fn store_change_event(cache_key: &str, result: &TileChangeResult) -> Result<()> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.build()?;
let body = serde_json::json!({
"category": "spatial-change",
"content": format!(
"Tile change detected for {cache_key}: diff={:.3}, changed={}/{}",
result.diff_score, result.changed_pixels, result.total_pixels,
),
});
client
.post("http://127.0.0.1:9876/memories")
.json(&body)
.send()
.await?;
Ok(())
}
// ---------------------------------------------------------------------------
// Night mode detection
// ---------------------------------------------------------------------------
/// Approximate check whether the current time is "night" at a given latitude.
///
/// Uses a simplified sunrise/sunset model based on the solar declination and
/// hour angle. When it is night the system should rely on CSI data only
/// (satellite imagery is not useful in darkness).
pub fn is_night(lat_deg: f64) -> bool {
let now = chrono::Utc::now();
is_night_at(lat_deg, now)
}
/// Testable version of [`is_night`] that accepts an explicit timestamp.
pub fn is_night_at(lat_deg: f64, utc: chrono::DateTime<chrono::Utc>) -> bool {
use chrono::Datelike;
use std::f64::consts::PI;
let day_of_year = utc.ordinal() as f64;
let hour_utc = utc.timestamp() % 86400;
let solar_hour = (hour_utc as f64) / 3600.0; // 0..24
// Solar declination (Spencer, 1971 — simplified)
let gamma = 2.0 * PI * (day_of_year - 1.0) / 365.0;
let decl = 0.006918
- 0.399912 * gamma.cos()
+ 0.070257 * gamma.sin()
- 0.006758 * (2.0 * gamma).cos()
+ 0.000907 * (2.0 * gamma).sin();
let lat_rad = lat_deg.to_radians();
// Cosine of the hour angle at sunrise/sunset (geometric, no refraction)
let cos_ha = -(lat_rad.tan() * decl.tan());
// Polar day / polar night
if cos_ha < -1.0 {
return false; // midnight sun — never night
}
if cos_ha > 1.0 {
return true; // polar night — always night
}
let ha_sunrise = cos_ha.acos(); // radians, symmetric about solar noon
let daylight_hours = 2.0 * ha_sunrise * 12.0 / PI;
let solar_noon = 12.0; // approximation (ignores longitude offset)
let sunrise = solar_noon - daylight_hours / 2.0;
let sunset = solar_noon + daylight_hours / 2.0;
solar_hour < sunrise || solar_hour > sunset
}
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_is_night_at_equator_noon() {
// Noon UTC at equator on March 20 — should be daytime.
let dt = chrono::NaiveDate::from_ymd_opt(2025, 3, 20)
.unwrap()
.and_hms_opt(12, 0, 0)
.unwrap()
.and_utc();
assert!(!is_night_at(0.0, dt));
}
#[test]
fn test_is_night_at_equator_midnight() {
// Midnight UTC at equator — should be night.
let dt = chrono::NaiveDate::from_ymd_opt(2025, 3, 20)
.unwrap()
.and_hms_opt(2, 0, 0)
.unwrap()
.and_utc();
assert!(is_night_at(0.0, dt));
}
#[test]
fn test_midnight_sun_arctic() {
// Late June at 70 N — midnight sun, never night.
let dt = chrono::NaiveDate::from_ymd_opt(2025, 6, 21)
.unwrap()
.and_hms_opt(0, 0, 0)
.unwrap()
.and_utc();
assert!(!is_night_at(70.0, dt));
}
#[test]
fn test_polar_night_arctic() {
// Late December at 80 N — polar night, always night.
let dt = chrono::NaiveDate::from_ymd_opt(2025, 12, 21)
.unwrap()
.and_hms_opt(12, 0, 0)
.unwrap()
.and_utc();
assert!(is_night_at(80.0, dt));
}
#[test]
fn test_detect_tile_changes_identical() {
let cache = TileCache::new("/tmp/ruview-test-tile-changes");
let data = vec![1u8, 2, 3, 4, 5];
// Prime the cache.
cache.put("test_tile_ident", &data).unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let result = rt.block_on(detect_tile_changes("test_tile_ident", &data, &cache)).unwrap();
assert!((result.diff_score - 0.0).abs() < 1e-9);
assert_eq!(result.changed_pixels, 0);
}
#[test]
fn test_detect_tile_changes_fully_different() {
let cache = TileCache::new("/tmp/ruview-test-tile-changes");
let old = vec![0u8; 100];
let new = vec![255u8; 100];
cache.put("test_tile_diff", &old).unwrap();
let rt = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.unwrap();
let result = rt.block_on(detect_tile_changes("test_tile_diff", &new, &cache)).unwrap();
assert!((result.diff_score - 1.0).abs() < 1e-9);
}
}

View File

@ -0,0 +1,110 @@
//! SRTM DEM parser — elevation data from NASA 1-arcsecond HGT files.
use crate::cache::TileCache;
use crate::types::{ElevationGrid, GeoPoint};
use anyhow::Result;
/// Download and parse SRTM HGT for a location.
pub async fn fetch_elevation(point: &GeoPoint, cache: &TileCache) -> Result<ElevationGrid> {
let lat_int = point.lat.floor() as i32;
let lon_int = point.lon.floor() as i32;
let ns = if lat_int >= 0 { 'N' } else { 'S' };
let ew = if lon_int >= 0 { 'E' } else { 'W' };
let filename = format!("{}{:02}{}{:03}.hgt", ns, lat_int.unsigned_abs(), ew, lon_int.unsigned_abs());
let cache_key = format!("srtm_{filename}");
if let Some(data) = cache.get(&cache_key) {
return parse_hgt(&data, lat_int as f64, lon_int as f64);
}
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.build()?;
// Primary: NASA SRTM public mirror (no auth required for .hgt)
let nasa_url = format!(
"https://e4ftl01.cr.usgs.gov/MEASURES/SRTMGL1.003/2000.02.11/{filename}"
);
if let Ok(resp) = client.get(&nasa_url).send().await {
if resp.status().is_success() {
let data = resp.bytes().await?.to_vec();
cache.put(&cache_key, &data)?;
return parse_hgt(&data, lat_int as f64, lon_int as f64);
}
}
// Fallback: viewfinderpanoramas.org
// Files are grouped by continent zip, but individual .hgt files can be
// fetched directly when the server exposes them.
let vfp_url = format!(
"http://viewfinderpanoramas.org/dem1/{filename}"
);
if let Ok(resp) = client.get(&vfp_url).send().await {
if resp.status().is_success() {
let data = resp.bytes().await?.to_vec();
cache.put(&cache_key, &data)?;
return parse_hgt(&data, lat_int as f64, lon_int as f64);
}
}
// Final fallback: flat terrain when all downloads fail
Ok(ElevationGrid {
origin_lat: lat_int as f64,
origin_lon: lon_int as f64,
cell_size_deg: 1.0 / 3600.0,
cols: 100, rows: 100,
heights: vec![0.0; 10000],
})
}
/// Parse SRTM HGT binary (3601x3601 big-endian i16).
pub fn parse_hgt(data: &[u8], origin_lat: f64, origin_lon: f64) -> Result<ElevationGrid> {
let n_samples = data.len() / 2;
let side = (n_samples as f64).sqrt() as usize;
let heights: Vec<f32> = data.chunks_exact(2)
.map(|c| {
let v = i16::from_be_bytes([c[0], c[1]]);
if v == -32768 { 0.0 } else { v as f32 } // -32768 = void
})
.collect();
Ok(ElevationGrid {
origin_lat, origin_lon,
cell_size_deg: 1.0 / (side - 1) as f64,
cols: side, rows: side,
heights,
})
}
/// Get elevation at a specific point from a grid.
pub fn elevation_at(grid: &ElevationGrid, point: &GeoPoint) -> f32 {
grid.get(point.lat, point.lon).unwrap_or(0.0)
}
/// Extract a small subgrid around a point.
pub fn extract_subgrid(grid: &ElevationGrid, center: &GeoPoint, radius_m: f64) -> ElevationGrid {
let radius_deg = radius_m / 111_320.0;
let min_row = ((grid.origin_lat + (grid.rows as f64 * grid.cell_size_deg) - center.lat - radius_deg) / grid.cell_size_deg).max(0.0) as usize;
let max_row = ((grid.origin_lat + (grid.rows as f64 * grid.cell_size_deg) - center.lat + radius_deg) / grid.cell_size_deg).min(grid.rows as f64) as usize;
let min_col = ((center.lon - radius_deg - grid.origin_lon) / grid.cell_size_deg).max(0.0) as usize;
let max_col = ((center.lon + radius_deg - grid.origin_lon) / grid.cell_size_deg).min(grid.cols as f64) as usize;
let rows = max_row.saturating_sub(min_row);
let cols = max_col.saturating_sub(min_col);
let mut heights = Vec::with_capacity(rows * cols);
for r in min_row..max_row {
for c in min_col..max_col {
heights.push(grid.heights.get(r * grid.cols + c).copied().unwrap_or(0.0));
}
}
ElevationGrid {
origin_lat: grid.origin_lat + (grid.rows - max_row) as f64 * grid.cell_size_deg,
origin_lon: grid.origin_lon + min_col as f64 * grid.cell_size_deg,
cell_size_deg: grid.cell_size_deg,
cols, rows, heights,
}
}

View File

@ -0,0 +1,80 @@
//! Satellite tile fetcher — XYZ/TMS tile download with caching.
use crate::cache::TileCache;
use crate::coord;
use crate::types::{GeoBBox, RasterTile, TileCoord};
use anyhow::Result;
/// Tile provider (all free, no API keys).
pub enum TileProvider {
/// Sentinel-2 cloudless mosaic (EOX, 10m, CC-BY-4.0)
Sentinel2Cloudless,
/// ESRI World Imagery (sub-meter, free tier)
EsriWorldImagery,
/// OpenStreetMap (map tiles, not satellite)
Osm,
}
impl TileProvider {
pub fn url(&self, coord: &TileCoord) -> String {
match self {
Self::Sentinel2Cloudless => format!(
"https://tiles.maps.eox.at/wmts/1.0.0/s2cloudless-2021_3857/default/g/{}/{}/{}.jpg",
coord.z, coord.y, coord.x
),
Self::EsriWorldImagery => format!(
"https://server.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/{}/{}/{}",
coord.z, coord.y, coord.x
),
Self::Osm => format!(
"https://tile.openstreetmap.org/{}/{}/{}.png",
coord.z, coord.x, coord.y
),
}
}
pub fn name(&self) -> &str {
match self {
Self::Sentinel2Cloudless => "sentinel2",
Self::EsriWorldImagery => "esri",
Self::Osm => "osm",
}
}
}
/// Fetch a single tile with caching.
pub async fn fetch_tile(provider: &TileProvider, coord: &TileCoord, cache: &TileCache) -> Result<RasterTile> {
let cache_key = format!("tiles_{}_{}_{}.dat", coord.z, coord.x, coord.y);
if let Some(data) = cache.get(&cache_key) {
return Ok(RasterTile { coord: coord.clone(), data, bounds: coord::tile_bounds(coord) });
}
let url = provider.url(coord);
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
.user_agent("RuView/0.1 (https://github.com/ruvnet/RuView)")
.build()?;
let resp = client.get(&url).send().await?;
if !resp.status().is_success() {
anyhow::bail!("Tile fetch failed: {} → {}", url, resp.status());
}
let data = resp.bytes().await?.to_vec();
cache.put(&cache_key, &data)?;
Ok(RasterTile { coord: coord.clone(), data, bounds: coord::tile_bounds(coord) })
}
/// Fetch all tiles covering a bounding box.
pub async fn fetch_area(provider: &TileProvider, bbox: &GeoBBox, zoom: u8, cache: &TileCache) -> Result<Vec<RasterTile>> {
let coords = coord::tiles_for_bbox(bbox, zoom);
let mut tiles = Vec::with_capacity(coords.len());
for c in &coords {
match fetch_tile(provider, c, cache).await {
Ok(t) => tiles.push(t),
Err(e) => eprintln!(" Tile {}/{}/{} failed: {}", c.z, c.x, c.y, e),
}
}
Ok(tiles)
}

View File

@ -0,0 +1,118 @@
//! Core geospatial types.
use serde::{Deserialize, Serialize};
/// WGS84 geographic coordinate.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct GeoPoint {
pub lat: f64,
pub lon: f64,
pub alt: f64,
}
/// Axis-aligned bounding box in WGS84.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct GeoBBox {
pub south: f64,
pub west: f64,
pub north: f64,
pub east: f64,
}
impl GeoBBox {
pub fn from_center(center: &GeoPoint, radius_m: f64) -> Self {
let dlat = radius_m / 111_320.0;
let dlon = radius_m / (111_320.0 * center.lat.to_radians().cos());
Self {
south: center.lat - dlat,
west: center.lon - dlon,
north: center.lat + dlat,
east: center.lon + dlon,
}
}
}
/// XYZ tile address.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct TileCoord {
pub z: u8,
pub x: u32,
pub y: u32,
}
/// Satellite raster tile.
#[derive(Clone, Debug)]
pub struct RasterTile {
pub coord: TileCoord,
pub data: Vec<u8>,
pub bounds: GeoBBox,
}
/// Elevation grid from SRTM DEM.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct ElevationGrid {
pub origin_lat: f64,
pub origin_lon: f64,
pub cell_size_deg: f64,
pub cols: usize,
pub rows: usize,
pub heights: Vec<f32>,
}
impl ElevationGrid {
pub fn get(&self, lat: f64, lon: f64) -> Option<f32> {
let row = ((self.origin_lat + (self.rows as f64 * self.cell_size_deg) - lat) / self.cell_size_deg) as usize;
let col = ((lon - self.origin_lon) / self.cell_size_deg) as usize;
if row < self.rows && col < self.cols {
Some(self.heights[row * self.cols + col])
} else {
None
}
}
}
/// OpenStreetMap feature.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub enum OsmFeature {
Building {
outline: Vec<[f64; 2]>,
height: Option<f32>,
name: Option<String>,
},
Road {
path: Vec<[f64; 2]>,
road_type: String,
name: Option<String>,
},
}
/// Geo-registration transform.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct GeoRegistration {
pub origin: GeoPoint,
pub heading_deg: f64,
pub scale: f64,
}
impl Default for GeoRegistration {
fn default() -> Self {
Self {
origin: GeoPoint { lat: 0.0, lon: 0.0, alt: 0.0 },
heading_deg: 0.0,
scale: 1.0,
}
}
}
/// Complete geo scene.
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct GeoScene {
pub location: GeoPoint,
pub bbox: GeoBBox,
pub elevation_m: f32,
pub buildings: Vec<OsmFeature>,
pub roads: Vec<OsmFeature>,
pub tile_count: usize,
pub registration: GeoRegistration,
pub last_updated: String,
}

View File

@ -0,0 +1,84 @@
use ruview_geo::*;
use ruview_geo::coord;
#[test]
fn test_haversine() {
let toronto = GeoPoint { lat: 43.6532, lon: -79.3832, alt: 0.0 };
let ottawa = GeoPoint { lat: 45.4215, lon: -75.6972, alt: 0.0 };
let dist = coord::haversine(&toronto, &ottawa);
assert!((dist - 353_000.0).abs() < 5_000.0, "Toronto-Ottawa ~353km, got {:.0}m", dist);
}
#[test]
fn test_wgs84_to_enu() {
let origin = GeoPoint { lat: 43.0, lon: -79.0, alt: 100.0 };
let point = GeoPoint { lat: 43.001, lon: -79.0, alt: 100.0 };
let enu = coord::wgs84_to_enu(&point, &origin);
assert!((enu[1] - 111.0).abs() < 5.0, "0.001 deg lat ~111m north, got {:.1}m", enu[1]);
assert!(enu[0].abs() < 1.0, "same longitude should have ~0 east, got {:.1}m", enu[0]);
}
#[test]
fn test_enu_roundtrip() {
let origin = GeoPoint { lat: 43.6532, lon: -79.3832, alt: 76.0 };
let local = [100.0, 200.0, 5.0]; // 100m east, 200m north, 5m up
let geo = coord::enu_to_wgs84(&local, &origin);
let back = coord::wgs84_to_enu(&geo, &origin);
assert!((back[0] - local[0]).abs() < 0.01);
assert!((back[1] - local[1]).abs() < 0.01);
assert!((back[2] - local[2]).abs() < 0.01);
}
#[test]
fn test_tile_coords() {
let tile = coord::wgs84_to_tile(43.6532, -79.3832, 16);
assert!(tile.x > 0 && tile.y > 0);
assert_eq!(tile.z, 16);
let bounds = coord::tile_bounds(&tile);
assert!(bounds.south < 43.66 && bounds.north > 43.64);
}
#[test]
fn test_tiles_for_bbox() {
let bbox = GeoBBox::from_center(
&GeoPoint { lat: 43.6532, lon: -79.3832, alt: 0.0 },
500.0,
);
let tiles = coord::tiles_for_bbox(&bbox, 16);
assert!(tiles.len() >= 4 && tiles.len() <= 25, "500m radius should need 4-25 tiles, got {}", tiles.len());
}
#[test]
fn test_geo_bbox_from_center() {
let center = GeoPoint { lat: 43.0, lon: -79.0, alt: 0.0 };
let bbox = GeoBBox::from_center(&center, 1000.0);
assert!(bbox.south < 43.0 && bbox.north > 43.0);
assert!(bbox.west < -79.0 && bbox.east > -79.0);
}
#[test]
fn test_hgt_parse() {
// Create minimal 3x3 HGT data (big-endian i16)
let mut data = Vec::new();
for h in [100i16, 110, 120, 105, 115, 125, 110, 120, 130] {
data.extend_from_slice(&h.to_be_bytes());
}
let grid = ruview_geo::terrain::parse_hgt(&data, 43.0, -79.0).unwrap();
assert_eq!(grid.heights[0], 100.0);
assert_eq!(grid.heights[4], 115.0);
}
#[test]
fn test_registration() {
let origin = GeoPoint { lat: 43.6532, lon: -79.3832, alt: 76.0 };
let reg = ruview_geo::register::auto_register(&origin);
let local = [10.0f32, 0.0, 20.0]; // 10m east, 20m forward
let geo = ruview_geo::register::local_to_wgs84(&reg, &local);
assert!((geo.lat - origin.lat).abs() < 0.001);
assert!((geo.lon - origin.lon).abs() < 0.001);
let back = ruview_geo::register::wgs84_to_local(&reg, &geo);
assert!((back[0] - local[0]).abs() < 0.1);
assert!((back[2] - local[2]).abs() < 0.1);
}

View File

@ -0,0 +1,90 @@
//! Brain bridge — sends spatial observations to the ruOS brain.
//!
//! Periodically summarizes the sensor pipeline state and stores it
//! as brain memories for the agent to reason about.
use crate::csi_pipeline::PipelineOutput;
use anyhow::Result;
const BRAIN_URL: &str = "http://127.0.0.1:9876";
/// Store a spatial observation in the brain.
async fn store_memory(category: &str, content: &str) -> Result<()> {
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.build()?;
let body = serde_json::json!({
"category": category,
"content": content,
});
client.post(format!("{BRAIN_URL}/memories"))
.json(&body)
.send()
.await?;
Ok(())
}
/// Summarize pipeline state and store in brain (called every 60 seconds).
pub async fn sync_to_brain(pipeline: &PipelineOutput, camera_frames: u64) {
// Only store if there's meaningful data
if pipeline.total_frames < 10 && camera_frames < 5 { return; }
// Store spatial summary
let motion_str = if pipeline.motion_detected { "detected" } else { "absent" };
let skeleton_str = if let Some(ref sk) = pipeline.skeleton {
format!("{} keypoints ({:.0}% conf)", sk.keypoints.len(), sk.confidence * 100.0)
} else {
"inactive".to_string()
};
let summary = format!(
"Room scan: {} camera frames, {} CSI frames from {} nodes. \
Motion {} ({:.0}%). Breathing {:.0} BPM. Skeleton: {}. \
Occupancy grid {}x{}x{} with {} occupied voxels.",
camera_frames,
pipeline.total_frames,
pipeline.num_nodes,
motion_str,
pipeline.vitals.motion_score * 100.0,
pipeline.vitals.breathing_rate,
skeleton_str,
pipeline.occupancy_dims.0,
pipeline.occupancy_dims.1,
pipeline.occupancy_dims.2,
pipeline.occupancy.iter().filter(|&&d| d > 0.3).count(),
);
let _ = store_memory("spatial-observation", &summary).await;
// Store motion events
if pipeline.motion_detected && pipeline.vitals.motion_score > 0.3 {
let _ = store_memory("spatial-motion",
&format!("Strong motion detected: {:.0}% score, {} CSI frames",
pipeline.vitals.motion_score * 100.0, pipeline.total_frames)
).await;
}
// Store vital signs if available
if pipeline.vitals.breathing_rate > 5.0 && pipeline.vitals.breathing_rate < 35.0 {
let _ = store_memory("spatial-vitals",
&format!("Vital signs: breathing {:.0} BPM, motion {:.0}%",
pipeline.vitals.breathing_rate, pipeline.vitals.motion_score * 100.0)
).await;
}
}
/// Check if brain is reachable.
pub async fn brain_available() -> bool {
reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(2))
.build()
.ok()
.and_then(|c| {
tokio::runtime::Handle::current().block_on(async {
c.get(format!("{BRAIN_URL}/health")).send().await.ok()
})
})
.is_some()
}

View File

@ -0,0 +1,601 @@
//! Complete CSI processing pipeline — ADR-018 parser → WiFlow pose → vitals → tomography.
//!
//! Receives raw UDP frames from ESP32 nodes, extracts I/Q subcarrier data,
//! runs the WiFlow pose model, detects motion, estimates vitals, and produces
//! 3D occupancy + skeleton for fusion with camera depth.
use std::collections::VecDeque;
use std::net::UdpSocket;
use std::sync::{Arc, Mutex};
// ─── ADR-018 Binary Frame Parser ────────────────────────────────────────────
const CSI_MAGIC_V6: u32 = 0xC511_0006;
const CSI_MAGIC_V1: u32 = 0xC511_0001;
const CSI_HEADER_SIZE: usize = 20;
#[derive(Clone, Debug)]
pub struct CsiFrame {
pub node_id: u8,
pub n_antennas: u8,
pub n_subcarriers: u16,
pub channel: u8,
pub rssi: i8,
pub noise_floor: i8,
pub timestamp_us: u32,
/// Raw I/Q data: [I0, Q0, I1, Q1, ...] for each subcarrier
pub iq_data: Vec<i8>,
/// Computed amplitude per subcarrier: sqrt(I^2 + Q^2)
pub amplitudes: Vec<f32>,
/// Computed phase per subcarrier: atan2(Q, I)
pub phases: Vec<f32>,
}
/// Parse an ADR-018 binary CSI frame from UDP packet.
pub fn parse_adr018(data: &[u8]) -> Option<CsiFrame> {
if data.len() < CSI_HEADER_SIZE { return None; }
let magic = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
if magic != CSI_MAGIC_V6 && magic != CSI_MAGIC_V1 { return None; }
let node_id = data[4];
let n_antennas = data[5].max(1);
let n_subcarriers = u16::from_le_bytes([data[6], data[7]]);
let channel = data[8];
let rssi = data[9] as i8;
let noise_floor = data[10] as i8;
let timestamp_us = u32::from_le_bytes([data[16], data[17], data[18], data[19]]);
let iq_len = (n_subcarriers as usize) * 2 * (n_antennas as usize);
if data.len() < CSI_HEADER_SIZE + iq_len { return None; }
let iq_data: Vec<i8> = data[CSI_HEADER_SIZE..CSI_HEADER_SIZE + iq_len]
.iter().map(|&b| b as i8).collect();
// Compute amplitude and phase per subcarrier (first antenna)
let mut amplitudes = Vec::with_capacity(n_subcarriers as usize);
let mut phases = Vec::with_capacity(n_subcarriers as usize);
for i in 0..n_subcarriers as usize {
let idx = i * 2;
if idx + 1 < iq_data.len() {
let ii = iq_data[idx] as f32;
let qq = iq_data[idx + 1] as f32;
amplitudes.push((ii * ii + qq * qq).sqrt());
phases.push(qq.atan2(ii));
}
}
Some(CsiFrame {
node_id, n_antennas, n_subcarriers, channel, rssi, noise_floor,
timestamp_us, iq_data, amplitudes, phases,
})
}
// ─── CSI Fingerprint Database ──────────────────────────────────────────────
#[derive(Clone, Debug, serde::Serialize)]
pub struct CsiFingerprint {
pub name: String,
pub mean_amplitudes: Vec<f32>,
pub rssi_mean: f32,
pub rssi_std: f32,
pub samples: u32,
}
// ─── CSI State — accumulates frames for WiFlow + vitals ─────────────────────
#[derive(Clone, Debug)]
pub struct Skeleton {
/// 17 COCO keypoints: [(x, y), ...] in [0, 1] normalized coordinates
pub keypoints: Vec<[f32; 2]>,
pub confidence: f32,
}
#[derive(Clone, Debug)]
pub struct VitalSigns {
pub breathing_rate: f32, // breaths per minute
pub heart_rate: f32, // beats per minute
pub motion_score: f32, // 0.0 = still, 1.0 = strong motion
}
pub struct CsiPipelineState {
/// Per-node frame history (node_id → last N frames)
pub node_frames: std::collections::HashMap<u8, VecDeque<CsiFrame>>,
/// Latest skeleton from WiFlow
pub skeleton: Option<Skeleton>,
/// Latest vital signs
pub vitals: VitalSigns,
/// Occupancy grid from RF tomography
pub occupancy: Vec<f64>,
pub occupancy_dims: (usize, usize, usize), // nx, ny, nz
/// Total frames received
pub total_frames: u64,
/// Motion detection
pub motion_detected: bool,
/// CSI fingerprint database for room/location identification
pub fingerprints: Vec<CsiFingerprint>,
/// Current identified location (name, confidence) — updated every 100 frames
pub current_location: Option<(String, f32)>,
/// Night mode — true when camera luminance is below threshold
pub is_dark: bool,
/// WiFlow model weights (loaded once)
wiflow_weights: Option<WiFlowModel>,
}
struct WiFlowModel {
/// TCN weights from wiflow-v1.json (simplified inference)
conv1_w: Vec<f32>,
conv1_b: Vec<f32>,
conv2_w: Vec<f32>,
conv2_b: Vec<f32>,
fc_w: Vec<f32>,
fc_b: Vec<f32>,
input_subcarriers: usize,
time_steps: usize,
}
impl Default for CsiPipelineState {
fn default() -> Self {
Self {
node_frames: std::collections::HashMap::new(),
skeleton: None,
vitals: VitalSigns { breathing_rate: 0.0, heart_rate: 0.0, motion_score: 0.0 },
occupancy: vec![0.0; 8 * 8 * 4],
occupancy_dims: (8, 8, 4),
total_frames: 0,
motion_detected: false,
fingerprints: Vec::new(),
current_location: None,
is_dark: false,
wiflow_weights: load_wiflow_model(),
}
}
}
// ─── WiFlow Model Loading ───────────────────────────────────────────────────
fn load_wiflow_model() -> Option<WiFlowModel> {
let paths = [
"/tmp/ruview-firmware/wiflow-v1.json",
"~/.local/share/ruview/wiflow-v1.json",
];
for p in &paths {
let expanded = p.replace('~', &std::env::var("HOME").unwrap_or_default());
if let Ok(data) = std::fs::read_to_string(&expanded) {
if let Ok(model) = serde_json::from_str::<serde_json::Value>(&data) {
if let Some(_weights_b64) = model.get("weightsBase64").and_then(|v| v.as_str()) {
eprintln!(" WiFlow: loaded from {expanded} ({} params)",
model.get("totalParams").and_then(|v| v.as_u64()).unwrap_or(0));
// For now, use simplified inference — full weight parsing would go here
return Some(WiFlowModel {
conv1_w: Vec::new(), conv1_b: Vec::new(),
conv2_w: Vec::new(), conv2_b: Vec::new(),
fc_w: Vec::new(), fc_b: Vec::new(),
input_subcarriers: 35,
time_steps: 20,
});
}
}
}
}
eprintln!(" WiFlow: model not found");
None
}
// ─── Pipeline Processing ────────────────────────────────────────────────────
impl CsiPipelineState {
/// Process a new CSI frame — updates motion, vitals, skeleton, occupancy.
pub fn process_frame(&mut self, frame: CsiFrame) {
let node_id = frame.node_id;
self.total_frames += 1;
// Store frame in per-node history
{
let history = self.node_frames.entry(node_id).or_insert_with(|| VecDeque::with_capacity(100));
history.push_back(frame.clone());
if history.len() > 100 { history.pop_front(); }
}
// 1. Motion detection (amplitude variance over last 20 frames)
self.detect_motion(node_id);
// 2. Vital signs (phase analysis over last 100 frames)
let has_enough = self.node_frames.get(&node_id).map(|h| h.len() >= 30).unwrap_or(false);
if has_enough {
self.estimate_vitals(node_id);
}
// 3. WiFlow pose estimation (every 20 frames = 1 second at ~20fps)
if self.total_frames % 20 == 0 {
self.estimate_pose();
}
// 4. RF tomography (update occupancy grid)
self.update_tomography();
// 5. Location fingerprint identification (every 100 frames)
if self.total_frames % 100 == 0 {
self.current_location = self.identify_location();
}
}
fn detect_motion(&mut self, node_id: u8) {
if let Some(history) = self.node_frames.get(&node_id) {
let recent: Vec<&CsiFrame> = history.iter().rev().take(20).collect();
if recent.len() < 5 { return; }
// Compute mean amplitude across subcarriers for each frame
let mean_amps: Vec<f32> = recent.iter()
.map(|f| f.amplitudes.iter().sum::<f32>() / f.amplitudes.len().max(1) as f32)
.collect();
let mean = mean_amps.iter().sum::<f32>() / mean_amps.len() as f32;
let variance = mean_amps.iter().map(|a| (a - mean).powi(2)).sum::<f32>() / mean_amps.len() as f32;
// High variance = motion
self.vitals.motion_score = (variance / 100.0).min(1.0);
self.motion_detected = self.vitals.motion_score > 0.15;
}
}
fn estimate_vitals(&mut self, node_id: u8) {
if let Some(history) = self.node_frames.get(&node_id) {
let frames: Vec<&CsiFrame> = history.iter().rev().take(100).collect();
if frames.len() < 30 { return; }
// Extract phase from a stable subcarrier (pick one with low variance)
let n_sub = frames[0].phases.len().min(35);
if n_sub == 0 { return; }
// Use subcarrier 15 (mid-band, typically stable)
let sub_idx = n_sub / 2;
let phase_series: Vec<f32> = frames.iter().rev()
.map(|f| f.phases.get(sub_idx).copied().unwrap_or(0.0))
.collect();
// Simple peak counting for breathing rate (0.15-0.5 Hz = 9-30 BPM)
let mut peaks = 0;
for i in 1..phase_series.len() - 1 {
if phase_series[i] > phase_series[i-1] && phase_series[i] > phase_series[i+1] {
peaks += 1;
}
}
// Assuming ~20fps capture, 100 frames = 5 seconds
let capture_secs = frames.len() as f32 / 20.0;
let breathing_bpm = (peaks as f32 / capture_secs) * 60.0;
self.vitals.breathing_rate = breathing_bpm.clamp(5.0, 40.0);
// Heart rate estimation (0.8-2.5 Hz) — need higher sampling rate
// For now, estimate from amplitude modulation
self.vitals.heart_rate = 0.0; // requires FFT for accurate detection
}
}
fn estimate_pose(&mut self) {
if self.wiflow_weights.is_none() { return; }
// Collect 20 frames from the primary node
let primary_node = self.node_frames.keys().next().copied();
if let Some(node_id) = primary_node {
if let Some(history) = self.node_frames.get(&node_id) {
let frames: Vec<&CsiFrame> = history.iter().rev().take(20).collect();
if frames.len() < 20 { return; }
// Build input: 35 subcarriers × 20 time steps
// Select top 35 subcarriers by variance (ruvector-solver O6)
let n_sub = frames[0].amplitudes.len().min(35);
let mut input = vec![0.0f32; 35 * 20];
for (t, frame) in frames.iter().rev().enumerate().take(20) {
for s in 0..n_sub {
input[t * 35 + s] = frame.amplitudes.get(s).copied().unwrap_or(0.0) / 128.0;
}
}
// Simplified WiFlow inference (without full weight loading)
// Generate estimated keypoints based on CSI signal statistics
let mean_amp = input.iter().sum::<f32>() / input.len() as f32;
let amp_var = input.iter().map(|a| (a - mean_amp).powi(2)).sum::<f32>() / input.len() as f32;
// If motion detected, generate pose estimate from signal characteristics
if self.motion_detected {
let mut keypoints = vec![[0.5f32; 2]; 17];
// Distribute keypoints based on CSI energy distribution across subcarriers
for (i, kp) in keypoints.iter_mut().enumerate() {
let sub_range = (i * n_sub / 17)..((i + 1) * n_sub / 17).min(n_sub);
let energy: f32 = sub_range.clone()
.filter_map(|s| frames.last().and_then(|f| f.amplitudes.get(s)))
.sum();
let norm_energy = energy / (sub_range.len().max(1) as f32 * 128.0);
kp[0] = 0.3 + norm_energy * 0.4; // x
kp[1] = (i as f32 / 17.0) * 0.8 + 0.1; // y (head to feet)
}
self.skeleton = Some(Skeleton {
keypoints,
confidence: amp_var.min(1.0),
});
} else {
self.skeleton = None;
}
}
}
}
/// Record a CSI fingerprint for the current location/room.
/// Computes mean amplitude and RSSI statistics from the last 50 frames
/// across all nodes and saves as a named fingerprint.
pub fn record_fingerprint(&mut self, name: &str) {
// Collect last 50 frames from all nodes
let mut all_amplitudes: Vec<Vec<f32>> = Vec::new();
let mut rssi_values: Vec<f32> = Vec::new();
for history in self.node_frames.values() {
for frame in history.iter().rev().take(50) {
all_amplitudes.push(frame.amplitudes.clone());
rssi_values.push(frame.rssi as f32);
}
}
if all_amplitudes.is_empty() {
return;
}
// Compute mean amplitude per subcarrier across all collected frames
let n_sub = all_amplitudes.iter().map(|a| a.len()).max().unwrap_or(0);
if n_sub == 0 {
return;
}
let mut mean_amplitudes = vec![0.0f32; n_sub];
let mut counts = vec![0u32; n_sub];
for amps in &all_amplitudes {
for (i, &a) in amps.iter().enumerate() {
if i < n_sub {
mean_amplitudes[i] += a;
counts[i] += 1;
}
}
}
for i in 0..n_sub {
if counts[i] > 0 {
mean_amplitudes[i] /= counts[i] as f32;
}
}
// RSSI statistics
let rssi_mean = rssi_values.iter().sum::<f32>() / rssi_values.len() as f32;
let rssi_var = rssi_values.iter()
.map(|r| (r - rssi_mean).powi(2))
.sum::<f32>() / rssi_values.len() as f32;
let rssi_std = rssi_var.sqrt();
let fingerprint = CsiFingerprint {
name: name.to_string(),
mean_amplitudes,
rssi_mean,
rssi_std,
samples: all_amplitudes.len() as u32,
};
// Replace existing fingerprint with same name, or append
if let Some(existing) = self.fingerprints.iter_mut().find(|f| f.name == name) {
*existing = fingerprint;
} else {
self.fingerprints.push(fingerprint);
}
}
/// Compare current CSI signals against saved fingerprints using cosine
/// similarity. Returns (name, confidence) if the best match exceeds 0.7.
pub fn identify_location(&self) -> Option<(String, f32)> {
if self.fingerprints.is_empty() {
return None;
}
// Build current mean amplitude vector from last 50 frames
let mut all_amplitudes: Vec<Vec<f32>> = Vec::new();
for history in self.node_frames.values() {
for frame in history.iter().rev().take(50) {
all_amplitudes.push(frame.amplitudes.clone());
}
}
if all_amplitudes.is_empty() {
return None;
}
let n_sub = all_amplitudes.iter().map(|a| a.len()).max().unwrap_or(0);
if n_sub == 0 {
return None;
}
let mut current = vec![0.0f32; n_sub];
let mut counts = vec![0u32; n_sub];
for amps in &all_amplitudes {
for (i, &a) in amps.iter().enumerate() {
if i < n_sub {
current[i] += a;
counts[i] += 1;
}
}
}
for i in 0..n_sub {
if counts[i] > 0 {
current[i] /= counts[i] as f32;
}
}
// Find best matching fingerprint by cosine similarity
let mut best: Option<(String, f32)> = None;
for fp in &self.fingerprints {
let sim = cosine_similarity(&current, &fp.mean_amplitudes);
if sim > 0.7 {
if best.as_ref().map_or(true, |(_, s)| sim > *s) {
best = Some((fp.name.clone(), sim));
}
}
}
best
}
/// Set the ambient light level from camera frame average luminance.
/// When luminance < 30 (out of 255), enables night/dark mode which
/// increases CSI processing frequency and skips camera depth.
pub fn set_light_level(&mut self, avg_luminance: f32) {
self.is_dark = avg_luminance < 30.0;
}
fn update_tomography(&mut self) {
let (nx, ny, nz) = self.occupancy_dims;
let total = nx * ny * nz;
// Simple backprojection from per-node RSSI
let mut new_occ = vec![0.0f64; total];
for (node_id, history) in &self.node_frames {
if let Some(latest) = history.back() {
// RSSI-based attenuation → voxel density
let atten = -(latest.rssi as f64);
let contribution = atten / 100.0; // normalize
// Distribute based on node ID position (simplified ray model)
let cx = match node_id {
1 => nx / 4,
2 => nx * 3 / 4,
_ => nx / 2,
};
let cy = ny / 2;
for iz in 0..nz {
for iy in 0..ny {
for ix in 0..nx {
let dx = (ix as f64 - cx as f64) / nx as f64;
let dy = (iy as f64 - cy as f64) / ny as f64;
let dist = (dx * dx + dy * dy).sqrt();
let idx = iz * ny * nx + iy * nx + ix;
// Gaussian-weighted contribution
new_occ[idx] += contribution * (-dist * dist * 8.0).exp();
}
}
}
}
}
// Normalize
let max = new_occ.iter().cloned().fold(0.0f64, f64::max);
if max > 0.0 {
for d in &mut new_occ { *d /= max; }
}
// Exponential moving average with previous occupancy
for i in 0..total {
self.occupancy[i] = self.occupancy[i] * 0.7 + new_occ[i] * 0.3;
}
}
}
/// Cosine similarity between two vectors. Returns 0.0 if either has zero magnitude.
fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
let len = a.len().min(b.len());
if len == 0 {
return 0.0;
}
let mut dot = 0.0f32;
let mut mag_a = 0.0f32;
let mut mag_b = 0.0f32;
for i in 0..len {
dot += a[i] * b[i];
mag_a += a[i] * a[i];
mag_b += b[i] * b[i];
}
let denom = mag_a.sqrt() * mag_b.sqrt();
if denom < 1e-9 {
0.0
} else {
dot / denom
}
}
// ─── UDP Receiver ───────────────────────────────────────────────────────────
/// Start the complete CSI pipeline — UDP receiver + processing.
pub fn start_pipeline(bind_addr: &str) -> Arc<Mutex<CsiPipelineState>> {
let state = Arc::new(Mutex::new(CsiPipelineState::default()));
let st = state.clone();
let addr = bind_addr.to_string();
std::thread::spawn(move || {
let socket = match UdpSocket::bind(&addr) {
Ok(s) => s,
Err(e) => {
eprintln!(" CSI pipeline: bind failed on {addr}: {e}");
return;
}
};
socket.set_read_timeout(Some(std::time::Duration::from_secs(1))).unwrap();
eprintln!(" CSI pipeline: listening on {addr}");
let mut buf = [0u8; 2048];
loop {
match socket.recv_from(&mut buf) {
Ok((n, _)) => {
if let Some(frame) = parse_adr018(&buf[..n]) {
st.lock().unwrap().process_frame(frame);
}
}
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => continue,
Err(_) => continue,
}
}
});
state
}
/// Get current pipeline output for fusion.
pub fn get_pipeline_output(state: &Arc<Mutex<CsiPipelineState>>) -> PipelineOutput {
let st = state.lock().unwrap();
PipelineOutput {
skeleton: st.skeleton.clone(),
vitals: st.vitals.clone(),
occupancy: st.occupancy.clone(),
occupancy_dims: st.occupancy_dims,
motion_detected: st.motion_detected,
total_frames: st.total_frames,
num_nodes: st.node_frames.len(),
current_location: st.current_location.clone(),
is_dark: st.is_dark,
}
}
#[derive(Clone, Debug, serde::Serialize)]
pub struct PipelineOutput {
pub skeleton: Option<Skeleton>,
pub vitals: VitalSigns,
pub occupancy: Vec<f64>,
pub occupancy_dims: (usize, usize, usize),
pub motion_detected: bool,
pub total_frames: u64,
pub num_nodes: usize,
pub current_location: Option<(String, f32)>,
pub is_dark: bool,
}
// Serialize implementations
impl serde::Serialize for Skeleton {
fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
use serde::ser::SerializeStruct;
let mut st = s.serialize_struct("Skeleton", 2)?;
st.serialize_field("keypoints", &self.keypoints)?;
st.serialize_field("confidence", &self.confidence)?;
st.end()
}
}
impl serde::Serialize for VitalSigns {
fn serialize<S: serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
use serde::ser::SerializeStruct;
let mut st = s.serialize_struct("VitalSigns", 3)?;
st.serialize_field("breathing_rate", &self.breathing_rate)?;
st.serialize_field("heart_rate", &self.heart_rate)?;
st.serialize_field("motion_score", &self.motion_score)?;
st.end()
}
}

View File

@ -10,8 +10,10 @@
//! ruview-pointcloud train # calibration training
//! ruview-pointcloud csi-test # send test CSI frames
mod brain_bridge;
mod camera;
mod csi;
mod csi_pipeline;
mod depth;
mod fusion;
mod pointcloud;

View File

@ -1,10 +1,11 @@
//! HTTP server — live camera + ESP32 CSI + fusion → real-time point cloud.
use crate::brain_bridge;
use crate::camera;
use crate::csi_pipeline;
use crate::depth;
use crate::fusion;
use crate::pointcloud;
use crate::serial_csi;
use axum::{
extract::State,
response::{Html, IntoResponse},
@ -16,32 +17,21 @@ use std::sync::{Arc, Mutex};
struct AppState {
latest_cloud: Mutex<pointcloud::PointCloud>,
latest_splats: Mutex<Vec<pointcloud::GaussianSplat>>,
latest_pipeline: Mutex<Option<csi_pipeline::PipelineOutput>>,
frame_count: Mutex<u64>,
use_camera: bool,
csi_state: Option<Arc<Mutex<serial_csi::CsiState>>>,
csi_pipeline: Option<Arc<Mutex<csi_pipeline::CsiPipelineState>>>,
}
pub async fn serve(host: &str, port: u16, _wifi_source: Option<&str>) -> anyhow::Result<()> {
let has_camera = camera::camera_available();
// CSI serial readers — only start if explicitly requested via env var
// (serial reader needs proper baud rate config to avoid reconnect loop)
let csi_state = if std::env::var("RUVIEW_CSI").is_ok() {
let mut csi_ports = Vec::new();
for p in &["/dev/ttyACM0", "/dev/ttyUSB0"] {
if std::path::Path::new(p).exists() { csi_ports.push(*p); }
}
if !csi_ports.is_empty() {
eprintln!(" CSI ports: {:?}", csi_ports);
Some(serial_csi::start_serial_readers(&csi_ports))
} else { None }
} else {
eprintln!(" CSI: disabled (set RUVIEW_CSI=1 to enable)");
None
};
// Start CSI pipeline — listens for UDP CSI data from ESP32 nodes
let csi_pipeline_state = csi_pipeline::start_pipeline("0.0.0.0:3333");
eprintln!(" CSI pipeline: UDP port 3333 (ADR-018 binary frames)");
let initial_cloud = if has_camera {
capture_live_cloud(csi_state.as_ref())
capture_camera_cloud()
} else {
demo_cloud()
};
@ -50,29 +40,55 @@ pub async fn serve(host: &str, port: u16, _wifi_source: Option<&str>) -> anyhow:
let state = Arc::new(AppState {
latest_cloud: Mutex::new(initial_cloud),
latest_splats: Mutex::new(initial_splats),
latest_pipeline: Mutex::new(None),
frame_count: Mutex::new(0),
use_camera: has_camera,
csi_state: csi_state.clone(),
csi_pipeline: Some(csi_pipeline_state.clone()),
});
// Background: capture + fuse every 500ms
// Background: capture + fuse every 500ms (motion-adaptive)
let bg = state.clone();
let bg_csi = csi_state.clone();
let bg_csi = Some(csi_pipeline_state.clone());
let bg_cam = has_camera;
tokio::spawn(async move {
let mut skip_depth = false;
loop {
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
let csi_clone = bg_csi.clone();
let cloud = if bg_cam {
tokio::task::spawn_blocking(move || capture_live_cloud(csi_clone.as_ref()))
// Motion-adaptive: check CSI motion score
let pipeline_out = bg_csi.as_ref().map(|c| csi_pipeline::get_pipeline_output(c));
if let Some(ref out) = pipeline_out {
// Only run expensive depth when motion detected or every 5th frame
let frame_num = *bg.frame_count.lock().unwrap();
skip_depth = !out.motion_detected && frame_num % 5 != 0;
}
let pipeline_clone = pipeline_out.clone();
*bg.latest_pipeline.lock().unwrap() = pipeline_out;
let pipeline_out = pipeline_clone;
let interval = if skip_depth { 1000 } else { 500 }; // slower when no motion
tokio::time::sleep(std::time::Duration::from_millis(interval)).await;
let cloud = if bg_cam && !skip_depth {
tokio::task::spawn_blocking(capture_camera_cloud)
.await.unwrap_or_else(|_| demo_cloud())
} else {
demo_cloud()
// Reuse previous cloud when no motion
bg.latest_cloud.lock().unwrap().clone()
};
let splats = pointcloud::to_gaussian_splats(&cloud);
*bg.latest_cloud.lock().unwrap() = cloud;
*bg.latest_splats.lock().unwrap() = splats;
*bg.frame_count.lock().unwrap() += 1;
let frame_num = {
let mut fc = bg.frame_count.lock().unwrap();
*fc += 1;
*fc
};
// Brain sync — sparse, every 120 frames (~60 seconds)
if frame_num % 120 == 0 {
if let Some(ref out) = pipeline_out {
brain_bridge::sync_to_brain(out, frame_num).await;
}
}
}
});
@ -98,70 +114,19 @@ pub async fn serve(host: &str, port: u16, _wifi_source: Option<&str>) -> anyhow:
Ok(())
}
fn capture_live_cloud(csi: Option<&Arc<Mutex<serial_csi::CsiState>>>) -> pointcloud::PointCloud {
// 1. Camera → depth → dense points
let cam_cloud = {
let config = camera::CameraConfig::default();
match camera::capture_frame(&config) {
Ok(frame) => {
match depth::estimate_depth(&frame.rgb, frame.width, frame.height) {
Ok(dm) => {
let intr = depth::CameraIntrinsics::default();
depth::backproject_depth(&dm, &intr, Some(&frame.rgb), 2)
}
Err(_) => depth::demo_depth_cloud(),
fn capture_camera_cloud() -> pointcloud::PointCloud {
let config = camera::CameraConfig::default();
match camera::capture_frame(&config) {
Ok(frame) => {
match depth::estimate_depth(&frame.rgb, frame.width, frame.height) {
Ok(dm) => {
let intr = depth::CameraIntrinsics::default();
depth::backproject_depth(&dm, &intr, Some(&frame.rgb), 2)
}
Err(_) => depth::demo_depth_cloud(),
}
Err(_) => depth::demo_depth_cloud(),
}
};
// 2. CSI → motion + presence → modify point cloud
let mut clouds: Vec<&pointcloud::PointCloud> = vec![&cam_cloud];
let csi_cloud;
if let Some(csi_state) = csi {
let (motion, distance, frames) = serial_csi::get_csi_influence(csi_state);
if frames > 0 {
// Create CSI-informed occupancy around detected presence
let mut occ = fusion::OccupancyVolume {
densities: vec![0.0; 8 * 8 * 4],
nx: 8, ny: 8, nz: 4,
bounds: [
-distance as f64, -distance as f64, 0.0,
distance as f64, distance as f64, 2.5,
],
occupied_count: 0,
};
// Place high density where CSI indicates presence
let cx: usize = 4; let cy: usize = 4;
let radius: usize = (motion * 3.0).max(1.0) as usize;
for iz in 0..4 {
for iy in (cy.saturating_sub(radius))..=(cy + radius).min(7) {
for ix in (cx.saturating_sub(radius))..=(cx + radius).min(7) {
let idx = iz * 64 + iy * 8 + ix;
let dx = (ix as f32 - cx as f32).abs() / radius as f32;
let dy = (iy as f32 - cy as f32).abs() / radius as f32;
let r2 = dx * dx + dy * dy;
if r2 < 1.0 {
occ.densities[idx] = (1.0 - r2 as f64) * (0.5 + motion as f64 * 0.5);
}
}
}
}
occ.occupied_count = occ.densities.iter().filter(|&&d| d > 0.3).count();
csi_cloud = fusion::occupancy_to_pointcloud(&occ);
clouds.push(&csi_cloud);
}
}
// 3. Fuse camera + CSI
if clouds.len() > 1 {
fusion::fuse_clouds(&clouds, 0.04)
} else {
cam_cloud
Err(_) => depth::demo_depth_cloud(),
}
}
@ -176,13 +141,13 @@ async fn api_cloud(State(state): State<Arc<AppState>>) -> Json<serde_json::Value
let cloud = state.latest_cloud.lock().unwrap();
let (min, max) = cloud.bounds();
let frames = *state.frame_count.lock().unwrap();
let csi_info = state.csi_state.as_ref().map(|c| serial_csi::get_csi_influence(c));
let pipeline = state.latest_pipeline.lock().unwrap();
Json(serde_json::json!({
"points": cloud.points.len(),
"bounds_min": min, "bounds_max": max,
"live": state.use_camera,
"frame": frames,
"csi": csi_info.map(|(m,d,f)| serde_json::json!({"motion":m,"distance_m":d,"frames":f})),
"pipeline": &*pipeline,
"cloud": cloud.points.iter().take(1000).collect::<Vec<_>>(),
}))
}
@ -190,29 +155,28 @@ async fn api_cloud(State(state): State<Arc<AppState>>) -> Json<serde_json::Value
async fn api_splats(State(state): State<Arc<AppState>>) -> Json<serde_json::Value> {
let splats = state.latest_splats.lock().unwrap();
let frames = *state.frame_count.lock().unwrap();
let csi_info = state.csi_state.as_ref().map(|c| serial_csi::get_csi_influence(c));
let pipeline = state.latest_pipeline.lock().unwrap();
Json(serde_json::json!({
"splats": &*splats,
"count": splats.len(),
"live": state.use_camera,
"frame": frames,
"csi": csi_info.map(|(m,d,f)| serde_json::json!({"motion":m,"distance_m":d,"frames":f})),
"pipeline": &*pipeline,
"timestamp": chrono::Utc::now().timestamp_millis(),
}))
}
async fn api_status(State(state): State<Arc<AppState>>) -> Json<serde_json::Value> {
let frames = *state.frame_count.lock().unwrap();
let csi_info = state.csi_state.as_ref().map(|c| serial_csi::get_csi_influence(c));
let pipeline = state.latest_pipeline.lock().unwrap();
Json(serde_json::json!({
"status": "ok",
"version": env!("CARGO_PKG_VERSION"),
"live": state.use_camera,
"camera": if state.use_camera { "/dev/video0" } else { "demo" },
"csi_ports": if state.csi_state.is_some() { "active" } else { "none" },
"csi": csi_info.map(|(m,d,f)| serde_json::json!({"motion":m,"distance_m":d,"frames":f})),
"csi_pipeline": "active (UDP:3333)",
"pipeline": &*pipeline,
"frames_captured": frames,
"fps": 2,
}))
}
@ -228,8 +192,10 @@ async fn index() -> Html<String> {
<style>
body { margin: 0; background: #0a0a0a; color: #e8a634; font-family: monospace; }
canvas { display: block; }
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 200px; }
#info { position: absolute; top: 10px; left: 10px; padding: 12px; background: rgba(0,0,0,0.85); border: 1px solid #e8a634; border-radius: 6px; min-width: 240px; font-size: 13px; line-height: 1.5; }
.live { color: #4f4; } .demo { color: #f44; }
.section { margin-top: 6px; padding-top: 6px; border-top: 1px solid #333; }
.label { color: #888; }
</style>
<script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/r128/three.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/controls/OrbitControls.js"></script>
@ -240,38 +206,171 @@ async fn index() -> Html<String> {
<div id="stats">Loading...</div>
</div>
<script>
const scene = new THREE.Scene();
var scene = new THREE.Scene();
scene.background = new THREE.Color(0x0a0a0a);
const camera = new THREE.PerspectiveCamera(75, window.innerWidth/window.innerHeight, 0.1, 100);
camera.position.set(0, 0, -2);
camera.lookAt(0, 0, 3);
var camera = new THREE.PerspectiveCamera(75, window.innerWidth/window.innerHeight, 0.1, 100);
camera.position.set(0, 2, -4);
camera.lookAt(0, 0, 2);
const renderer = new THREE.WebGLRenderer({ antialias: true });
var renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);
const controls = new THREE.OrbitControls(camera, renderer.domElement);
var controls = new THREE.OrbitControls(camera, renderer.domElement);
controls.enableDamping = true;
controls.target.set(0, 0, 3);
controls.target.set(0, 0, 2);
let pointsMesh = null;
let lastFrame = -1;
var pointsMesh = null;
var lastFrame = -1;
var skeletonGroup = null;
var prevTimestamp = 0;
var frameRateVal = 0;
// COCO skeleton connections: pairs of keypoint indices
// 0=nose 1=leftEye 2=rightEye 3=leftEar 4=rightEar
// 5=leftShoulder 6=rightShoulder 7=leftElbow 8=rightElbow
// 9=leftWrist 10=rightWrist 11=leftHip 12=rightHip
// 13=leftKnee 14=rightKnee 15=leftAnkle 16=rightAnkle
var COCO_BONES = [
[0,1],[0,2],[1,3],[2,4],
[5,6],[5,7],[7,9],[6,8],[8,10],
[5,11],[6,12],[11,12],
[11,13],[13,15],[12,14],[14,16]
];
function clearSkeleton() {
if (skeletonGroup) {
scene.remove(skeletonGroup);
skeletonGroup.traverse(function(obj) {
if (obj.geometry) obj.geometry.dispose();
if (obj.material) obj.material.dispose();
});
skeletonGroup = null;
}
}
function drawSkeleton(keypoints) {
clearSkeleton();
if (!keypoints || keypoints.length < 17) return;
skeletonGroup = new THREE.Group();
// Map keypoints from [0,1] to scene coords
// x: [-2, 2], y: [2, -2] (flip y), z: fixed at 2
var sphereGeo = new THREE.SphereGeometry(0.04, 8, 8);
var sphereMat = new THREE.MeshBasicMaterial({ color: 0xffff00 });
var positions3D = [];
var i, kp, sx, sy;
for (i = 0; i < 17; i++) {
kp = keypoints[i];
if (!kp) { positions3D.push(null); continue; }
sx = (kp[0] - 0.5) * 4;
sy = (0.5 - kp[1]) * 4;
positions3D.push([sx, sy, 2]);
var sphere = new THREE.Mesh(sphereGeo, sphereMat);
sphere.position.set(sx, sy, 2);
skeletonGroup.add(sphere);
}
// Draw bones as white lines
var lineMat = new THREE.LineBasicMaterial({ color: 0xffffff, linewidth: 2 });
var b, a, bIdx;
for (b = 0; b < COCO_BONES.length; b++) {
a = COCO_BONES[b][0];
bIdx = COCO_BONES[b][1];
if (!positions3D[a] || !positions3D[bIdx]) continue;
var lineGeo = new THREE.BufferGeometry();
var verts = new Float32Array([
positions3D[a][0], positions3D[a][1], positions3D[a][2],
positions3D[bIdx][0], positions3D[bIdx][1], positions3D[bIdx][2]
]);
lineGeo.setAttribute("position", new THREE.BufferAttribute(verts, 3));
var line = new THREE.Line(lineGeo, lineMat);
skeletonGroup.add(line);
}
scene.add(skeletonGroup);
}
async function fetchCloud() {
try {
const resp = await fetch('/api/splats');
const data = await resp.json();
var resp = await fetch("/api/splats");
var data = await resp.json();
if (data.splats && data.frame !== lastFrame) {
// Compute CSI frame rate
var now = Date.now();
if (prevTimestamp > 0) {
var dt = (now - prevTimestamp) / 1000.0;
if (dt > 0) frameRateVal = (1.0 / dt).toFixed(1);
}
prevTimestamp = now;
lastFrame = data.frame;
updateSplats(data.splats);
const mode = data.live ? '<span class="live"> LIVE</span>' : '<span class="demo"> DEMO</span>';
let csiInfo = '';
if (data.csi) {
const m = (data.csi.motion * 100).toFixed(0);
csiInfo = `<br>CSI: ${data.csi.frames} frames, motion ${m}%<br>Distance: ${data.csi.distance_m.toFixed(1)}m`;
// Draw skeleton if available
var pipe = data.pipeline;
if (pipe && pipe.skeleton && pipe.skeleton.keypoints) {
drawSkeleton(pipe.skeleton.keypoints);
} else {
clearSkeleton();
}
document.getElementById('stats').innerHTML =
`${mode} Camera + CSI<br>Splats: ${data.count}<br>Frame: ${data.frame}${csiInfo}`;
// Build info panel
var mode = data.live
? '<span class="live">&#9679; LIVE</span>'
: '<span class="demo">&#9679; DEMO</span>';
var html = mode + " Camera + CSI<br>"
+ "Splats: " + data.count + "<br>"
+ "Frame: " + data.frame;
// CSI frame rate
html += '<div class="section">'
+ '<span class="label">CSI Rate:</span> '
+ frameRateVal + " fps</div>";
// Skeleton confidence
if (pipe && pipe.skeleton && pipe.skeleton.confidence !== undefined) {
var conf = (pipe.skeleton.confidence * 100).toFixed(0);
html += '<div class="section">'
+ '<span class="label">Skeleton:</span> '
+ conf + "% confidence</div>";
}
// Weather data
if (pipe && pipe.weather) {
var w = pipe.weather;
html += '<div class="section">'
+ '<span class="label">Weather:</span> ';
if (w.temperature !== undefined) {
html += w.temperature + "&deg;C";
}
if (w.conditions) {
html += " " + w.conditions;
}
html += "</div>";
}
// Building count from geo
if (pipe && pipe.geo && pipe.geo.building_count !== undefined) {
html += '<div class="section">'
+ '<span class="label">Buildings:</span> '
+ pipe.geo.building_count + "</div>";
}
// Vitals
if (pipe && pipe.vitals) {
var v = pipe.vitals;
html += '<div class="section">'
+ '<span class="label">Vitals:</span> ';
if (v.breathing_rate !== undefined) {
html += "BR " + v.breathing_rate + "/min";
}
if (v.motion_score !== undefined) {
html += " Motion " + (v.motion_score * 100).toFixed(0) + "%";
}
html += "</div>";
}
document.getElementById("stats").innerHTML = html;
}
} catch(e) {}
}
@ -280,21 +379,23 @@ async fn index() -> Html<String> {
function updateSplats(splats) {
if (pointsMesh) scene.remove(pointsMesh);
const geometry = new THREE.BufferGeometry();
const positions = new Float32Array(splats.length * 3);
const colors = new Float32Array(splats.length * 3);
splats.forEach((s, i) => {
var geometry = new THREE.BufferGeometry();
var positions = new Float32Array(splats.length * 3);
var colors = new Float32Array(splats.length * 3);
var i, s;
for (i = 0; i < splats.length; i++) {
s = splats[i];
positions[i*3] = s.center[0];
positions[i*3+1] = -s.center[1];
positions[i*3+2] = s.center[2];
colors[i*3] = s.color[0];
colors[i*3+1] = s.color[1];
colors[i*3+2] = s.color[2];
});
geometry.setAttribute('position', new THREE.BufferAttribute(positions, 3));
geometry.setAttribute('color', new THREE.BufferAttribute(colors, 3));
}
geometry.setAttribute("position", new THREE.BufferAttribute(positions, 3));
geometry.setAttribute("color", new THREE.BufferAttribute(colors, 3));
pointsMesh = new THREE.Points(geometry, new THREE.PointsMaterial({
size: 0.025, vertexColors: true, sizeAttenuation: true,
size: 0.02, vertexColors: true, sizeAttenuation: true
}));
scene.add(pointsMesh);
}
@ -305,7 +406,7 @@ async fn index() -> Html<String> {
renderer.render(scene, camera);
}
animate();
window.addEventListener('resize', () => {
window.addEventListener("resize", function() {
camera.aspect = window.innerWidth / window.innerHeight;
camera.updateProjectionMatrix();
renderer.setSize(window.innerWidth, window.innerHeight);