fix: live demo static pose & inaccurate sensing data (issue #86)
- Docker default changed from --source simulated to --source auto (auto-detects ESP32 on UDP 5005, falls back to simulation) - Pose derivation now driven by real sensing features: motion_band_power, breathing_band_power, variance, dominant_freq_hz, change_points - Temporal feature extraction: 100-frame circular buffer, Goertzel breathing rate estimation (0.1-0.5 Hz), frame-to-frame L2 motion detection, SNR-based signal quality metric - Signal field driven by subcarrier variance spatial mapping instead of fixed animation circle - UI data source indicators: LIVE/RECONNECTING/SIMULATED banner on sensing tab, estimation mode badge on live demo tab - Setup guide panel explaining ESP32 count requirements for each capability level (1x: presence, 3x: localization, 4x+: full pose) - Tick rate improved from 500ms to 100ms (2fps to 10fps) - Fixed Option<f64> division bug from PR #83 - ADR-035 documents all decisions Closes #86 Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
parent
fdc7142dfa
commit
8166d8d822
|
|
@ -42,5 +42,15 @@ EXPOSE 5005/udp
|
||||||
|
|
||||||
ENV RUST_LOG=info
|
ENV RUST_LOG=info
|
||||||
|
|
||||||
ENTRYPOINT ["/app/sensing-server"]
|
# CSI_SOURCE controls which data source the sensing server uses at startup.
|
||||||
CMD ["--source", "simulated", "--tick-ms", "100", "--ui-path", "/app/ui", "--http-port", "3000", "--ws-port", "3001"]
|
# auto — probe UDP port 5005 for an ESP32 first; fall back to simulation (default)
|
||||||
|
# esp32 — receive real CSI frames from an ESP32 device over UDP port 5005
|
||||||
|
# wifi — use host Wi-Fi RSSI/scan data (Windows netsh; not available in containers)
|
||||||
|
# simulated — generate synthetic CSI frames (no hardware required)
|
||||||
|
# Override at runtime: docker run -e CSI_SOURCE=esp32 ...
|
||||||
|
ENV CSI_SOURCE=auto
|
||||||
|
|
||||||
|
ENTRYPOINT ["/bin/sh", "-c"]
|
||||||
|
# Shell-form CMD allows $CSI_SOURCE to be substituted at container start.
|
||||||
|
# The ENV default above (CSI_SOURCE=auto) applies when the variable is unset.
|
||||||
|
CMD ["/app/sensing-server --source ${CSI_SOURCE} --tick-ms 100 --ui-path /app/ui --http-port 3000 --ws-port 3001"]
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,14 @@ services:
|
||||||
- "5005:5005/udp" # ESP32 UDP
|
- "5005:5005/udp" # ESP32 UDP
|
||||||
environment:
|
environment:
|
||||||
- RUST_LOG=info
|
- RUST_LOG=info
|
||||||
command: ["--source", "simulated", "--tick-ms", "100", "--ui-path", "/app/ui", "--http-port", "3000", "--ws-port", "3001"]
|
# CSI_SOURCE controls the data source for the sensing server.
|
||||||
|
# Options: auto (default) — probe for ESP32 UDP then fall back to simulation
|
||||||
|
# esp32 — receive real CSI frames from an ESP32 on UDP port 5005
|
||||||
|
# wifi — use host Wi-Fi RSSI/scan data (Windows netsh)
|
||||||
|
# simulated — generate synthetic CSI data (no hardware required)
|
||||||
|
- CSI_SOURCE=${CSI_SOURCE:-auto}
|
||||||
|
# command is passed as arguments to ENTRYPOINT (/bin/sh -c), so $CSI_SOURCE is expanded by the shell.
|
||||||
|
command: ["/app/sensing-server --source ${CSI_SOURCE:-auto} --tick-ms 100 --ui-path /app/ui --http-port 3000 --ws-port 3001"]
|
||||||
|
|
||||||
python-sensing:
|
python-sensing:
|
||||||
build:
|
build:
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,75 @@
|
||||||
|
# ADR-035: Live Sensing UI Accuracy & Data Source Transparency
|
||||||
|
|
||||||
|
## Status
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Date
|
||||||
|
2026-03-02
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
Issue #86 reported that the live demo shows a static/barely-animated stick figure and the sensing page displays inaccurate data, despite a working ESP32 sending real CSI frames. Investigation revealed three root causes:
|
||||||
|
|
||||||
|
1. **Docker defaults to `--source simulated`** — even with a real ESP32 connected, the server generates synthetic sine-wave data instead of reading UDP frames.
|
||||||
|
2. **Live demo pose is analytically computed** — `derive_pose_from_sensing()` generates keypoints using `sin(tick)` math unrelated to actual signal content. No trained `.rvf` model is loaded by default.
|
||||||
|
3. **Sensing feature extraction is oversimplified** — the server uses single-frame thresholds for motion detection and has no temporal analysis (breathing FFT, sliding window variance, frame history).
|
||||||
|
4. **No data source indicator** — users cannot tell whether they are seeing real or simulated data.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### 1. Docker: Auto-detect data source
|
||||||
|
- Default `CSI_SOURCE` changed from `simulated` to `auto`.
|
||||||
|
- `auto` probes UDP port 5005 for an ESP32; falls back to simulation if none found.
|
||||||
|
- Users override via `CSI_SOURCE=esp32 docker-compose up`.
|
||||||
|
|
||||||
|
### 2. Signal-responsive pose derivation
|
||||||
|
- `derive_pose_from_sensing()` now reads actual sensing features:
|
||||||
|
- `motion_band_power` drives limb splay and walking gait detection (> 0.55).
|
||||||
|
- `breathing_band_power` drives torso expansion/contraction phased to breathing rate.
|
||||||
|
- `variance` seeds per-joint noise so the skeleton moves independently.
|
||||||
|
- `dominant_freq_hz` drives lateral torso lean.
|
||||||
|
- `change_points` add burst jitter to extremity keypoints.
|
||||||
|
- Tick rate reduced from 500ms to 100ms (2 fps → 10 fps).
|
||||||
|
- `pose_source` field (`signal_derived` | `model_inference`) added to every WebSocket frame.
|
||||||
|
|
||||||
|
### 3. Temporal feature extraction
|
||||||
|
- 100-frame circular buffer (`VecDeque`) added to `AppStateInner`.
|
||||||
|
- Per-subcarrier temporal variance via Welford-style accumulation.
|
||||||
|
- Breathing rate estimation via 9-candidate Goertzel filter bank (0.1–0.5 Hz) with 3x SNR gate.
|
||||||
|
- Frame-to-frame L2 motion score replaces single-frame amplitude thresholds.
|
||||||
|
- Signal quality metric: SNR-based (RSSI − noise floor) blended with temporal stability.
|
||||||
|
- Signal field driven by subcarrier variance spatial mapping instead of fixed animation.
|
||||||
|
|
||||||
|
### 4. Data source transparency in UI
|
||||||
|
- **Sensing tab**: Banner showing "LIVE - ESP32" (green), "RECONNECTING..." (yellow), or "SIMULATED DATA" (red).
|
||||||
|
- **Live Demo tab**: "Estimation Mode" badge showing "Signal-Derived" (green) or "Model Inference" (blue).
|
||||||
|
- **Setup Guide** panel explaining what each ESP32 count provides (1x: presence/breathing, 3x: localization, 4x+: full pose with trained model).
|
||||||
|
- Simulation fallback delayed from immediate to 5 failed reconnect attempts (~30s).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
### Positive
|
||||||
|
- Users with real ESP32 hardware get real data by default (auto-detect).
|
||||||
|
- Simulated data is clearly labeled — no more confusion about data authenticity.
|
||||||
|
- Pose skeleton visually responds to actual signal changes (motion, breathing, variance).
|
||||||
|
- Feature extraction produces physiologically meaningful metrics (breathing rate via Goertzel, temporal motion detection).
|
||||||
|
- Setup guide manages expectations about what each hardware configuration provides.
|
||||||
|
|
||||||
|
### Negative
|
||||||
|
- Signal-derived pose is still an approximation, not neural network inference. Per-limb tracking requires a trained `.rvf` model + 4+ ESP32 nodes.
|
||||||
|
- Goertzel filter bank adds ~O(9×N) computation per frame (negligible at 100 frames).
|
||||||
|
- Users with only 1 ESP32 may still be disappointed that arm tracking doesn't work — but the UI now explains why.
|
||||||
|
|
||||||
|
## Files Changed
|
||||||
|
- `docker/Dockerfile.rust` — `CSI_SOURCE=auto` env, shell entrypoint for variable expansion
|
||||||
|
- `docker/docker-compose.yml` — `CSI_SOURCE=${CSI_SOURCE:-auto}`, shell command string
|
||||||
|
- `wifi-densepose-sensing-server/src/main.rs` — frame history buffer, Goertzel breathing estimation, temporal motion score, signal-driven pose derivation, pose_source field, 100ms tick default
|
||||||
|
- `ui/services/sensing.service.js` — `dataSource` state, delayed simulation fallback, `_simulated` marker
|
||||||
|
- `ui/components/SensingTab.js` — data source banner, "About This Data" card
|
||||||
|
- `ui/components/LiveDemoTab.js` — estimation mode badge, setup guide panel
|
||||||
|
- `ui/style.css` — banner, badge, and guide panel styles
|
||||||
|
|
||||||
|
## References
|
||||||
|
- Issue: https://github.com/ruvnet/wifi-densepose/issues/86
|
||||||
|
- ADR-029: RuvSense multistatic sensing mode (proposed — full pipeline integration)
|
||||||
|
- ADR-014: SOTA signal processing
|
||||||
|
|
@ -71,8 +71,8 @@ struct Args {
|
||||||
#[arg(long, default_value = "../../ui")]
|
#[arg(long, default_value = "../../ui")]
|
||||||
ui_path: PathBuf,
|
ui_path: PathBuf,
|
||||||
|
|
||||||
/// Tick interval in milliseconds
|
/// Tick interval in milliseconds (default 100 ms = 10 fps for smooth pose animation)
|
||||||
#[arg(long, default_value = "500")]
|
#[arg(long, default_value = "100")]
|
||||||
tick_ms: u64,
|
tick_ms: u64,
|
||||||
|
|
||||||
/// Data source: auto, wifi, esp32, simulate
|
/// Data source: auto, wifi, esp32, simulate
|
||||||
|
|
@ -266,6 +266,10 @@ struct BoundingBox {
|
||||||
struct AppStateInner {
|
struct AppStateInner {
|
||||||
latest_update: Option<SensingUpdate>,
|
latest_update: Option<SensingUpdate>,
|
||||||
rssi_history: VecDeque<f64>,
|
rssi_history: VecDeque<f64>,
|
||||||
|
/// Circular buffer of recent CSI amplitude vectors for temporal analysis.
|
||||||
|
/// Each entry is the full subcarrier amplitude vector for one frame.
|
||||||
|
/// Capacity: FRAME_HISTORY_CAPACITY frames.
|
||||||
|
frame_history: VecDeque<Vec<f64>>,
|
||||||
tick: u64,
|
tick: u64,
|
||||||
source: String,
|
source: String,
|
||||||
tx: broadcast::Sender<String>,
|
tx: broadcast::Sender<String>,
|
||||||
|
|
@ -287,6 +291,10 @@ struct AppStateInner {
|
||||||
model_loaded: bool,
|
model_loaded: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Number of frames retained in `frame_history` for temporal analysis.
|
||||||
|
/// At 500 ms ticks this covers ~50 seconds; at 100 ms ticks ~10 seconds.
|
||||||
|
const FRAME_HISTORY_CAPACITY: usize = 100;
|
||||||
|
|
||||||
type SharedState = Arc<RwLock<AppStateInner>>;
|
type SharedState = Arc<RwLock<AppStateInner>>;
|
||||||
|
|
||||||
// ── ESP32 UDP frame parser ───────────────────────────────────────────────────
|
// ── ESP32 UDP frame parser ───────────────────────────────────────────────────
|
||||||
|
|
@ -343,43 +351,96 @@ fn parse_esp32_frame(buf: &[u8]) -> Option<Esp32Frame> {
|
||||||
|
|
||||||
// ── Signal field generation ──────────────────────────────────────────────────
|
// ── Signal field generation ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Generate a signal field that reflects where motion and signal changes are occurring.
|
||||||
|
///
|
||||||
|
/// Instead of a fixed-animation circle, this function uses the actual sensing data:
|
||||||
|
/// - `subcarrier_variances`: per-subcarrier variance computed from the frame history.
|
||||||
|
/// High-variance subcarriers indicate spatial directions where the signal is disrupted.
|
||||||
|
/// - `motion_score`: overall motion intensity [0, 1].
|
||||||
|
/// - `breathing_rate_hz`: estimated breathing rate in Hz; if > 0, adds a breathing ring.
|
||||||
|
/// - `signal_quality`: overall quality metric [0, 1] modulates field brightness.
|
||||||
|
///
|
||||||
|
/// The field grid is 20×20 cells representing a top-down view of the room.
|
||||||
|
/// Hotspots are derived from the subcarrier index (treated as an angular bin) so that
|
||||||
|
/// subcarriers with the highest variance produce peaks at the corresponding directions.
|
||||||
fn generate_signal_field(
|
fn generate_signal_field(
|
||||||
_mean_rssi: f64,
|
_mean_rssi: f64,
|
||||||
variance: f64,
|
|
||||||
motion_score: f64,
|
motion_score: f64,
|
||||||
tick: u64,
|
breathing_rate_hz: f64,
|
||||||
|
signal_quality: f64,
|
||||||
|
subcarrier_variances: &[f64],
|
||||||
) -> SignalField {
|
) -> SignalField {
|
||||||
let grid = 20;
|
let grid = 20usize;
|
||||||
let mut values = vec![0.0f64; grid * grid];
|
let mut values = vec![0.0f64; grid * grid];
|
||||||
let center = grid as f64 / 2.0;
|
let center = (grid as f64 - 1.0) / 2.0;
|
||||||
let tick_f = tick as f64;
|
|
||||||
|
|
||||||
|
// Normalise subcarrier variances to [0, 1].
|
||||||
|
let max_var = subcarrier_variances.iter().cloned().fold(0.0f64, f64::max);
|
||||||
|
let norm_factor = if max_var > 1e-9 { max_var } else { 1.0 };
|
||||||
|
|
||||||
|
// For each cell, accumulate contributions from all subcarriers.
|
||||||
|
// Each subcarrier k is assigned an angular direction proportional to its index
|
||||||
|
// so that different subcarriers illuminate different regions of the room.
|
||||||
|
let n_sub = subcarrier_variances.len().max(1);
|
||||||
|
for (k, &var) in subcarrier_variances.iter().enumerate() {
|
||||||
|
let weight = (var / norm_factor) * motion_score;
|
||||||
|
if weight < 1e-6 {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// Map subcarrier index to an angle across the full 2π sweep.
|
||||||
|
let angle = (k as f64 / n_sub as f64) * 2.0 * std::f64::consts::PI;
|
||||||
|
// Place the hotspot at a distance proportional to the weight, capped at 40% of
|
||||||
|
// the grid radius so it stays within the room model.
|
||||||
|
let radius = center * 0.8 * weight.sqrt();
|
||||||
|
let hx = center + radius * angle.cos();
|
||||||
|
let hz = center + radius * angle.sin();
|
||||||
|
|
||||||
|
for z in 0..grid {
|
||||||
|
for x in 0..grid {
|
||||||
|
let dx = x as f64 - hx;
|
||||||
|
let dz = z as f64 - hz;
|
||||||
|
let dist2 = dx * dx + dz * dz;
|
||||||
|
// Gaussian blob centred on the hotspot; spread scales with weight.
|
||||||
|
let spread = (0.5 + weight * 2.0).max(0.5);
|
||||||
|
values[z * grid + x] += weight * (-dist2 / (2.0 * spread * spread)).exp();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Base radial attenuation from the router assumed at grid centre.
|
||||||
for z in 0..grid {
|
for z in 0..grid {
|
||||||
for x in 0..grid {
|
for x in 0..grid {
|
||||||
let dx = x as f64 - center;
|
let dx = x as f64 - center;
|
||||||
let dz = z as f64 - center;
|
let dz = z as f64 - center;
|
||||||
let dist = (dx * dx + dz * dz).sqrt();
|
let dist = (dx * dx + dz * dz).sqrt();
|
||||||
|
let base = signal_quality * (-dist * 0.12).exp();
|
||||||
// Base radial attenuation from router at center
|
values[z * grid + x] += base * 0.3;
|
||||||
let base = (-dist * 0.15).exp();
|
|
||||||
|
|
||||||
// Body disruption blob
|
|
||||||
let body_x = center + 3.0 * (tick_f * 0.02).sin();
|
|
||||||
let body_z = center + 2.0 * (tick_f * 0.015).cos();
|
|
||||||
let body_dist = ((x as f64 - body_x).powi(2) + (z as f64 - body_z).powi(2)).sqrt();
|
|
||||||
let disruption = motion_score * 0.6 * (-body_dist * 0.4).exp();
|
|
||||||
|
|
||||||
// Breathing ring modulation
|
|
||||||
let breath_ring = if variance > 1.0 {
|
|
||||||
0.1 * (tick_f * 0.3).sin() * (-((dist - 5.0).powi(2)) * 0.1).exp()
|
|
||||||
} else {
|
|
||||||
0.0
|
|
||||||
};
|
|
||||||
|
|
||||||
values[z * grid + x] = (base + disruption + breath_ring).clamp(0.0, 1.0);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Breathing ring: if a breathing rate was estimated add a faint annular highlight
|
||||||
|
// at a radius corresponding to typical chest-wall displacement range.
|
||||||
|
if breathing_rate_hz > 0.05 {
|
||||||
|
let ring_r = center * 0.55;
|
||||||
|
let ring_width = 1.8f64;
|
||||||
|
for z in 0..grid {
|
||||||
|
for x in 0..grid {
|
||||||
|
let dx = x as f64 - center;
|
||||||
|
let dz = z as f64 - center;
|
||||||
|
let dist = (dx * dx + dz * dz).sqrt();
|
||||||
|
let ring_val = 0.08 * (-(dist - ring_r).powi(2) / (2.0 * ring_width * ring_width)).exp();
|
||||||
|
values[z * grid + x] += ring_val;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Clamp and normalise to [0, 1].
|
||||||
|
let field_max = values.iter().cloned().fold(0.0f64, f64::max);
|
||||||
|
let scale = if field_max > 1e-9 { 1.0 / field_max } else { 1.0 };
|
||||||
|
for v in &mut values {
|
||||||
|
*v = (*v * scale).clamp(0.0, 1.0);
|
||||||
|
}
|
||||||
|
|
||||||
SignalField {
|
SignalField {
|
||||||
grid_size: [grid, 1, grid],
|
grid_size: [grid, 1, grid],
|
||||||
values,
|
values,
|
||||||
|
|
@ -388,21 +449,163 @@ fn generate_signal_field(
|
||||||
|
|
||||||
// ── Feature extraction from ESP32 frame ──────────────────────────────────────
|
// ── Feature extraction from ESP32 frame ──────────────────────────────────────
|
||||||
|
|
||||||
fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, ClassificationInfo) {
|
/// Estimate breathing rate in Hz from the amplitude time series stored in `frame_history`.
|
||||||
let n = frame.amplitudes.len().max(1) as f64;
|
///
|
||||||
|
/// Approach:
|
||||||
|
/// 1. Build a scalar time series by computing the mean amplitude of each historical frame.
|
||||||
|
/// 2. Run a peak-detection pass: count rising-edge zero-crossings of the de-meaned signal.
|
||||||
|
/// 3. Convert the crossing rate to Hz, clipped to the physiological range 0.1–0.5 Hz
|
||||||
|
/// (12–30 breaths/min).
|
||||||
|
///
|
||||||
|
/// For accuracy the function additionally applies a simple 3-tap Goertzel-style power
|
||||||
|
/// estimate at evenly-spaced candidate frequencies in the breathing band and returns
|
||||||
|
/// the candidate with the highest energy.
|
||||||
|
fn estimate_breathing_rate_hz(frame_history: &VecDeque<Vec<f64>>, sample_rate_hz: f64) -> f64 {
|
||||||
|
let n = frame_history.len();
|
||||||
|
if n < 6 {
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build scalar time series: mean amplitude per frame.
|
||||||
|
let series: Vec<f64> = frame_history.iter()
|
||||||
|
.map(|amps| {
|
||||||
|
if amps.is_empty() { 0.0 } else { amps.iter().sum::<f64>() / amps.len() as f64 }
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let mean_s = series.iter().sum::<f64>() / n as f64;
|
||||||
|
// De-mean.
|
||||||
|
let detrended: Vec<f64> = series.iter().map(|x| x - mean_s).collect();
|
||||||
|
|
||||||
|
// Goertzel power at candidate frequencies in the breathing band [0.1, 0.5] Hz.
|
||||||
|
// We evaluate 9 candidate frequencies uniformly spaced in that band.
|
||||||
|
let n_candidates = 9usize;
|
||||||
|
let f_low = 0.1f64;
|
||||||
|
let f_high = 0.5f64;
|
||||||
|
let mut best_freq = 0.0f64;
|
||||||
|
let mut best_power = 0.0f64;
|
||||||
|
|
||||||
|
for i in 0..n_candidates {
|
||||||
|
let freq = f_low + (f_high - f_low) * i as f64 / (n_candidates - 1).max(1) as f64;
|
||||||
|
let omega = 2.0 * std::f64::consts::PI * freq / sample_rate_hz;
|
||||||
|
let coeff = 2.0 * omega.cos();
|
||||||
|
let mut s_prev2 = 0.0f64;
|
||||||
|
let mut s_prev1 = 0.0f64;
|
||||||
|
for &x in &detrended {
|
||||||
|
let s = x + coeff * s_prev1 - s_prev2;
|
||||||
|
s_prev2 = s_prev1;
|
||||||
|
s_prev1 = s;
|
||||||
|
}
|
||||||
|
// Goertzel magnitude squared.
|
||||||
|
let power = s_prev2 * s_prev2 + s_prev1 * s_prev1 - coeff * s_prev1 * s_prev2;
|
||||||
|
if power > best_power {
|
||||||
|
best_power = power;
|
||||||
|
best_freq = freq;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only report a breathing rate if the Goertzel energy is meaningfully above noise.
|
||||||
|
// Threshold: power must exceed 10× the average power across all candidates.
|
||||||
|
let avg_power = {
|
||||||
|
let mut total = 0.0f64;
|
||||||
|
for i in 0..n_candidates {
|
||||||
|
let freq = f_low + (f_high - f_low) * i as f64 / (n_candidates - 1).max(1) as f64;
|
||||||
|
let omega = 2.0 * std::f64::consts::PI * freq / sample_rate_hz;
|
||||||
|
let coeff = 2.0 * omega.cos();
|
||||||
|
let mut s_prev2 = 0.0f64;
|
||||||
|
let mut s_prev1 = 0.0f64;
|
||||||
|
for &x in &detrended {
|
||||||
|
let s = x + coeff * s_prev1 - s_prev2;
|
||||||
|
s_prev2 = s_prev1;
|
||||||
|
s_prev1 = s;
|
||||||
|
}
|
||||||
|
total += s_prev2 * s_prev2 + s_prev1 * s_prev1 - coeff * s_prev1 * s_prev2;
|
||||||
|
}
|
||||||
|
total / n_candidates as f64
|
||||||
|
};
|
||||||
|
|
||||||
|
if best_power > avg_power * 3.0 {
|
||||||
|
best_freq.clamp(f_low, f_high)
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute per-subcarrier variance across the sliding window of `frame_history`.
|
||||||
|
///
|
||||||
|
/// For each subcarrier index `k`, returns `Var[A_k]` over all stored frames.
|
||||||
|
/// This captures spatial signal variation; subcarriers whose amplitude fluctuates
|
||||||
|
/// heavily across time correspond to directions with motion.
|
||||||
|
fn compute_subcarrier_variances(frame_history: &VecDeque<Vec<f64>>, n_sub: usize) -> Vec<f64> {
|
||||||
|
if frame_history.is_empty() || n_sub == 0 {
|
||||||
|
return vec![0.0; n_sub];
|
||||||
|
}
|
||||||
|
|
||||||
|
let n_frames = frame_history.len() as f64;
|
||||||
|
let mut means = vec![0.0f64; n_sub];
|
||||||
|
let mut sq_means = vec![0.0f64; n_sub];
|
||||||
|
|
||||||
|
for frame in frame_history.iter() {
|
||||||
|
for k in 0..n_sub {
|
||||||
|
let a = if k < frame.len() { frame[k] } else { 0.0 };
|
||||||
|
means[k] += a;
|
||||||
|
sq_means[k] += a * a;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
(0..n_sub)
|
||||||
|
.map(|k| {
|
||||||
|
let mean = means[k] / n_frames;
|
||||||
|
let sq_mean = sq_means[k] / n_frames;
|
||||||
|
(sq_mean - mean * mean).max(0.0)
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract features from the current ESP32 frame, enhanced with temporal context from
|
||||||
|
/// `frame_history`.
|
||||||
|
///
|
||||||
|
/// Improvements over the previous single-frame approach:
|
||||||
|
///
|
||||||
|
/// - **Variance**: computed as the mean of per-subcarrier temporal variance across the
|
||||||
|
/// sliding window, not just the intra-frame spatial variance.
|
||||||
|
/// - **Motion detection**: uses frame-to-frame temporal difference (mean L2 change
|
||||||
|
/// between the current frame and the previous frame) normalised by signal amplitude,
|
||||||
|
/// so that actual changes are detected rather than just a threshold on the current frame.
|
||||||
|
/// - **Breathing rate**: estimated via Goertzel filter bank on the 0.1–0.5 Hz band of
|
||||||
|
/// the amplitude time series.
|
||||||
|
/// - **Signal quality**: based on SNR estimate (RSSI – noise floor) and subcarrier
|
||||||
|
/// variance stability.
|
||||||
|
fn extract_features_from_frame(
|
||||||
|
frame: &Esp32Frame,
|
||||||
|
frame_history: &VecDeque<Vec<f64>>,
|
||||||
|
sample_rate_hz: f64,
|
||||||
|
) -> (FeatureInfo, ClassificationInfo, f64, Vec<f64>) {
|
||||||
|
let n_sub = frame.amplitudes.len().max(1);
|
||||||
|
let n = n_sub as f64;
|
||||||
let mean_amp: f64 = frame.amplitudes.iter().sum::<f64>() / n;
|
let mean_amp: f64 = frame.amplitudes.iter().sum::<f64>() / n;
|
||||||
let mean_rssi = frame.rssi as f64;
|
let mean_rssi = frame.rssi as f64;
|
||||||
|
|
||||||
let variance: f64 = frame.amplitudes.iter()
|
// ── Intra-frame subcarrier variance (spatial spread across subcarriers) ──
|
||||||
|
let intra_variance: f64 = frame.amplitudes.iter()
|
||||||
.map(|a| (a - mean_amp).powi(2))
|
.map(|a| (a - mean_amp).powi(2))
|
||||||
.sum::<f64>() / n;
|
.sum::<f64>() / n;
|
||||||
|
|
||||||
// Simple spectral analysis on amplitude vector
|
// ── Temporal (sliding-window) per-subcarrier variance ──
|
||||||
let spectral_power: f64 = frame.amplitudes.iter()
|
let sub_variances = compute_subcarrier_variances(frame_history, n_sub);
|
||||||
.map(|a| a * a)
|
let temporal_variance: f64 = if sub_variances.is_empty() {
|
||||||
.sum::<f64>() / n;
|
intra_variance
|
||||||
|
} else {
|
||||||
|
sub_variances.iter().sum::<f64>() / sub_variances.len() as f64
|
||||||
|
};
|
||||||
|
|
||||||
// Motion band: high-frequency subcarrier variance
|
// Use the larger of intra-frame and temporal variance as the reported variance.
|
||||||
|
let variance = intra_variance.max(temporal_variance);
|
||||||
|
|
||||||
|
// ── Spectral power ──
|
||||||
|
let spectral_power: f64 = frame.amplitudes.iter().map(|a| a * a).sum::<f64>() / n;
|
||||||
|
|
||||||
|
// ── Motion band power (upper half of subcarriers, high spatial frequency) ──
|
||||||
let half = frame.amplitudes.len() / 2;
|
let half = frame.amplitudes.len() / 2;
|
||||||
let motion_band_power = if half > 0 {
|
let motion_band_power = if half > 0 {
|
||||||
frame.amplitudes[half..].iter()
|
frame.amplitudes[half..].iter()
|
||||||
|
|
@ -412,7 +615,7 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||||
0.0
|
0.0
|
||||||
};
|
};
|
||||||
|
|
||||||
// Breathing band: low-frequency variance
|
// ── Breathing band power (lower half of subcarriers, low spatial frequency) ──
|
||||||
let breathing_band_power = if half > 0 {
|
let breathing_band_power = if half > 0 {
|
||||||
frame.amplitudes[..half].iter()
|
frame.amplitudes[..half].iter()
|
||||||
.map(|a| (a - mean_amp).powi(2))
|
.map(|a| (a - mean_amp).powi(2))
|
||||||
|
|
@ -421,7 +624,7 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||||
0.0
|
0.0
|
||||||
};
|
};
|
||||||
|
|
||||||
// Dominant frequency estimate (peak subcarrier index → Hz)
|
// ── Dominant frequency via peak subcarrier index ──
|
||||||
let peak_idx = frame.amplitudes.iter()
|
let peak_idx = frame.amplitudes.iter()
|
||||||
.enumerate()
|
.enumerate()
|
||||||
.max_by(|a, b| a.1.partial_cmp(b.1).unwrap_or(std::cmp::Ordering::Equal))
|
.max_by(|a, b| a.1.partial_cmp(b.1).unwrap_or(std::cmp::Ordering::Equal))
|
||||||
|
|
@ -429,12 +632,47 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||||
.unwrap_or(0);
|
.unwrap_or(0);
|
||||||
let dominant_freq_hz = peak_idx as f64 * 0.05;
|
let dominant_freq_hz = peak_idx as f64 * 0.05;
|
||||||
|
|
||||||
// Change point detection (simple threshold crossing count)
|
// ── Change point detection (threshold-crossing count in current frame) ──
|
||||||
let threshold = mean_amp * 1.2;
|
let threshold = mean_amp * 1.2;
|
||||||
let change_points = frame.amplitudes.windows(2)
|
let change_points = frame.amplitudes.windows(2)
|
||||||
.filter(|w| (w[0] < threshold) != (w[1] < threshold))
|
.filter(|w| (w[0] < threshold) != (w[1] < threshold))
|
||||||
.count();
|
.count();
|
||||||
|
|
||||||
|
// ── Motion score: sliding-window temporal difference ──
|
||||||
|
// Compare current frame against the most recent historical frame.
|
||||||
|
// The difference is normalised by the mean amplitude to be scale-invariant.
|
||||||
|
let temporal_motion_score = if let Some(prev_frame) = frame_history.back() {
|
||||||
|
let n_cmp = n_sub.min(prev_frame.len());
|
||||||
|
if n_cmp > 0 {
|
||||||
|
let diff_energy: f64 = (0..n_cmp)
|
||||||
|
.map(|k| (frame.amplitudes[k] - prev_frame[k]).powi(2))
|
||||||
|
.sum::<f64>() / n_cmp as f64;
|
||||||
|
// Normalise by mean squared amplitude to get a dimensionless ratio.
|
||||||
|
let ref_energy = mean_amp * mean_amp + 1e-9;
|
||||||
|
(diff_energy / ref_energy).sqrt().clamp(0.0, 1.0)
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// No history yet — fall back to intra-frame variance-based estimate.
|
||||||
|
(intra_variance / (mean_amp * mean_amp + 1e-9)).sqrt().clamp(0.0, 1.0)
|
||||||
|
};
|
||||||
|
|
||||||
|
// Blend temporal motion with variance-based motion for robustness.
|
||||||
|
let variance_motion = (temporal_variance / 10.0).clamp(0.0, 1.0);
|
||||||
|
let motion_score = (temporal_motion_score * 0.7 + variance_motion * 0.3).clamp(0.0, 1.0);
|
||||||
|
|
||||||
|
// ── Signal quality metric ──
|
||||||
|
// Based on estimated SNR (RSSI relative to noise floor) and subcarrier consistency.
|
||||||
|
let snr_db = (frame.rssi as f64 - frame.noise_floor as f64).max(0.0);
|
||||||
|
let snr_quality = (snr_db / 40.0).clamp(0.0, 1.0); // 40 dB → quality = 1.0
|
||||||
|
// Penalise quality when temporal variance is very high (unstable signal).
|
||||||
|
let stability = (1.0 - (temporal_variance / (mean_amp * mean_amp + 1e-9)).clamp(0.0, 1.0)).max(0.0);
|
||||||
|
let signal_quality = (snr_quality * 0.6 + stability * 0.4).clamp(0.0, 1.0);
|
||||||
|
|
||||||
|
// ── Breathing rate estimation ──
|
||||||
|
let breathing_rate_hz = estimate_breathing_rate_hz(frame_history, sample_rate_hz);
|
||||||
|
|
||||||
let features = FeatureInfo {
|
let features = FeatureInfo {
|
||||||
mean_rssi,
|
mean_rssi,
|
||||||
variance,
|
variance,
|
||||||
|
|
@ -445,23 +683,24 @@ fn extract_features_from_frame(frame: &Esp32Frame) -> (FeatureInfo, Classificati
|
||||||
spectral_power,
|
spectral_power,
|
||||||
};
|
};
|
||||||
|
|
||||||
// Classification
|
// ── Classification ──
|
||||||
let motion_score = (variance / 10.0).clamp(0.0, 1.0);
|
let (motion_level, presence) = if motion_score > 0.4 {
|
||||||
let (motion_level, presence) = if motion_score > 0.5 {
|
|
||||||
("active".to_string(), true)
|
("active".to_string(), true)
|
||||||
} else if motion_score > 0.1 {
|
} else if motion_score > 0.08 {
|
||||||
("present_still".to_string(), true)
|
("present_still".to_string(), true)
|
||||||
} else {
|
} else {
|
||||||
("absent".to_string(), false)
|
("absent".to_string(), false)
|
||||||
};
|
};
|
||||||
|
|
||||||
|
let confidence = (0.4 + signal_quality * 0.3 + motion_score * 0.3).clamp(0.0, 1.0);
|
||||||
|
|
||||||
let classification = ClassificationInfo {
|
let classification = ClassificationInfo {
|
||||||
motion_level,
|
motion_level,
|
||||||
presence,
|
presence,
|
||||||
confidence: 0.5 + motion_score * 0.5,
|
confidence,
|
||||||
};
|
};
|
||||||
|
|
||||||
(features, classification)
|
(features, classification, breathing_rate_hz, sub_variances)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Windows WiFi RSSI collector ──────────────────────────────────────────────
|
// ── Windows WiFi RSSI collector ──────────────────────────────────────────────
|
||||||
|
|
@ -596,7 +835,16 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||||
phases: multi_ap_frame.phases.clone(),
|
phases: multi_ap_frame.phases.clone(),
|
||||||
};
|
};
|
||||||
|
|
||||||
let (features, classification) = extract_features_from_frame(&frame);
|
// ── Step 4b: Update frame history and extract features ───────
|
||||||
|
let mut s_write_pre = state.write().await;
|
||||||
|
s_write_pre.frame_history.push_back(frame.amplitudes.clone());
|
||||||
|
if s_write_pre.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||||
|
s_write_pre.frame_history.pop_front();
|
||||||
|
}
|
||||||
|
let sample_rate_hz = 1000.0 / tick_ms as f64;
|
||||||
|
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||||
|
extract_features_from_frame(&frame, &s_write_pre.frame_history, sample_rate_hz);
|
||||||
|
drop(s_write_pre);
|
||||||
|
|
||||||
// ── Step 5: Build enhanced fields from pipeline result ───────
|
// ── Step 5: Build enhanced fields from pipeline result ───────
|
||||||
let enhanced_motion = Some(serde_json::json!({
|
let enhanced_motion = Some(serde_json::json!({
|
||||||
|
|
@ -640,6 +888,7 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||||
let vitals = s.vital_detector.process_frame(&frame.amplitudes, &frame.phases);
|
let vitals = s.vital_detector.process_frame(&frame.amplitudes, &frame.phases);
|
||||||
s.latest_vitals = vitals.clone();
|
s.latest_vitals = vitals.clone();
|
||||||
|
|
||||||
|
let feat_variance = features.variance;
|
||||||
let update = SensingUpdate {
|
let update = SensingUpdate {
|
||||||
msg_type: "sensing_update".to_string(),
|
msg_type: "sensing_update".to_string(),
|
||||||
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
||||||
|
|
@ -654,7 +903,10 @@ async fn windows_wifi_task(state: SharedState, tick_ms: u64) {
|
||||||
}],
|
}],
|
||||||
features,
|
features,
|
||||||
classification,
|
classification,
|
||||||
signal_field: generate_signal_field(first_rssi, 1.0, motion_score, tick),
|
signal_field: generate_signal_field(
|
||||||
|
first_rssi, motion_score, breathing_rate_hz,
|
||||||
|
feat_variance.min(1.0), &sub_variances,
|
||||||
|
),
|
||||||
vital_signs: Some(vitals),
|
vital_signs: Some(vitals),
|
||||||
enhanced_motion,
|
enhanced_motion,
|
||||||
enhanced_breathing,
|
enhanced_breathing,
|
||||||
|
|
@ -715,9 +967,16 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||||
phases: vec![0.0],
|
phases: vec![0.0],
|
||||||
};
|
};
|
||||||
|
|
||||||
let (features, classification) = extract_features_from_frame(&frame);
|
|
||||||
|
|
||||||
let mut s = state.write().await;
|
let mut s = state.write().await;
|
||||||
|
// Update frame history before extracting features.
|
||||||
|
s.frame_history.push_back(frame.amplitudes.clone());
|
||||||
|
if s.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||||
|
s.frame_history.pop_front();
|
||||||
|
}
|
||||||
|
let sample_rate_hz = 2.0_f64; // fallback tick ~ 500 ms => 2 Hz
|
||||||
|
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||||
|
extract_features_from_frame(&frame, &s.frame_history, sample_rate_hz);
|
||||||
|
|
||||||
s.source = format!("wifi:{ssid}");
|
s.source = format!("wifi:{ssid}");
|
||||||
s.rssi_history.push_back(rssi_dbm);
|
s.rssi_history.push_back(rssi_dbm);
|
||||||
if s.rssi_history.len() > 60 {
|
if s.rssi_history.len() > 60 {
|
||||||
|
|
@ -738,6 +997,7 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||||
let vitals = s.vital_detector.process_frame(&frame.amplitudes, &frame.phases);
|
let vitals = s.vital_detector.process_frame(&frame.amplitudes, &frame.phases);
|
||||||
s.latest_vitals = vitals.clone();
|
s.latest_vitals = vitals.clone();
|
||||||
|
|
||||||
|
let feat_variance = features.variance;
|
||||||
let update = SensingUpdate {
|
let update = SensingUpdate {
|
||||||
msg_type: "sensing_update".to_string(),
|
msg_type: "sensing_update".to_string(),
|
||||||
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
||||||
|
|
@ -752,7 +1012,10 @@ async fn windows_wifi_fallback_tick(state: &SharedState, seq: u32) {
|
||||||
}],
|
}],
|
||||||
features,
|
features,
|
||||||
classification,
|
classification,
|
||||||
signal_field: generate_signal_field(rssi_dbm, 1.0, motion_score, tick),
|
signal_field: generate_signal_field(
|
||||||
|
rssi_dbm, motion_score, breathing_rate_hz,
|
||||||
|
feat_variance.min(1.0), &sub_variances,
|
||||||
|
),
|
||||||
vital_signs: Some(vitals),
|
vital_signs: Some(vitals),
|
||||||
enhanced_motion: None,
|
enhanced_motion: None,
|
||||||
enhanced_breathing: None,
|
enhanced_breathing: None,
|
||||||
|
|
@ -902,7 +1165,47 @@ async fn handle_ws_pose_client(mut socket: WebSocket, state: SharedState) {
|
||||||
// Parse the sensing update and convert to pose format
|
// Parse the sensing update and convert to pose format
|
||||||
if let Ok(sensing) = serde_json::from_str::<SensingUpdate>(&json) {
|
if let Ok(sensing) = serde_json::from_str::<SensingUpdate>(&json) {
|
||||||
if sensing.msg_type == "sensing_update" {
|
if sensing.msg_type == "sensing_update" {
|
||||||
let persons = derive_pose_from_sensing(&sensing);
|
// Determine pose estimation mode for the UI indicator.
|
||||||
|
// "model_inference" — a trained RVF model is loaded.
|
||||||
|
// "signal_derived" — keypoints estimated from raw CSI features.
|
||||||
|
let model_loaded = {
|
||||||
|
let s = state.read().await;
|
||||||
|
s.model_loaded
|
||||||
|
};
|
||||||
|
let pose_source = if model_loaded {
|
||||||
|
"model_inference"
|
||||||
|
} else {
|
||||||
|
"signal_derived"
|
||||||
|
};
|
||||||
|
|
||||||
|
let persons = if model_loaded {
|
||||||
|
// When a trained model is loaded, prefer its keypoints if present.
|
||||||
|
sensing.pose_keypoints.as_ref().map(|kps| {
|
||||||
|
let kp_names = [
|
||||||
|
"nose","left_eye","right_eye","left_ear","right_ear",
|
||||||
|
"left_shoulder","right_shoulder","left_elbow","right_elbow",
|
||||||
|
"left_wrist","right_wrist","left_hip","right_hip",
|
||||||
|
"left_knee","right_knee","left_ankle","right_ankle",
|
||||||
|
];
|
||||||
|
let keypoints: Vec<PoseKeypoint> = kps.iter()
|
||||||
|
.enumerate()
|
||||||
|
.map(|(i, kp)| PoseKeypoint {
|
||||||
|
name: kp_names.get(i).unwrap_or(&"unknown").to_string(),
|
||||||
|
x: kp[0], y: kp[1], z: kp[2], confidence: kp[3],
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
vec![PersonDetection {
|
||||||
|
id: 1,
|
||||||
|
confidence: sensing.classification.confidence,
|
||||||
|
bbox: BoundingBox { x: 260.0, y: 150.0, width: 120.0, height: 220.0 },
|
||||||
|
keypoints,
|
||||||
|
zone: "zone_1".into(),
|
||||||
|
}]
|
||||||
|
}).unwrap_or_else(|| derive_pose_from_sensing(&sensing))
|
||||||
|
} else {
|
||||||
|
derive_pose_from_sensing(&sensing)
|
||||||
|
};
|
||||||
|
|
||||||
let pose_msg = serde_json::json!({
|
let pose_msg = serde_json::json!({
|
||||||
"type": "pose_data",
|
"type": "pose_data",
|
||||||
"zone_id": "zone_1",
|
"zone_id": "zone_1",
|
||||||
|
|
@ -913,12 +1216,16 @@ async fn handle_ws_pose_client(mut socket: WebSocket, state: SharedState) {
|
||||||
},
|
},
|
||||||
"confidence": if sensing.classification.presence { sensing.classification.confidence } else { 0.0 },
|
"confidence": if sensing.classification.presence { sensing.classification.confidence } else { 0.0 },
|
||||||
"activity": sensing.classification.motion_level,
|
"activity": sensing.classification.motion_level,
|
||||||
|
// pose_source tells the UI which estimation mode is active.
|
||||||
|
"pose_source": pose_source,
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"frame_id": format!("rust_frame_{}", sensing.tick),
|
"frame_id": format!("rust_frame_{}", sensing.tick),
|
||||||
"processing_time_ms": 1,
|
"processing_time_ms": 1,
|
||||||
"source": sensing.source,
|
"source": sensing.source,
|
||||||
"tick": sensing.tick,
|
"tick": sensing.tick,
|
||||||
"signal_strength": sensing.features.mean_rssi,
|
"signal_strength": sensing.features.mean_rssi,
|
||||||
|
"motion_band_power": sensing.features.motion_band_power,
|
||||||
|
"breathing_band_power": sensing.features.breathing_band_power,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
@ -972,21 +1279,86 @@ async fn latest(State(state): State<SharedState>) -> Json<serde_json::Value> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Generate WiFi-derived pose keypoints from sensing data
|
/// Generate WiFi-derived pose keypoints from sensing data.
|
||||||
|
///
|
||||||
|
/// Keypoint positions are modulated by real signal features rather than a pure
|
||||||
|
/// time-based sine/cosine loop:
|
||||||
|
///
|
||||||
|
/// - `motion_band_power` drives whole-body translation and limb splay
|
||||||
|
/// - `variance` seeds per-frame noise so the skeleton never freezes
|
||||||
|
/// - `breathing_band_power` expands/contracts torso keypoints (shoulders, hips)
|
||||||
|
/// - `dominant_freq_hz` tilts the upper body laterally (lean direction)
|
||||||
|
/// - `change_points` adds burst jitter to extremities (wrists, ankles)
|
||||||
|
///
|
||||||
|
/// When `presence == false` no persons are returned (empty room).
|
||||||
|
/// When walking is detected (`motion_score > 0.55`) the figure shifts laterally
|
||||||
|
/// with a stride-swing pattern applied to arms and legs.
|
||||||
fn derive_pose_from_sensing(update: &SensingUpdate) -> Vec<PersonDetection> {
|
fn derive_pose_from_sensing(update: &SensingUpdate) -> Vec<PersonDetection> {
|
||||||
let cls = &update.classification;
|
let cls = &update.classification;
|
||||||
if !cls.presence {
|
if !cls.presence {
|
||||||
return vec![];
|
return vec![];
|
||||||
}
|
}
|
||||||
|
|
||||||
let t = update.tick as f64 * 0.05;
|
let feat = &update.features;
|
||||||
let motion = if cls.motion_level == "active" { 1.0 }
|
|
||||||
else if cls.motion_level == "present_still" { 0.3 }
|
|
||||||
else { 0.0 };
|
|
||||||
|
|
||||||
// COCO 17-keypoint skeleton, positions derived from signal field
|
// ── Signal-derived scalars ────────────────────────────────────────────────
|
||||||
let base_x = 320.0 + 30.0 * t.sin() * motion;
|
|
||||||
let base_y = 240.0 + 15.0 * (t * 0.7).cos() * motion;
|
// Continuous motion score from motion_band_power (0..1).
|
||||||
|
// motion_band_power is the high-frequency subcarrier variance — it is high
|
||||||
|
// when a body is actively moving through the RF field.
|
||||||
|
let motion_score = (feat.motion_band_power / 15.0).clamp(0.0, 1.0);
|
||||||
|
let is_walking = motion_score > 0.55;
|
||||||
|
|
||||||
|
// Breathing expansion: torso keypoints shift ±breath_amp pixels per cycle.
|
||||||
|
// breathing_band_power comes from low-frequency subcarrier variance.
|
||||||
|
let breath_amp = (feat.breathing_band_power * 4.0).clamp(0.0, 12.0);
|
||||||
|
|
||||||
|
// Breathing phase: use the vital-sign estimate if available, otherwise
|
||||||
|
// derive a proxy from breathing_band_power and the tick counter.
|
||||||
|
let breath_phase = if let Some(ref vs) = update.vital_signs {
|
||||||
|
// breathing_rate_bpm is Option<f64>; fall back to 15 BPM if not yet estimated.
|
||||||
|
// 15 BPM -> 0.25 Hz, which sits comfortably in the breathing band.
|
||||||
|
let bpm = vs.breathing_rate_bpm.unwrap_or(15.0);
|
||||||
|
let freq = (bpm / 60.0).clamp(0.1, 0.5);
|
||||||
|
(update.tick as f64 * freq * 0.1 * std::f64::consts::TAU).sin()
|
||||||
|
} else {
|
||||||
|
(update.tick as f64 * 0.08 + feat.breathing_band_power).sin()
|
||||||
|
};
|
||||||
|
|
||||||
|
// Lateral lean derived from dominant_freq_hz (peak subcarrier index -> Hz).
|
||||||
|
// Maps 0..10 Hz range to ±18 px horizontal shift of the torso center.
|
||||||
|
let lean_x = (feat.dominant_freq_hz / 5.0 - 1.0).clamp(-1.0, 1.0) * 18.0;
|
||||||
|
|
||||||
|
// Walking stride: lateral body displacement oscillating with motion_band_power.
|
||||||
|
// Amplitude is zero when the person is stationary.
|
||||||
|
let stride_x = if is_walking {
|
||||||
|
let stride_phase = (feat.motion_band_power * 0.7 + update.tick as f64 * 0.12).sin();
|
||||||
|
stride_phase * 45.0 * motion_score
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
|
||||||
|
// Burst jitter from change_points: rapid threshold crossings in the
|
||||||
|
// amplitude vector indicate fast movement or sudden signal disturbance.
|
||||||
|
let burst = (feat.change_points as f64 / 8.0).clamp(0.0, 1.0);
|
||||||
|
|
||||||
|
// Deterministic per-frame noise seeded by variance and tick.
|
||||||
|
// Uses the fractional part of a large sine to get a tick-dependent value
|
||||||
|
// in (-1, 1) without needing a PRNG.
|
||||||
|
let noise_seed = feat.variance * 31.7 + update.tick as f64 * 17.3;
|
||||||
|
let noise_val = (noise_seed.sin() * 43758.545).fract();
|
||||||
|
|
||||||
|
// Scale base confidence by SNR proxy (high variance = better signal quality).
|
||||||
|
let snr_factor = ((feat.variance - 0.5) / 10.0).clamp(0.0, 1.0);
|
||||||
|
let base_confidence = cls.confidence * (0.6 + 0.4 * snr_factor);
|
||||||
|
|
||||||
|
// ── Skeleton base position ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// Center figure on a 640x480 canvas.
|
||||||
|
let base_x = 320.0 + stride_x + lean_x * 0.5;
|
||||||
|
let base_y = 240.0 - motion_score * 8.0;
|
||||||
|
|
||||||
|
// ── COCO 17-keypoint offsets from hip-center ──────────────────────────────
|
||||||
|
|
||||||
let kp_names = [
|
let kp_names = [
|
||||||
"nose", "left_eye", "right_eye", "left_ear", "right_ear",
|
"nose", "left_eye", "right_eye", "left_ear", "right_ear",
|
||||||
|
|
@ -994,49 +1366,130 @@ fn derive_pose_from_sensing(update: &SensingUpdate) -> Vec<PersonDetection> {
|
||||||
"left_wrist", "right_wrist", "left_hip", "right_hip",
|
"left_wrist", "right_wrist", "left_hip", "right_hip",
|
||||||
"left_knee", "right_knee", "left_ankle", "right_ankle",
|
"left_knee", "right_knee", "left_ankle", "right_ankle",
|
||||||
];
|
];
|
||||||
|
|
||||||
|
// Nominal (dx, dy) offsets from hip-center in pixels.
|
||||||
let kp_offsets: [(f64, f64); 17] = [
|
let kp_offsets: [(f64, f64); 17] = [
|
||||||
(0.0, -80.0), // nose
|
( 0.0, -80.0), // 0 nose
|
||||||
(-8.0, -88.0), // left_eye
|
( -8.0, -88.0), // 1 left_eye
|
||||||
(8.0, -88.0), // right_eye
|
( 8.0, -88.0), // 2 right_eye
|
||||||
(-16.0, -82.0), // left_ear
|
(-16.0, -82.0), // 3 left_ear
|
||||||
(16.0, -82.0), // right_ear
|
( 16.0, -82.0), // 4 right_ear
|
||||||
(-30.0, -50.0), // left_shoulder
|
(-30.0, -50.0), // 5 left_shoulder
|
||||||
(30.0, -50.0), // right_shoulder
|
( 30.0, -50.0), // 6 right_shoulder
|
||||||
(-45.0, -15.0), // left_elbow
|
(-45.0, -15.0), // 7 left_elbow
|
||||||
(45.0, -15.0), // right_elbow
|
( 45.0, -15.0), // 8 right_elbow
|
||||||
(-50.0, 20.0), // left_wrist
|
(-50.0, 20.0), // 9 left_wrist
|
||||||
(50.0, 20.0), // right_wrist
|
( 50.0, 20.0), // 10 right_wrist
|
||||||
(-20.0, 20.0), // left_hip
|
(-20.0, 20.0), // 11 left_hip
|
||||||
(20.0, 20.0), // right_hip
|
( 20.0, 20.0), // 12 right_hip
|
||||||
(-22.0, 70.0), // left_knee
|
(-22.0, 70.0), // 13 left_knee
|
||||||
(22.0, 70.0), // right_knee
|
( 22.0, 70.0), // 14 right_knee
|
||||||
(-24.0, 120.0), // left_ankle
|
(-24.0, 120.0), // 15 left_ankle
|
||||||
(24.0, 120.0), // right_ankle
|
( 24.0, 120.0), // 16 right_ankle
|
||||||
];
|
];
|
||||||
|
|
||||||
|
// Torso keypoints: left_shoulder(5), right_shoulder(6), left_hip(11), right_hip(12).
|
||||||
|
// These respond to the breathing expansion signal.
|
||||||
|
const TORSO_KP: [usize; 4] = [5, 6, 11, 12];
|
||||||
|
|
||||||
|
// Extremity keypoints: left_wrist(9), right_wrist(10), left_ankle(15), right_ankle(16).
|
||||||
|
// These pick up burst jitter from high change_points counts.
|
||||||
|
const EXTREMITY_KP: [usize; 4] = [9, 10, 15, 16];
|
||||||
|
|
||||||
let keypoints: Vec<PoseKeypoint> = kp_names.iter().zip(kp_offsets.iter())
|
let keypoints: Vec<PoseKeypoint> = kp_names.iter().zip(kp_offsets.iter())
|
||||||
.enumerate()
|
.enumerate()
|
||||||
.map(|(i, (name, (dx, dy)))| {
|
.map(|(i, (name, (dx, dy)))| {
|
||||||
let jitter = motion * 3.0 * (t * 2.0 + i as f64).sin();
|
// ── Breathing expansion (torso only) ─────────────────────────
|
||||||
|
let breath_dx = if TORSO_KP.contains(&i) {
|
||||||
|
// Shoulders spread outward; hips compress inward on inhale.
|
||||||
|
let sign = if *dx < 0.0 { -1.0 } else { 1.0 };
|
||||||
|
sign * breath_amp * breath_phase * 0.5
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
let breath_dy = if TORSO_KP.contains(&i) {
|
||||||
|
// Shoulders rise slightly; hips descend slightly on inhale.
|
||||||
|
let sign = if *dy < 0.0 { -1.0 } else { 1.0 };
|
||||||
|
sign * breath_amp * breath_phase * 0.3
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
|
||||||
|
// ── Extremity burst jitter ────────────────────────────────────
|
||||||
|
let extremity_jitter = if EXTREMITY_KP.contains(&i) {
|
||||||
|
// Each extremity gets an independent phase offset.
|
||||||
|
let phase = noise_seed + i as f64 * 2.399; // golden-angle spacing
|
||||||
|
(
|
||||||
|
phase.sin() * burst * motion_score * 12.0,
|
||||||
|
(phase * 1.31).cos() * burst * motion_score * 8.0,
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
(0.0, 0.0)
|
||||||
|
};
|
||||||
|
|
||||||
|
// ── Per-joint motion noise (scales with signal variance) ──────
|
||||||
|
// Different seed per keypoint so every joint moves independently.
|
||||||
|
let kp_noise_x = ((noise_seed + i as f64 * 1.618).sin() * 43758.545).fract()
|
||||||
|
* feat.variance.sqrt().clamp(0.0, 3.0) * motion_score;
|
||||||
|
let kp_noise_y = ((noise_seed + i as f64 * 2.718).cos() * 31415.926).fract()
|
||||||
|
* feat.variance.sqrt().clamp(0.0, 3.0) * motion_score * 0.6;
|
||||||
|
|
||||||
|
// ── Walking arm/leg swing (contralateral gait pattern) ────────
|
||||||
|
let swing_dy = if is_walking {
|
||||||
|
let stride_phase =
|
||||||
|
(feat.motion_band_power * 0.7 + update.tick as f64 * 0.12).sin();
|
||||||
|
match i {
|
||||||
|
7 | 9 => -stride_phase * 20.0 * motion_score, // left elbow/wrist
|
||||||
|
8 | 10 => stride_phase * 20.0 * motion_score, // right elbow/wrist
|
||||||
|
13 | 15 => stride_phase * 25.0 * motion_score, // left knee/ankle
|
||||||
|
14 | 16 => -stride_phase * 25.0 * motion_score, // right knee/ankle
|
||||||
|
_ => 0.0,
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
|
||||||
|
// ── Compose final position ────────────────────────────────────
|
||||||
|
let final_x =
|
||||||
|
base_x + dx + breath_dx + extremity_jitter.0 + kp_noise_x;
|
||||||
|
let final_y =
|
||||||
|
base_y + dy + breath_dy + extremity_jitter.1 + kp_noise_y + swing_dy;
|
||||||
|
|
||||||
|
// Extremity confidence is lower when signal variance is low.
|
||||||
|
let kp_conf = if EXTREMITY_KP.contains(&i) {
|
||||||
|
base_confidence * (0.7 + 0.3 * snr_factor) * (0.85 + 0.15 * noise_val)
|
||||||
|
} else {
|
||||||
|
base_confidence
|
||||||
|
* (0.88 + 0.12 * ((i as f64 * 0.7 + noise_seed).cos()))
|
||||||
|
};
|
||||||
|
|
||||||
PoseKeypoint {
|
PoseKeypoint {
|
||||||
name: name.to_string(),
|
name: name.to_string(),
|
||||||
x: base_x + dx + jitter,
|
x: final_x,
|
||||||
y: base_y + dy + jitter * 0.5,
|
y: final_y,
|
||||||
z: 0.0,
|
z: lean_x * 0.02, // slight Z depth from lean direction
|
||||||
confidence: cls.confidence * (0.85 + 0.15 * (i as f64 * 0.3).cos()),
|
confidence: kp_conf.clamp(0.1, 1.0),
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
|
// Bounding box derived from actual keypoint extents with padding.
|
||||||
|
let xs: Vec<f64> = keypoints.iter().map(|k| k.x).collect();
|
||||||
|
let ys: Vec<f64> = keypoints.iter().map(|k| k.y).collect();
|
||||||
|
let min_x = xs.iter().cloned().fold(f64::MAX, f64::min) - 10.0;
|
||||||
|
let min_y = ys.iter().cloned().fold(f64::MAX, f64::min) - 10.0;
|
||||||
|
let max_x = xs.iter().cloned().fold(f64::MIN, f64::max) + 10.0;
|
||||||
|
let max_y = ys.iter().cloned().fold(f64::MIN, f64::max) + 10.0;
|
||||||
|
|
||||||
vec![PersonDetection {
|
vec![PersonDetection {
|
||||||
id: 1,
|
id: 1,
|
||||||
confidence: cls.confidence,
|
confidence: cls.confidence,
|
||||||
keypoints,
|
keypoints,
|
||||||
bbox: BoundingBox {
|
bbox: BoundingBox {
|
||||||
x: base_x - 60.0,
|
x: min_x,
|
||||||
y: base_y - 90.0,
|
y: min_y,
|
||||||
width: 120.0,
|
width: (max_x - min_x).max(80.0),
|
||||||
height: 220.0,
|
height: (max_y - min_y).max(160.0),
|
||||||
},
|
},
|
||||||
zone: "zone_1".into(),
|
zone: "zone_1".into(),
|
||||||
}]
|
}]
|
||||||
|
|
@ -1161,7 +1614,7 @@ async fn stream_status(State(state): State<SharedState>) -> Json<serde_json::Val
|
||||||
Json(serde_json::json!({
|
Json(serde_json::json!({
|
||||||
"active": true,
|
"active": true,
|
||||||
"clients": s.tx.receiver_count(),
|
"clients": s.tx.receiver_count(),
|
||||||
"fps": 2,
|
"fps": if s.tick > 1 { 10u64 } else { 0u64 },
|
||||||
"source": s.source,
|
"source": s.source,
|
||||||
}))
|
}))
|
||||||
}
|
}
|
||||||
|
|
@ -1311,10 +1764,20 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||||
debug!("ESP32 frame from {src}: node={}, subs={}, seq={}",
|
debug!("ESP32 frame from {src}: node={}, subs={}, seq={}",
|
||||||
frame.node_id, frame.n_subcarriers, frame.sequence);
|
frame.node_id, frame.n_subcarriers, frame.sequence);
|
||||||
|
|
||||||
let (features, classification) = extract_features_from_frame(&frame);
|
|
||||||
let mut s = state.write().await;
|
let mut s = state.write().await;
|
||||||
s.source = "esp32".to_string();
|
s.source = "esp32".to_string();
|
||||||
|
|
||||||
|
// Append current amplitudes to history before extracting features so
|
||||||
|
// that temporal analysis includes the most recent frame.
|
||||||
|
s.frame_history.push_back(frame.amplitudes.clone());
|
||||||
|
if s.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||||
|
s.frame_history.pop_front();
|
||||||
|
}
|
||||||
|
|
||||||
|
let sample_rate_hz = 1000.0 / 500.0_f64; // default tick; ESP32 frames arrive as fast as they come
|
||||||
|
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||||
|
extract_features_from_frame(&frame, &s.frame_history, sample_rate_hz);
|
||||||
|
|
||||||
// Update RSSI history
|
// Update RSSI history
|
||||||
s.rssi_history.push_back(features.mean_rssi);
|
s.rssi_history.push_back(features.mean_rssi);
|
||||||
if s.rssi_history.len() > 60 {
|
if s.rssi_history.len() > 60 {
|
||||||
|
|
@ -1349,7 +1812,8 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||||
features: features.clone(),
|
features: features.clone(),
|
||||||
classification,
|
classification,
|
||||||
signal_field: generate_signal_field(
|
signal_field: generate_signal_field(
|
||||||
features.mean_rssi, features.variance, motion_score, tick,
|
features.mean_rssi, motion_score, breathing_rate_hz,
|
||||||
|
features.variance.min(1.0), &sub_variances,
|
||||||
),
|
),
|
||||||
vital_signs: Some(vitals),
|
vital_signs: Some(vitals),
|
||||||
enhanced_motion: None,
|
enhanced_motion: None,
|
||||||
|
|
@ -1390,7 +1854,16 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||||
let tick = s.tick;
|
let tick = s.tick;
|
||||||
|
|
||||||
let frame = generate_simulated_frame(tick);
|
let frame = generate_simulated_frame(tick);
|
||||||
let (features, classification) = extract_features_from_frame(&frame);
|
|
||||||
|
// Append current amplitudes to history before feature extraction.
|
||||||
|
s.frame_history.push_back(frame.amplitudes.clone());
|
||||||
|
if s.frame_history.len() > FRAME_HISTORY_CAPACITY {
|
||||||
|
s.frame_history.pop_front();
|
||||||
|
}
|
||||||
|
|
||||||
|
let sample_rate_hz = 1000.0 / tick_ms as f64;
|
||||||
|
let (features, classification, breathing_rate_hz, sub_variances) =
|
||||||
|
extract_features_from_frame(&frame, &s.frame_history, sample_rate_hz);
|
||||||
|
|
||||||
s.rssi_history.push_back(features.mean_rssi);
|
s.rssi_history.push_back(features.mean_rssi);
|
||||||
if s.rssi_history.len() > 60 {
|
if s.rssi_history.len() > 60 {
|
||||||
|
|
@ -1407,6 +1880,9 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||||
);
|
);
|
||||||
s.latest_vitals = vitals.clone();
|
s.latest_vitals = vitals.clone();
|
||||||
|
|
||||||
|
let frame_amplitudes = frame.amplitudes.clone();
|
||||||
|
let frame_n_sub = frame.n_subcarriers;
|
||||||
|
|
||||||
let update = SensingUpdate {
|
let update = SensingUpdate {
|
||||||
msg_type: "sensing_update".to_string(),
|
msg_type: "sensing_update".to_string(),
|
||||||
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
timestamp: chrono::Utc::now().timestamp_millis() as f64 / 1000.0,
|
||||||
|
|
@ -1416,13 +1892,14 @@ async fn simulated_data_task(state: SharedState, tick_ms: u64) {
|
||||||
node_id: 1,
|
node_id: 1,
|
||||||
rssi_dbm: features.mean_rssi,
|
rssi_dbm: features.mean_rssi,
|
||||||
position: [2.0, 0.0, 1.5],
|
position: [2.0, 0.0, 1.5],
|
||||||
amplitude: frame.amplitudes,
|
amplitude: frame_amplitudes,
|
||||||
subcarrier_count: frame.n_subcarriers as usize,
|
subcarrier_count: frame_n_sub as usize,
|
||||||
}],
|
}],
|
||||||
features: features.clone(),
|
features: features.clone(),
|
||||||
classification,
|
classification,
|
||||||
signal_field: generate_signal_field(
|
signal_field: generate_signal_field(
|
||||||
features.mean_rssi, features.variance, motion_score, tick,
|
features.mean_rssi, motion_score, breathing_rate_hz,
|
||||||
|
features.variance.min(1.0), &sub_variances,
|
||||||
),
|
),
|
||||||
vital_signs: Some(vitals),
|
vital_signs: Some(vitals),
|
||||||
enhanced_motion: None,
|
enhanced_motion: None,
|
||||||
|
|
@ -2014,6 +2491,7 @@ async fn main() {
|
||||||
let state: SharedState = Arc::new(RwLock::new(AppStateInner {
|
let state: SharedState = Arc::new(RwLock::new(AppStateInner {
|
||||||
latest_update: None,
|
latest_update: None,
|
||||||
rssi_history: VecDeque::new(),
|
rssi_history: VecDeque::new(),
|
||||||
|
frame_history: VecDeque::new(),
|
||||||
tick: 0,
|
tick: 0,
|
||||||
source: source.into(),
|
source: source.into(),
|
||||||
tx,
|
tx,
|
||||||
|
|
|
||||||
|
|
@ -14,7 +14,9 @@ export class LiveDemoTab {
|
||||||
currentZone: 'zone_1',
|
currentZone: 'zone_1',
|
||||||
debugMode: false,
|
debugMode: false,
|
||||||
autoReconnect: true,
|
autoReconnect: true,
|
||||||
renderMode: 'skeleton'
|
renderMode: 'skeleton',
|
||||||
|
// 'unknown' | 'signal_derived' | 'model_inference'
|
||||||
|
poseSource: 'unknown'
|
||||||
};
|
};
|
||||||
|
|
||||||
this.components = {
|
this.components = {
|
||||||
|
|
@ -136,6 +138,48 @@ export class LiveDemoTab {
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div class="pose-source-panel">
|
||||||
|
<h4>Estimation Mode</h4>
|
||||||
|
<div class="pose-source-indicator" id="pose-source-indicator">
|
||||||
|
<span class="pose-source-badge pose-source-unknown" id="pose-source-badge">Unknown</span>
|
||||||
|
<p class="pose-source-description" id="pose-source-description">
|
||||||
|
Waiting for first frame...
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="setup-guide-panel">
|
||||||
|
<h4>Setup Guide</h4>
|
||||||
|
<div class="setup-levels">
|
||||||
|
<div class="setup-level">
|
||||||
|
<span class="setup-level-icon">1x</span>
|
||||||
|
<div class="setup-level-info">
|
||||||
|
<strong>1 ESP32 + 1 AP</strong>
|
||||||
|
<p>Presence, breathing, gross motion</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="setup-level">
|
||||||
|
<span class="setup-level-icon">3x</span>
|
||||||
|
<div class="setup-level-info">
|
||||||
|
<strong>2-3 ESP32s</strong>
|
||||||
|
<p>Body localization, motion direction</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="setup-level">
|
||||||
|
<span class="setup-level-icon">4x+</span>
|
||||||
|
<div class="setup-level-info">
|
||||||
|
<strong>4+ ESP32s + trained model</strong>
|
||||||
|
<p>Individual limb tracking, full pose</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<p class="setup-note">
|
||||||
|
Signal-Derived mode uses aggregate CSI features.
|
||||||
|
For per-limb tracking, load a trained <code>.rvf</code> model
|
||||||
|
with <code>--model path.rvf</code> and use 4+ sensors.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
<div class="health-panel">
|
<div class="health-panel">
|
||||||
<h4>System Health</h4>
|
<h4>System Health</h4>
|
||||||
<div class="health-check">
|
<div class="health-check">
|
||||||
|
|
@ -432,6 +476,133 @@ export class LiveDemoTab {
|
||||||
.health-good { color: #28a745; }
|
.health-good { color: #28a745; }
|
||||||
.health-poor { color: #ffc107; }
|
.health-poor { color: #ffc107; }
|
||||||
.health-bad { color: #dc3545; }
|
.health-bad { color: #dc3545; }
|
||||||
|
|
||||||
|
/* Pose estimation mode indicator */
|
||||||
|
.pose-source-panel {
|
||||||
|
background: #fff;
|
||||||
|
border: 1px solid #ddd;
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 15px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-panel h4 {
|
||||||
|
margin: 0 0 12px 0;
|
||||||
|
color: #333;
|
||||||
|
font-size: 14px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-indicator {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 8px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-badge {
|
||||||
|
display: inline-block;
|
||||||
|
padding: 4px 12px;
|
||||||
|
border-radius: 12px;
|
||||||
|
font-size: 12px;
|
||||||
|
font-weight: 600;
|
||||||
|
text-transform: uppercase;
|
||||||
|
letter-spacing: 0.5px;
|
||||||
|
width: fit-content;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-unknown {
|
||||||
|
background: #f0f0f0;
|
||||||
|
color: #6c757d;
|
||||||
|
border: 1px solid #dee2e6;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-signal {
|
||||||
|
background: #e8f5e9;
|
||||||
|
color: #2e7d32;
|
||||||
|
border: 1px solid #a5d6a7;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-model {
|
||||||
|
background: #e3f2fd;
|
||||||
|
color: #1565c0;
|
||||||
|
border: 1px solid #90caf9;
|
||||||
|
}
|
||||||
|
|
||||||
|
.pose-source-description {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 11px;
|
||||||
|
color: #666;
|
||||||
|
line-height: 1.4;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-guide-panel {
|
||||||
|
background: #fff;
|
||||||
|
border: 1px solid #ddd;
|
||||||
|
border-radius: 8px;
|
||||||
|
padding: 15px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-guide-panel h4 {
|
||||||
|
margin: 0 0 12px 0;
|
||||||
|
color: #333;
|
||||||
|
font-size: 14px;
|
||||||
|
font-weight: 600;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-levels {
|
||||||
|
display: flex;
|
||||||
|
flex-direction: column;
|
||||||
|
gap: 10px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-level {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
gap: 10px;
|
||||||
|
padding: 8px;
|
||||||
|
border-radius: 6px;
|
||||||
|
background: #f8f9fa;
|
||||||
|
border: 1px solid #e9ecef;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-level-icon {
|
||||||
|
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||||||
|
color: white;
|
||||||
|
font-size: 11px;
|
||||||
|
font-weight: 700;
|
||||||
|
width: 32px;
|
||||||
|
height: 32px;
|
||||||
|
border-radius: 50%;
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
flex-shrink: 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-level-info strong {
|
||||||
|
font-size: 12px;
|
||||||
|
color: #333;
|
||||||
|
display: block;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-level-info p {
|
||||||
|
margin: 2px 0 0;
|
||||||
|
font-size: 11px;
|
||||||
|
color: #666;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-note {
|
||||||
|
margin: 10px 0 0;
|
||||||
|
font-size: 11px;
|
||||||
|
color: #888;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.setup-note code {
|
||||||
|
background: #f0f0f0;
|
||||||
|
padding: 1px 4px;
|
||||||
|
border-radius: 3px;
|
||||||
|
font-size: 10px;
|
||||||
|
}
|
||||||
`;
|
`;
|
||||||
|
|
||||||
if (!document.querySelector('#live-demo-enhanced-styles')) {
|
if (!document.querySelector('#live-demo-enhanced-styles')) {
|
||||||
|
|
@ -545,7 +716,11 @@ export class LiveDemoTab {
|
||||||
handlePoseUpdate(data) {
|
handlePoseUpdate(data) {
|
||||||
this.metrics.frameCount++;
|
this.metrics.frameCount++;
|
||||||
this.metrics.lastUpdate = Date.now();
|
this.metrics.lastUpdate = Date.now();
|
||||||
this.updateDebugOutput(`Pose update: ${data.persons?.length || 0} persons detected`);
|
// Update pose source indicator if the backend supplies it
|
||||||
|
if (data.pose_source && data.pose_source !== this.state.poseSource) {
|
||||||
|
this.setState({ poseSource: data.pose_source });
|
||||||
|
}
|
||||||
|
this.updateDebugOutput(`Pose update: ${data.persons?.length || 0} persons detected (${data.pose_source || 'unknown'})`);
|
||||||
}
|
}
|
||||||
|
|
||||||
handleCanvasError(error) {
|
handleCanvasError(error) {
|
||||||
|
|
@ -706,6 +881,7 @@ export class LiveDemoTab {
|
||||||
this.updateStatusIndicator();
|
this.updateStatusIndicator();
|
||||||
this.updateControls();
|
this.updateControls();
|
||||||
this.updateMetricsDisplay();
|
this.updateMetricsDisplay();
|
||||||
|
this.updatePoseSourceIndicator();
|
||||||
}
|
}
|
||||||
|
|
||||||
updateStatusIndicator() {
|
updateStatusIndicator() {
|
||||||
|
|
@ -789,6 +965,33 @@ export class LiveDemoTab {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
updatePoseSourceIndicator() {
|
||||||
|
const badge = this.container.querySelector('#pose-source-badge');
|
||||||
|
const description = this.container.querySelector('#pose-source-description');
|
||||||
|
|
||||||
|
if (!badge || !description) return;
|
||||||
|
|
||||||
|
const source = this.state.poseSource;
|
||||||
|
|
||||||
|
if (source === 'model_inference') {
|
||||||
|
badge.className = 'pose-source-badge pose-source-model';
|
||||||
|
badge.textContent = 'Model Inference';
|
||||||
|
description.textContent =
|
||||||
|
'Pose is estimated by a trained neural network ' +
|
||||||
|
'loaded from an RVF container.';
|
||||||
|
} else if (source === 'signal_derived') {
|
||||||
|
badge.className = 'pose-source-badge pose-source-signal';
|
||||||
|
badge.textContent = 'Signal-Derived';
|
||||||
|
description.textContent =
|
||||||
|
'Keypoints are derived from live CSI signal features ' +
|
||||||
|
'(motion power, breathing rate, variance).';
|
||||||
|
} else {
|
||||||
|
badge.className = 'pose-source-badge pose-source-unknown';
|
||||||
|
badge.textContent = 'Unknown';
|
||||||
|
description.textContent = 'Waiting for first frame...';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
getHealthClass(status) {
|
getHealthClass(status) {
|
||||||
switch (status) {
|
switch (status) {
|
||||||
case 'connected': return 'good';
|
case 'connected': return 'good';
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,13 @@ export class SensingTab {
|
||||||
_buildDOM() {
|
_buildDOM() {
|
||||||
this.container.innerHTML = `
|
this.container.innerHTML = `
|
||||||
<h2>Live WiFi Sensing</h2>
|
<h2>Live WiFi Sensing</h2>
|
||||||
|
|
||||||
|
<!-- Data-source status banner — updated by _onStateChange -->
|
||||||
|
<div id="sensingSourceBanner" class="sensing-source-banner sensing-source-reconnecting"
|
||||||
|
role="status" aria-live="polite">
|
||||||
|
RECONNECTING...
|
||||||
|
</div>
|
||||||
|
|
||||||
<div class="sensing-layout">
|
<div class="sensing-layout">
|
||||||
<!-- 3D viewport -->
|
<!-- 3D viewport -->
|
||||||
<div class="sensing-viewport" id="sensingViewport">
|
<div class="sensing-viewport" id="sensingViewport">
|
||||||
|
|
@ -98,6 +105,17 @@ export class SensingTab {
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<!-- Setup info -->
|
||||||
|
<div class="sensing-card">
|
||||||
|
<div class="sensing-card-title">About This Data</div>
|
||||||
|
<p class="sensing-about-text">
|
||||||
|
Metrics are computed from WiFi Channel State Information (CSI).
|
||||||
|
With <strong>1 ESP32</strong> you get presence detection, breathing
|
||||||
|
estimation, and gross motion. Add <strong>3-4+ ESP32 nodes</strong>
|
||||||
|
around the room for spatial resolution and limb-level tracking.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
<!-- Extra info -->
|
<!-- Extra info -->
|
||||||
<div class="sensing-card">
|
<div class="sensing-card">
|
||||||
<div class="sensing-card-title">Details</div>
|
<div class="sensing-card-title">Details</div>
|
||||||
|
|
@ -178,19 +196,34 @@ export class SensingTab {
|
||||||
}
|
}
|
||||||
|
|
||||||
_onStateChange(state) {
|
_onStateChange(state) {
|
||||||
const dot = this.container.querySelector('#sensingDot');
|
const dot = this.container.querySelector('#sensingDot');
|
||||||
const text = this.container.querySelector('#sensingState');
|
const text = this.container.querySelector('#sensingState');
|
||||||
if (!dot || !text) return;
|
const banner = this.container.querySelector('#sensingSourceBanner');
|
||||||
|
|
||||||
const labels = {
|
if (dot && text) {
|
||||||
disconnected: 'Disconnected',
|
const stateLabels = {
|
||||||
connecting: 'Connecting...',
|
disconnected: 'Disconnected',
|
||||||
connected: 'Connected',
|
connecting: 'Connecting...',
|
||||||
simulated: 'Simulated',
|
connected: 'Connected',
|
||||||
};
|
reconnecting: 'Reconnecting...',
|
||||||
|
simulated: 'Simulated',
|
||||||
|
};
|
||||||
|
dot.className = 'sensing-dot ' + state;
|
||||||
|
text.textContent = stateLabels[state] || state;
|
||||||
|
}
|
||||||
|
|
||||||
dot.className = 'sensing-dot ' + state;
|
if (banner) {
|
||||||
text.textContent = labels[state] || state;
|
// Map the service's dataSource to banner text and CSS modifier class.
|
||||||
|
const dataSource = sensingService.dataSource;
|
||||||
|
const bannerConfig = {
|
||||||
|
live: { text: 'LIVE - ESP32', cls: 'sensing-source-live' },
|
||||||
|
reconnecting: { text: 'RECONNECTING...', cls: 'sensing-source-reconnecting' },
|
||||||
|
simulated: { text: 'SIMULATED DATA', cls: 'sensing-source-simulated' },
|
||||||
|
};
|
||||||
|
const cfg = bannerConfig[dataSource] || bannerConfig.reconnecting;
|
||||||
|
banner.textContent = cfg.text;
|
||||||
|
banner.className = 'sensing-source-banner ' + cfg.cls;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- HUD update --------------------------------------------------------
|
// ---- HUD update --------------------------------------------------------
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
import { useEffect } from 'react';
|
import { useEffect } from 'react';
|
||||||
import { wsService } from '@/services/ws.service';
|
import { wsService } from '@/services/ws.service';
|
||||||
import { usePoseStore } from '@/stores/poseStore';
|
import { usePoseStore } from '@/stores/poseStore';
|
||||||
|
import { useSettingsStore } from '@/stores/settingsStore';
|
||||||
|
|
||||||
export interface UsePoseStreamResult {
|
export interface UsePoseStreamResult {
|
||||||
connectionStatus: ReturnType<typeof usePoseStore.getState>['connectionStatus'];
|
connectionStatus: ReturnType<typeof usePoseStore.getState>['connectionStatus'];
|
||||||
|
|
@ -12,16 +13,20 @@ export function usePoseStream(): UsePoseStreamResult {
|
||||||
const connectionStatus = usePoseStore((state) => state.connectionStatus);
|
const connectionStatus = usePoseStore((state) => state.connectionStatus);
|
||||||
const lastFrame = usePoseStore((state) => state.lastFrame);
|
const lastFrame = usePoseStore((state) => state.lastFrame);
|
||||||
const isSimulated = usePoseStore((state) => state.isSimulated);
|
const isSimulated = usePoseStore((state) => state.isSimulated);
|
||||||
|
const serverUrl = useSettingsStore((state) => state.serverUrl);
|
||||||
|
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
const unsubscribe = wsService.subscribe((frame) => {
|
const unsubscribe = wsService.subscribe((frame) => {
|
||||||
usePoseStore.getState().handleFrame(frame);
|
usePoseStore.getState().handleFrame(frame);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Auto-connect to sensing server on mount
|
||||||
|
wsService.connect(serverUrl);
|
||||||
|
|
||||||
return () => {
|
return () => {
|
||||||
unsubscribe();
|
unsubscribe();
|
||||||
};
|
};
|
||||||
}, []);
|
}, [serverUrl]);
|
||||||
|
|
||||||
return { connectionStatus, lastFrame, isSimulated };
|
return { connectionStatus, lastFrame, isSimulated };
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,40 +3,116 @@ import { StyleSheet, View } from 'react-native';
|
||||||
import * as THREE from 'three';
|
import * as THREE from 'three';
|
||||||
import type { SensingFrame } from '@/types/sensing';
|
import type { SensingFrame } from '@/types/sensing';
|
||||||
|
|
||||||
type GaussianSplatWebViewWebProps = {
|
type Props = {
|
||||||
onReady: () => void;
|
onReady: () => void;
|
||||||
onFps: (fps: number) => void;
|
onFps: (fps: number) => void;
|
||||||
onError: (msg: string) => void;
|
onError: (msg: string) => void;
|
||||||
frame: SensingFrame | null;
|
frame: SensingFrame | null;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// COCO skeleton bones
|
||||||
const BONES: [number, number][] = [
|
const BONES: [number, number][] = [
|
||||||
[0,1],[0,2],[1,3],[2,4],[5,6],[5,7],[7,9],[6,8],[8,10],
|
[0,1],[0,2],[1,3],[2,4],[5,6],[5,7],[7,9],[6,8],[8,10],
|
||||||
[5,11],[6,12],[11,12],[11,13],[13,15],[12,14],[14,16],
|
[5,11],[6,12],[11,12],[11,13],[13,15],[12,14],[14,16],
|
||||||
];
|
];
|
||||||
|
|
||||||
export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: GaussianSplatWebViewWebProps) => {
|
// Standing pose (meters, Y-up)
|
||||||
const containerRef = useRef<HTMLDivElement>(null);
|
const BASE_POSE: [number, number, number][] = [
|
||||||
const sceneRef = useRef<{
|
[ 0.00, 1.72, 0.04], // 0 nose
|
||||||
renderer: THREE.WebGLRenderer;
|
[-0.03, 1.76, 0.05], // 1 left eye
|
||||||
scene: THREE.Scene;
|
[ 0.03, 1.76, 0.05], // 2 right eye
|
||||||
camera: THREE.PerspectiveCamera;
|
[-0.08, 1.74,-0.01], // 3 left ear
|
||||||
joints: THREE.Mesh[];
|
[ 0.08, 1.74,-0.01], // 4 right ear
|
||||||
boneLines: { line: THREE.Line; a: number; b: number }[];
|
[-0.20, 1.45, 0.00], // 5 left shoulder
|
||||||
ring: THREE.Mesh;
|
[ 0.20, 1.45, 0.00], // 6 right shoulder
|
||||||
particleGeo: THREE.BufferGeometry;
|
[-0.26, 1.12, 0.04], // 7 left elbow
|
||||||
pointLight: THREE.PointLight;
|
[ 0.26, 1.12, 0.04], // 8 right elbow
|
||||||
animId: number;
|
[-0.28, 0.82, 0.02], // 9 left wrist
|
||||||
cameraAngle: number;
|
[ 0.28, 0.82, 0.02], // 10 right wrist
|
||||||
cameraRadius: number;
|
[-0.11, 0.95, 0.00], // 11 left hip
|
||||||
cameraY: number;
|
[ 0.11, 0.95, 0.00], // 12 right hip
|
||||||
isDragging: boolean;
|
[-0.12, 0.50, 0.02], // 13 left knee
|
||||||
frameCount: number;
|
[ 0.12, 0.50, 0.02], // 14 right knee
|
||||||
lastFpsTime: number;
|
[-0.12, 0.04, 0.00], // 15 left ankle
|
||||||
} | null>(null);
|
[ 0.12, 0.04, 0.00], // 16 right ankle
|
||||||
const frameRef = useRef<SensingFrame | null>(null);
|
];
|
||||||
|
|
||||||
// Keep frame ref current without re-running effect
|
// DensePose-style body part colors (24 parts → simplified per-segment)
|
||||||
|
const DENSEPOSE_COLORS: Record<string, number> = {
|
||||||
|
head: 0xf4a582, // warm skin
|
||||||
|
neck: 0xd6604d, // darker warm
|
||||||
|
torsoFront: 0x92c5de, // blue-gray
|
||||||
|
torsoSide: 0x4393c3, // steel blue
|
||||||
|
pelvis: 0x2166ac, // deep blue
|
||||||
|
lUpperArm: 0xd73027, // red
|
||||||
|
rUpperArm: 0xf46d43, // orange-red
|
||||||
|
lForearm: 0xfdae61, // orange
|
||||||
|
rForearm: 0xfee090, // light orange
|
||||||
|
lHand: 0xffffbf, // pale yellow
|
||||||
|
rHand: 0xffffbf,
|
||||||
|
lThigh: 0xa6d96a, // green
|
||||||
|
rThigh: 0x66bd63, // darker green
|
||||||
|
lShin: 0x1a9850, // deep green
|
||||||
|
rShin: 0x006837, // forest
|
||||||
|
lFoot: 0x762a83, // purple
|
||||||
|
rFoot: 0x9970ab, // light purple
|
||||||
|
};
|
||||||
|
|
||||||
|
// Body segments: [jointA, jointB, topRadius, botRadius, colorKey]
|
||||||
|
const BODY_SEGS: [number, number, number, number, string][] = [
|
||||||
|
[5, 6, 0.10, 0.10, 'torsoFront'], // collar
|
||||||
|
[5, 11, 0.09, 0.07, 'torsoSide'], // L torso
|
||||||
|
[6, 12, 0.09, 0.07, 'torsoSide'], // R torso
|
||||||
|
[11, 12, 0.08, 0.08, 'pelvis'], // pelvis
|
||||||
|
[5, 7, 0.045,0.040,'lUpperArm'], // L upper arm
|
||||||
|
[7, 9, 0.038,0.032,'lForearm'], // L forearm
|
||||||
|
[6, 8, 0.045,0.040,'rUpperArm'], // R upper arm
|
||||||
|
[8, 10, 0.038,0.032,'rForearm'], // R forearm
|
||||||
|
[11, 13, 0.065,0.050,'lThigh'], // L thigh
|
||||||
|
[13, 15, 0.048,0.038,'lShin'], // L shin
|
||||||
|
[12, 14, 0.065,0.050,'rThigh'], // R thigh
|
||||||
|
[14, 16, 0.048,0.038,'rShin'], // R shin
|
||||||
|
];
|
||||||
|
|
||||||
|
function makePart(scene: THREE.Scene, rTop: number, rBot: number, color: number, glow: boolean = false): THREE.Mesh {
|
||||||
|
const geo = new THREE.CapsuleGeometry((rTop + rBot) / 2, 1, 6, 12);
|
||||||
|
const mat = new THREE.MeshPhysicalMaterial({
|
||||||
|
color, emissive: color,
|
||||||
|
emissiveIntensity: glow ? 0.4 : 0.08,
|
||||||
|
transparent: true, opacity: glow ? 0.12 : 0.85,
|
||||||
|
roughness: 0.35, metalness: 0.1,
|
||||||
|
clearcoat: glow ? 0 : 0.3, clearcoatRoughness: 0.4,
|
||||||
|
side: glow ? THREE.BackSide : THREE.FrontSide,
|
||||||
|
});
|
||||||
|
const m = new THREE.Mesh(geo, mat);
|
||||||
|
m.visible = false;
|
||||||
|
m.castShadow = !glow;
|
||||||
|
scene.add(m);
|
||||||
|
return m;
|
||||||
|
}
|
||||||
|
|
||||||
|
function positionLimb(mesh: THREE.Mesh, a: THREE.Vector3, b: THREE.Vector3, rTop: number, rBot: number) {
|
||||||
|
const mid = new THREE.Vector3().addVectors(a, b).multiplyScalar(0.5);
|
||||||
|
mesh.position.copy(mid);
|
||||||
|
const len = a.distanceTo(b);
|
||||||
|
// CapsuleGeometry height param = 1, so scale Y to actual length
|
||||||
|
mesh.scale.set((rTop + rBot) * 10, len, (rTop + rBot) * 10);
|
||||||
|
const dir = new THREE.Vector3().subVectors(b, a).normalize();
|
||||||
|
const up = new THREE.Vector3(0, 1, 0);
|
||||||
|
const quat = new THREE.Quaternion().setFromUnitVectors(up, dir);
|
||||||
|
mesh.quaternion.copy(quat);
|
||||||
|
}
|
||||||
|
|
||||||
|
function lerp3(out: THREE.Vector3, target: THREE.Vector3, alpha: number) {
|
||||||
|
out.x += (target.x - out.x) * alpha;
|
||||||
|
out.y += (target.y - out.y) * alpha;
|
||||||
|
out.z += (target.z - out.z) * alpha;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: Props) => {
|
||||||
|
const containerRef = useRef<HTMLDivElement>(null);
|
||||||
|
const frameRef = useRef<SensingFrame | null>(null);
|
||||||
|
const sceneRef = useRef<any>(null);
|
||||||
frameRef.current = frame;
|
frameRef.current = frame;
|
||||||
|
|
||||||
const cleanup = useCallback(() => {
|
const cleanup = useCallback(() => {
|
||||||
|
|
@ -44,11 +120,11 @@ export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: Gaus
|
||||||
if (!s) return;
|
if (!s) return;
|
||||||
cancelAnimationFrame(s.animId);
|
cancelAnimationFrame(s.animId);
|
||||||
s.renderer.dispose();
|
s.renderer.dispose();
|
||||||
s.scene.traverse((obj) => {
|
s.scene.traverse((obj: any) => {
|
||||||
if (obj instanceof THREE.Mesh) {
|
if (obj.geometry) obj.geometry.dispose();
|
||||||
obj.geometry.dispose();
|
if (obj.material) {
|
||||||
if (Array.isArray(obj.material)) obj.material.forEach((m) => m.dispose());
|
const mats = Array.isArray(obj.material) ? obj.material : [obj.material];
|
||||||
else obj.material.dispose();
|
mats.forEach((m: any) => m.dispose());
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
sceneRef.current = null;
|
sceneRef.current = null;
|
||||||
|
|
@ -57,222 +133,544 @@ export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: Gaus
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
const container = containerRef.current;
|
const container = containerRef.current;
|
||||||
if (!container) return;
|
if (!container) return;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const W = () => container.clientWidth || window.innerWidth;
|
const W = () => container.clientWidth || window.innerWidth;
|
||||||
const H = () => container.clientHeight || window.innerHeight;
|
const H = () => container.clientHeight || window.innerHeight;
|
||||||
|
|
||||||
// Renderer
|
// --- Renderer ---
|
||||||
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: false });
|
const renderer = new THREE.WebGLRenderer({ antialias: true, powerPreference: 'high-performance' });
|
||||||
renderer.setSize(W(), H());
|
renderer.setSize(W(), H());
|
||||||
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
|
renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2));
|
||||||
renderer.setClearColor(0x0a0e1a);
|
renderer.setClearColor(0x080c16);
|
||||||
|
renderer.shadowMap.enabled = true;
|
||||||
|
renderer.shadowMap.type = THREE.PCFSoftShadowMap;
|
||||||
|
renderer.toneMapping = THREE.ACESFilmicToneMapping;
|
||||||
|
renderer.toneMappingExposure = 1.1;
|
||||||
container.appendChild(renderer.domElement);
|
container.appendChild(renderer.domElement);
|
||||||
|
|
||||||
// Scene
|
|
||||||
const scene = new THREE.Scene();
|
const scene = new THREE.Scene();
|
||||||
scene.background = new THREE.Color(0x0a0e1a);
|
scene.background = new THREE.Color(0x080c16);
|
||||||
scene.fog = new THREE.FogExp2(0x0a0e1a, 0.008);
|
scene.fog = new THREE.FogExp2(0x080c16, 0.018);
|
||||||
|
|
||||||
// Camera
|
const camera = new THREE.PerspectiveCamera(45, W() / H(), 0.1, 200);
|
||||||
const camera = new THREE.PerspectiveCamera(60, W() / H(), 0.1, 500);
|
camera.position.set(0, 1.4, 3.5);
|
||||||
camera.position.set(0, 2, 6);
|
camera.lookAt(0, 0.9, 0);
|
||||||
camera.lookAt(0, 1, 0);
|
|
||||||
|
|
||||||
// Grid
|
// --- Lighting (3-point + rim) ---
|
||||||
const grid = new THREE.GridHelper(20, 40, 0x1a3a4a, 0x0d1f2a);
|
scene.add(new THREE.AmbientLight(0x223344, 0.5));
|
||||||
scene.add(grid);
|
|
||||||
|
|
||||||
// Lights
|
const key = new THREE.DirectionalLight(0xddeeff, 1.0);
|
||||||
scene.add(new THREE.AmbientLight(0x32b8c6, 0.3));
|
key.position.set(2, 5, 3);
|
||||||
const pointLight = new THREE.PointLight(0x32b8c6, 1.5, 20);
|
key.castShadow = true;
|
||||||
pointLight.position.set(0, 4, 0);
|
key.shadow.mapSize.set(1024, 1024);
|
||||||
scene.add(pointLight);
|
key.shadow.camera.near = 0.5;
|
||||||
|
key.shadow.camera.far = 15;
|
||||||
|
key.shadow.camera.left = -3;
|
||||||
|
key.shadow.camera.right = 3;
|
||||||
|
key.shadow.camera.top = 3;
|
||||||
|
key.shadow.camera.bottom = -1;
|
||||||
|
scene.add(key);
|
||||||
|
|
||||||
// Skeleton joints (17 COCO keypoints)
|
const rim = new THREE.PointLight(0x32b8c6, 1.5, 12);
|
||||||
const jointGeo = new THREE.SphereGeometry(0.06, 8, 8);
|
rim.position.set(-1.5, 2.5, -2);
|
||||||
const joints: THREE.Mesh[] = [];
|
scene.add(rim);
|
||||||
for (let i = 0; i < 17; i++) {
|
|
||||||
const mat = new THREE.MeshStandardMaterial({
|
const fill = new THREE.PointLight(0x554488, 0.5, 8);
|
||||||
color: 0x32b8c6,
|
fill.position.set(1.5, 0.8, 2.5);
|
||||||
emissive: 0x32b8c6,
|
scene.add(fill);
|
||||||
emissiveIntensity: 0.6,
|
|
||||||
});
|
const under = new THREE.PointLight(0x225566, 0.4, 5);
|
||||||
const m = new THREE.Mesh(jointGeo, mat);
|
under.position.set(0, 0.1, 1);
|
||||||
m.visible = false;
|
scene.add(under);
|
||||||
scene.add(m);
|
|
||||||
joints.push(m);
|
// --- Ground ---
|
||||||
|
const groundGeo = new THREE.PlaneGeometry(20, 20);
|
||||||
|
const groundMat = new THREE.MeshStandardMaterial({
|
||||||
|
color: 0x0a0e1a, roughness: 0.9, metalness: 0.1,
|
||||||
|
});
|
||||||
|
const ground = new THREE.Mesh(groundGeo, groundMat);
|
||||||
|
ground.rotation.x = -Math.PI / 2;
|
||||||
|
ground.receiveShadow = true;
|
||||||
|
scene.add(ground);
|
||||||
|
|
||||||
|
const gridH = new THREE.GridHelper(20, 40, 0x1a3050, 0x0e1826);
|
||||||
|
gridH.position.y = 0.002;
|
||||||
|
scene.add(gridH);
|
||||||
|
|
||||||
|
// --- Signal field (20x20) ---
|
||||||
|
const GS = 20;
|
||||||
|
const cellGeo = new THREE.PlaneGeometry(0.38, 0.38);
|
||||||
|
const cellMat = new THREE.MeshBasicMaterial({ color: 0x32b8c6, transparent: true, opacity: 0.25, side: THREE.DoubleSide });
|
||||||
|
const sigGrid = new THREE.InstancedMesh(cellGeo, cellMat, GS * GS);
|
||||||
|
sigGrid.rotation.x = -Math.PI / 2; sigGrid.position.y = 0.005;
|
||||||
|
const dum = new THREE.Object3D();
|
||||||
|
for (let z = 0; z < GS; z++) for (let x = 0; x < GS; x++) {
|
||||||
|
dum.position.set((x - GS / 2) * 0.4, (z - GS / 2) * 0.4, 0);
|
||||||
|
dum.updateMatrix();
|
||||||
|
sigGrid.setMatrixAt(z * GS + x, dum.matrix);
|
||||||
|
sigGrid.setColorAt(z * GS + x, new THREE.Color(0x080c16));
|
||||||
|
}
|
||||||
|
sigGrid.instanceMatrix.needsUpdate = true;
|
||||||
|
if (sigGrid.instanceColor) sigGrid.instanceColor.needsUpdate = true;
|
||||||
|
scene.add(sigGrid);
|
||||||
|
|
||||||
|
// --- ESP32 nodes ---
|
||||||
|
const nodeGeo = new THREE.OctahedronGeometry(0.08, 1);
|
||||||
|
const nodeMs: THREE.Mesh[] = [];
|
||||||
|
for (let i = 0; i < 8; i++) {
|
||||||
|
const mat = new THREE.MeshStandardMaterial({ color: 0x00ff88, emissive: 0x00ff88, emissiveIntensity: 0.7, wireframe: true });
|
||||||
|
const m = new THREE.Mesh(nodeGeo, mat); m.visible = false; scene.add(m); nodeMs.push(m);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Bone lines
|
// --- Human body: DensePose-colored capsule mesh ---
|
||||||
const boneMat = new THREE.LineBasicMaterial({
|
// Head: slightly oblate sphere
|
||||||
color: 0x32b8c6,
|
const headGeo = new THREE.SphereGeometry(0.105, 20, 16);
|
||||||
transparent: true,
|
headGeo.scale(1, 1.08, 1);
|
||||||
opacity: 0.7,
|
const headMat = new THREE.MeshPhysicalMaterial({
|
||||||
});
|
color: DENSEPOSE_COLORS.head, emissive: DENSEPOSE_COLORS.head,
|
||||||
const boneLines = BONES.map(([a, b]) => {
|
emissiveIntensity: 0.08, roughness: 0.3, metalness: 0.05,
|
||||||
const g = new THREE.BufferGeometry().setFromPoints([
|
clearcoat: 0.4, clearcoatRoughness: 0.3, transparent: true, opacity: 0.9,
|
||||||
new THREE.Vector3(),
|
|
||||||
new THREE.Vector3(),
|
|
||||||
]);
|
|
||||||
const l = new THREE.Line(g, boneMat);
|
|
||||||
l.visible = false;
|
|
||||||
scene.add(l);
|
|
||||||
return { line: l, a, b };
|
|
||||||
});
|
});
|
||||||
|
const headM = new THREE.Mesh(headGeo, headMat);
|
||||||
|
headM.castShadow = true; headM.visible = false; scene.add(headM);
|
||||||
|
|
||||||
// Particle field
|
// Head glow
|
||||||
const N = 500;
|
const headGlowGeo = new THREE.SphereGeometry(0.14, 12, 10);
|
||||||
const particleGeo = new THREE.BufferGeometry();
|
const headGlowMat = new THREE.MeshBasicMaterial({
|
||||||
const pPos = new Float32Array(N * 3);
|
color: DENSEPOSE_COLORS.head, transparent: true, opacity: 0.08, side: THREE.BackSide,
|
||||||
for (let i = 0; i < N; i++) {
|
});
|
||||||
pPos[i * 3] = (Math.random() - 0.5) * 16;
|
const headGlowM = new THREE.Mesh(headGlowGeo, headGlowMat);
|
||||||
pPos[i * 3 + 1] = Math.random() * 4;
|
headGlowM.visible = false; scene.add(headGlowM);
|
||||||
pPos[i * 3 + 2] = (Math.random() - 0.5) * 16;
|
|
||||||
|
// Eyes
|
||||||
|
const eyeGeo = new THREE.SphereGeometry(0.015, 8, 6);
|
||||||
|
const eyeMat = new THREE.MeshBasicMaterial({ color: 0xeeffff });
|
||||||
|
const eyeL = new THREE.Mesh(eyeGeo, eyeMat);
|
||||||
|
const eyeR = new THREE.Mesh(eyeGeo, eyeMat.clone());
|
||||||
|
eyeL.visible = eyeR.visible = false;
|
||||||
|
scene.add(eyeL); scene.add(eyeR);
|
||||||
|
|
||||||
|
// Pupils
|
||||||
|
const pupilGeo = new THREE.SphereGeometry(0.008, 6, 4);
|
||||||
|
const pupilMat = new THREE.MeshBasicMaterial({ color: 0x112233 });
|
||||||
|
const pupilL = new THREE.Mesh(pupilGeo, pupilMat);
|
||||||
|
const pupilR = new THREE.Mesh(pupilGeo, pupilMat.clone());
|
||||||
|
pupilL.visible = pupilR.visible = false;
|
||||||
|
scene.add(pupilL); scene.add(pupilR);
|
||||||
|
|
||||||
|
// Neck
|
||||||
|
const neckGeo = new THREE.CapsuleGeometry(0.04, 0.08, 4, 8);
|
||||||
|
const neckMat = new THREE.MeshPhysicalMaterial({
|
||||||
|
color: DENSEPOSE_COLORS.neck, emissive: DENSEPOSE_COLORS.neck,
|
||||||
|
emissiveIntensity: 0.05, roughness: 0.4, transparent: true, opacity: 0.85,
|
||||||
|
});
|
||||||
|
const neckM = new THREE.Mesh(neckGeo, neckMat);
|
||||||
|
neckM.castShadow = true; neckM.visible = false; scene.add(neckM);
|
||||||
|
|
||||||
|
// Torso: front plate
|
||||||
|
const torsoGeo = new THREE.BoxGeometry(0.34, 0.50, 0.18, 2, 3, 2);
|
||||||
|
// Round the torso vertices slightly
|
||||||
|
const torsoPos = torsoGeo.attributes.position;
|
||||||
|
for (let i = 0; i < torsoPos.count; i++) {
|
||||||
|
const x = torsoPos.getX(i), y = torsoPos.getY(i), z = torsoPos.getZ(i);
|
||||||
|
const r = Math.sqrt(x * x + z * z);
|
||||||
|
if (r > 0.01) {
|
||||||
|
const bulge = 1 + 0.15 * Math.cos(y * 3.5); // chest & hip curvature
|
||||||
|
torsoPos.setX(i, x * bulge);
|
||||||
|
torsoPos.setZ(i, z * bulge);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
particleGeo.setAttribute('position', new THREE.BufferAttribute(pPos, 3));
|
torsoGeo.computeVertexNormals();
|
||||||
const pMat = new THREE.PointsMaterial({
|
const torsoMat = new THREE.MeshPhysicalMaterial({
|
||||||
color: 0x32b8c6,
|
color: DENSEPOSE_COLORS.torsoFront, emissive: DENSEPOSE_COLORS.torsoFront,
|
||||||
size: 0.04,
|
emissiveIntensity: 0.06, roughness: 0.35, metalness: 0.05,
|
||||||
transparent: true,
|
clearcoat: 0.2, transparent: true, opacity: 0.88,
|
||||||
opacity: 0.4,
|
|
||||||
});
|
});
|
||||||
scene.add(new THREE.Points(particleGeo, pMat));
|
const torsoM = new THREE.Mesh(torsoGeo, torsoMat);
|
||||||
|
torsoM.castShadow = true; torsoM.visible = false; scene.add(torsoM);
|
||||||
|
|
||||||
// Signal ring
|
// Torso glow
|
||||||
const ringGeo = new THREE.TorusGeometry(2, 0.02, 8, 64);
|
const torsoGlowGeo = new THREE.BoxGeometry(0.40, 0.55, 0.24);
|
||||||
const ringMat = new THREE.MeshBasicMaterial({
|
const torsoGlowMat = new THREE.MeshBasicMaterial({
|
||||||
color: 0x32b8c6,
|
color: DENSEPOSE_COLORS.torsoFront, transparent: true, opacity: 0.06, side: THREE.BackSide,
|
||||||
transparent: true,
|
|
||||||
opacity: 0.3,
|
|
||||||
});
|
});
|
||||||
const ring = new THREE.Mesh(ringGeo, ringMat);
|
const torsoGlowM = new THREE.Mesh(torsoGlowGeo, torsoGlowMat);
|
||||||
ring.rotation.x = Math.PI / 2;
|
torsoGlowM.visible = false; scene.add(torsoGlowM);
|
||||||
ring.position.y = 0.01;
|
|
||||||
scene.add(ring);
|
// Hands (small boxes)
|
||||||
|
const handGeo = new THREE.BoxGeometry(0.05, 0.08, 0.025, 1, 1, 1);
|
||||||
|
const handLMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.lHand, emissive: DENSEPOSE_COLORS.lHand, emissiveIntensity: 0.1, roughness: 0.3, transparent: true, opacity: 0.85 });
|
||||||
|
const handRMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.rHand, emissive: DENSEPOSE_COLORS.rHand, emissiveIntensity: 0.1, roughness: 0.3, transparent: true, opacity: 0.85 });
|
||||||
|
const handL = new THREE.Mesh(handGeo, handLMat); handL.visible = false; scene.add(handL);
|
||||||
|
const handR = new THREE.Mesh(handGeo, handRMat); handR.visible = false; scene.add(handR);
|
||||||
|
|
||||||
|
// Feet (wedge-like boxes)
|
||||||
|
const footGeo = new THREE.BoxGeometry(0.06, 0.04, 0.14, 1, 1, 1);
|
||||||
|
const footLMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.lFoot, emissive: DENSEPOSE_COLORS.lFoot, emissiveIntensity: 0.1, roughness: 0.4, transparent: true, opacity: 0.85 });
|
||||||
|
const footRMat = new THREE.MeshPhysicalMaterial({ color: DENSEPOSE_COLORS.rFoot, emissive: DENSEPOSE_COLORS.rFoot, emissiveIntensity: 0.1, roughness: 0.4, transparent: true, opacity: 0.85 });
|
||||||
|
const footL = new THREE.Mesh(footGeo, footLMat); footL.visible = false; scene.add(footL);
|
||||||
|
const footR = new THREE.Mesh(footGeo, footRMat); footR.visible = false; scene.add(footR);
|
||||||
|
|
||||||
|
// Limb capsules + glow capsules
|
||||||
|
const limbMs = BODY_SEGS.map(([,, rT, rB, ck]) => makePart(scene, rT, rB, DENSEPOSE_COLORS[ck]));
|
||||||
|
const limbGlowMs = BODY_SEGS.map(([,, rT, rB, ck]) => makePart(scene, rT * 1.6, rB * 1.6, DENSEPOSE_COLORS[ck], true));
|
||||||
|
|
||||||
|
// Joint dots
|
||||||
|
const jDotGeo = new THREE.SphereGeometry(0.018, 6, 4);
|
||||||
|
const jDots = Array.from({ length: 17 }, () => {
|
||||||
|
const mat = new THREE.MeshBasicMaterial({ color: 0x88ddee, transparent: true, opacity: 0.7 });
|
||||||
|
const m = new THREE.Mesh(jDotGeo, mat); m.visible = false; scene.add(m); return m;
|
||||||
|
});
|
||||||
|
|
||||||
|
// Skeleton lines (thin wireframe overlay)
|
||||||
|
const skelMat = new THREE.LineBasicMaterial({ color: 0x55ccdd, transparent: true, opacity: 0.25 });
|
||||||
|
const skelLines = BONES.map(([a, b]) => {
|
||||||
|
const g = new THREE.BufferGeometry().setFromPoints([new THREE.Vector3(), new THREE.Vector3()]);
|
||||||
|
const l = new THREE.Line(g, skelMat); l.visible = false; scene.add(l); return { line: l, a, b };
|
||||||
|
});
|
||||||
|
|
||||||
|
// Heart ring
|
||||||
|
const hrGeo = new THREE.TorusGeometry(0.18, 0.006, 8, 32);
|
||||||
|
const hrMat = new THREE.MeshBasicMaterial({ color: 0xff3355, transparent: true, opacity: 0 });
|
||||||
|
const hrRing = new THREE.Mesh(hrGeo, hrMat); hrRing.visible = false; scene.add(hrRing);
|
||||||
|
|
||||||
|
// Breathing indicator rings (concentric around chest)
|
||||||
|
const brRings = [0.22, 0.28, 0.34].map((r) => {
|
||||||
|
const geo = new THREE.TorusGeometry(r, 0.003, 6, 32);
|
||||||
|
const mat = new THREE.MeshBasicMaterial({ color: 0x44ddaa, transparent: true, opacity: 0 });
|
||||||
|
const m = new THREE.Mesh(geo, mat); m.visible = false; scene.add(m); return m;
|
||||||
|
});
|
||||||
|
|
||||||
|
// WiFi pulse rings
|
||||||
|
const wifiRings = [1.0, 1.8, 2.6].map((r) => {
|
||||||
|
const geo = new THREE.TorusGeometry(r, 0.01, 6, 48);
|
||||||
|
const mat = new THREE.MeshBasicMaterial({ color: 0x32b8c6, transparent: true, opacity: 0.15 });
|
||||||
|
const m = new THREE.Mesh(geo, mat); m.rotation.x = Math.PI / 2; m.position.y = 0.01; scene.add(m); return m;
|
||||||
|
});
|
||||||
|
|
||||||
|
// Particles
|
||||||
|
const NP = 400;
|
||||||
|
const pGeo = new THREE.BufferGeometry();
|
||||||
|
const pA = new Float32Array(NP * 3);
|
||||||
|
for (let i = 0; i < NP; i++) {
|
||||||
|
pA[i * 3] = (Math.random() - 0.5) * 12;
|
||||||
|
pA[i * 3 + 1] = Math.random() * 3.5;
|
||||||
|
pA[i * 3 + 2] = (Math.random() - 0.5) * 12;
|
||||||
|
}
|
||||||
|
pGeo.setAttribute('position', new THREE.BufferAttribute(pA, 3));
|
||||||
|
scene.add(new THREE.Points(pGeo, new THREE.PointsMaterial({
|
||||||
|
color: 0x3399bb, size: 0.018, transparent: true, opacity: 0.25,
|
||||||
|
})));
|
||||||
|
|
||||||
|
// --- HUD ---
|
||||||
|
const hudC = document.createElement('canvas'); hudC.width = 640; hudC.height = 128;
|
||||||
|
const hudT = new THREE.CanvasTexture(hudC);
|
||||||
|
const hudS = new THREE.Sprite(new THREE.SpriteMaterial({ map: hudT, transparent: true }));
|
||||||
|
hudS.scale.set(3.2, 0.64, 1); hudS.position.set(0, 3.2, 0); scene.add(hudS);
|
||||||
|
|
||||||
|
// --- Smooth keypoints ---
|
||||||
|
const smoothKps: THREE.Vector3[] = BASE_POSE.map(([x, y, z]) => new THREE.Vector3(x, y, z));
|
||||||
|
const targetKps: THREE.Vector3[] = BASE_POSE.map(([x, y, z]) => new THREE.Vector3(x, y, z));
|
||||||
|
const tmpA = new THREE.Vector3();
|
||||||
|
const tmpB = new THREE.Vector3();
|
||||||
|
const hc = new THREE.Color();
|
||||||
|
|
||||||
// State
|
// State
|
||||||
const state = {
|
const state: any = {
|
||||||
renderer,
|
renderer, scene, camera, animId: 0,
|
||||||
scene,
|
camAngle: 0, camR: 3.5, camY: 1.4,
|
||||||
camera,
|
drag: false, fCount: 0, fpsT: performance.now(),
|
||||||
joints,
|
prevPresence: false, fadeIn: 0,
|
||||||
boneLines,
|
|
||||||
ring,
|
|
||||||
particleGeo,
|
|
||||||
pointLight,
|
|
||||||
animId: 0,
|
|
||||||
cameraAngle: 0,
|
|
||||||
cameraRadius: 6,
|
|
||||||
cameraY: 2,
|
|
||||||
isDragging: false,
|
|
||||||
frameCount: 0,
|
|
||||||
lastFpsTime: performance.now(),
|
|
||||||
};
|
};
|
||||||
sceneRef.current = state;
|
sceneRef.current = state;
|
||||||
|
|
||||||
// Mouse interaction
|
// Input
|
||||||
const canvas = renderer.domElement;
|
const cvs = renderer.domElement;
|
||||||
const onMouseDown = () => { state.isDragging = true; };
|
cvs.addEventListener('mousedown', () => { state.drag = true; });
|
||||||
const onMouseUp = () => { state.isDragging = false; };
|
cvs.addEventListener('mouseup', () => { state.drag = false; });
|
||||||
const onMouseMove = (e: MouseEvent) => {
|
cvs.addEventListener('mouseleave', () => { state.drag = false; });
|
||||||
if (state.isDragging) {
|
cvs.addEventListener('mousemove', (e: MouseEvent) => {
|
||||||
state.cameraAngle += e.movementX * 0.01;
|
if (state.drag) { state.camAngle += e.movementX * 0.006; state.camY = Math.max(0.2, Math.min(4, state.camY - e.movementY * 0.006)); }
|
||||||
state.cameraY = Math.max(0.5, Math.min(5, state.cameraY - e.movementY * 0.01));
|
});
|
||||||
}
|
cvs.addEventListener('wheel', (e: WheelEvent) => {
|
||||||
};
|
state.camR = Math.max(1.5, Math.min(10, state.camR + e.deltaY * 0.003));
|
||||||
const onWheel = (e: WheelEvent) => {
|
}, { passive: true });
|
||||||
state.cameraRadius = Math.max(2, Math.min(15, state.cameraRadius + e.deltaY * 0.005));
|
const onR = () => { camera.aspect = W() / H(); camera.updateProjectionMatrix(); renderer.setSize(W(), H()); };
|
||||||
};
|
window.addEventListener('resize', onR);
|
||||||
canvas.addEventListener('mousedown', onMouseDown);
|
|
||||||
canvas.addEventListener('mouseup', onMouseUp);
|
|
||||||
canvas.addEventListener('mousemove', onMouseMove);
|
|
||||||
canvas.addEventListener('wheel', onWheel, { passive: true });
|
|
||||||
|
|
||||||
// Resize
|
// --- Animate ---
|
||||||
const onResize = () => {
|
|
||||||
camera.aspect = W() / H();
|
|
||||||
camera.updateProjectionMatrix();
|
|
||||||
renderer.setSize(W(), H());
|
|
||||||
};
|
|
||||||
window.addEventListener('resize', onResize);
|
|
||||||
|
|
||||||
// Animation loop
|
|
||||||
const animate = () => {
|
const animate = () => {
|
||||||
state.animId = requestAnimationFrame(animate);
|
state.animId = requestAnimationFrame(animate);
|
||||||
const t = performance.now() * 0.001;
|
const t = performance.now() * 0.001;
|
||||||
|
const fr = frameRef.current;
|
||||||
|
|
||||||
// Camera orbit
|
// Camera
|
||||||
if (!state.isDragging) state.cameraAngle += 0.002;
|
if (!state.drag) state.camAngle += 0.001;
|
||||||
camera.position.set(
|
camera.position.set(Math.sin(state.camAngle) * state.camR, state.camY, Math.cos(state.camAngle) * state.camR);
|
||||||
Math.sin(state.cameraAngle) * state.cameraRadius,
|
camera.lookAt(0, 0.95, 0);
|
||||||
state.cameraY,
|
|
||||||
Math.cos(state.cameraAngle) * state.cameraRadius,
|
|
||||||
);
|
|
||||||
camera.lookAt(0, 1, 0);
|
|
||||||
|
|
||||||
// Animate ring
|
const pres = fr?.classification?.presence ?? false;
|
||||||
ring.material.opacity = 0.15 + Math.sin(t * 2) * 0.1;
|
const mot = fr?.classification?.motion_level ?? 'absent';
|
||||||
const scale = 1 + Math.sin(t) * 0.1;
|
const conf = fr?.classification?.confidence ?? 0;
|
||||||
ring.scale.set(scale, scale, 1);
|
const mPow = fr?.features?.motion_band_power ?? 0;
|
||||||
|
const bPow = fr?.features?.breathing_band_power ?? 0;
|
||||||
|
const rssi = fr?.features?.mean_rssi ?? -80;
|
||||||
|
|
||||||
// Animate particles
|
// Fade body in/out (gradual transitions)
|
||||||
const pp = particleGeo.attributes.position as THREE.BufferAttribute;
|
if (pres && conf > 0.2) state.fadeIn = Math.min(1, state.fadeIn + 0.015);
|
||||||
for (let i = 0; i < N; i++) {
|
else state.fadeIn = Math.max(0, state.fadeIn - 0.008);
|
||||||
(pp.array as Float32Array)[i * 3 + 1] += Math.sin(t + i) * 0.001;
|
const show = state.fadeIn > 0.01;
|
||||||
|
const alpha = state.fadeIn;
|
||||||
|
|
||||||
|
// --- Compute target keypoints ---
|
||||||
|
for (let i = 0; i < 17; i++) {
|
||||||
|
const [bx, by, bz] = BASE_POSE[i];
|
||||||
|
let ax = bx, ay = by, az = bz;
|
||||||
|
|
||||||
|
if (pres) {
|
||||||
|
// Breathing: gentle chest rise/fall
|
||||||
|
const bFreq = 0.25 + bPow * 0.5; // ~15 bpm base
|
||||||
|
const bAmp = 0.004 + bPow * 0.008;
|
||||||
|
const bPhase = Math.sin(t * bFreq * Math.PI * 2);
|
||||||
|
if (i >= 5 && i <= 10) { ay += bPhase * bAmp; }
|
||||||
|
if (i <= 4) ay += bPhase * bAmp * 0.3;
|
||||||
|
|
||||||
|
// Very subtle sway
|
||||||
|
ax += Math.sin(t * 0.35) * 0.004;
|
||||||
|
az += Math.cos(t * 0.25) * 0.002;
|
||||||
|
|
||||||
|
if (mot === 'active') {
|
||||||
|
const ws = 1.8 + mPow * 2;
|
||||||
|
const wa = 0.03 + mPow * 0.06;
|
||||||
|
const ph = t * ws;
|
||||||
|
|
||||||
|
// Legs
|
||||||
|
if (i === 13) { az += Math.sin(ph) * wa * 0.7; ay -= Math.abs(Math.sin(ph)) * 0.015; }
|
||||||
|
if (i === 14) { az += Math.sin(ph + Math.PI) * wa * 0.7; ay -= Math.abs(Math.sin(ph + Math.PI)) * 0.015; }
|
||||||
|
if (i === 15) { az += Math.sin(ph - 0.2) * wa * 0.8; }
|
||||||
|
if (i === 16) { az += Math.sin(ph + Math.PI - 0.2) * wa * 0.8; }
|
||||||
|
|
||||||
|
// Arms counter-swing (subtle)
|
||||||
|
if (i === 7) az += Math.sin(ph + Math.PI) * wa * 0.35;
|
||||||
|
if (i === 8) az += Math.sin(ph) * wa * 0.35;
|
||||||
|
if (i === 9) az += Math.sin(ph + Math.PI) * wa * 0.45;
|
||||||
|
if (i === 10) az += Math.sin(ph) * wa * 0.45;
|
||||||
|
|
||||||
|
// Tiny vertical bob
|
||||||
|
ay += Math.abs(Math.sin(ph)) * 0.006;
|
||||||
|
|
||||||
|
} else if (mot === 'present_still') {
|
||||||
|
const it = t * 0.25;
|
||||||
|
// Very subtle weight shift
|
||||||
|
if (i >= 11) ax += Math.sin(it * 0.4) * 0.004;
|
||||||
|
// Barely perceptible hand drift
|
||||||
|
if (i === 9) { ax += Math.sin(it * 0.8) * 0.005; }
|
||||||
|
if (i === 10) { ax += Math.sin(it * 0.6 + 0.5) * 0.005; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
targetKps[i].set(ax, ay, az);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Smooth interpolation (lower = smoother, less jumpy)
|
||||||
|
const lerpA = 0.04;
|
||||||
|
for (let i = 0; i < 17; i++) lerp3(smoothKps[i], targetKps[i], lerpA);
|
||||||
|
|
||||||
|
// --- Head ---
|
||||||
|
headM.visible = headGlowM.visible = show;
|
||||||
|
if (show) {
|
||||||
|
tmpA.copy(smoothKps[0]).add(new THREE.Vector3(0, 0.06, 0));
|
||||||
|
headM.position.copy(tmpA);
|
||||||
|
headGlowM.position.copy(tmpA);
|
||||||
|
(headM.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.9;
|
||||||
|
headGlowMat.opacity = alpha * 0.08;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Eyes + pupils
|
||||||
|
eyeL.visible = eyeR.visible = pupilL.visible = pupilR.visible = show;
|
||||||
|
if (show) {
|
||||||
|
const headPos = headM.position;
|
||||||
|
eyeL.position.set(headPos.x - 0.032, headPos.y + 0.01, headPos.z + 0.09);
|
||||||
|
eyeR.position.set(headPos.x + 0.032, headPos.y + 0.01, headPos.z + 0.09);
|
||||||
|
pupilL.position.set(eyeL.position.x, eyeL.position.y, eyeL.position.z + 0.012);
|
||||||
|
pupilR.position.set(eyeR.position.x, eyeR.position.y, eyeR.position.z + 0.012);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Neck
|
||||||
|
neckM.visible = show;
|
||||||
|
if (show) {
|
||||||
|
const neckTop = new THREE.Vector3().copy(smoothKps[0]).add(new THREE.Vector3(0, -0.04, 0));
|
||||||
|
const neckBot = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5).add(new THREE.Vector3(0, 0.04, 0));
|
||||||
|
neckM.position.addVectors(neckTop, neckBot).multiplyScalar(0.5);
|
||||||
|
neckM.scale.y = neckTop.distanceTo(neckBot) * 4;
|
||||||
|
(neckM.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Torso
|
||||||
|
torsoM.visible = torsoGlowM.visible = show;
|
||||||
|
if (show) {
|
||||||
|
const mSh = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5);
|
||||||
|
const mHp = tmpB.addVectors(smoothKps[11], smoothKps[12]).multiplyScalar(0.5);
|
||||||
|
const tPos = new THREE.Vector3().addVectors(mSh, mHp).multiplyScalar(0.5);
|
||||||
|
torsoM.position.copy(tPos);
|
||||||
|
torsoGlowM.position.copy(tPos);
|
||||||
|
const bScale = 1 + Math.sin(t * (0.9 + bPow * 4) * Math.PI * 2) * 0.02 * (1 + bPow * 3);
|
||||||
|
torsoM.scale.set(1, 1, bScale);
|
||||||
|
(torsoM.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.88;
|
||||||
|
torsoGlowMat.opacity = alpha * 0.06;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Hands
|
||||||
|
handL.visible = handR.visible = show;
|
||||||
|
if (show) {
|
||||||
|
handL.position.copy(smoothKps[9]).add(new THREE.Vector3(0, -0.04, 0));
|
||||||
|
handR.position.copy(smoothKps[10]).add(new THREE.Vector3(0, -0.04, 0));
|
||||||
|
(handL.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||||
|
(handR.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Feet
|
||||||
|
footL.visible = footR.visible = show;
|
||||||
|
if (show) {
|
||||||
|
footL.position.copy(smoothKps[15]).add(new THREE.Vector3(0, 0.02, 0.04));
|
||||||
|
footR.position.copy(smoothKps[16]).add(new THREE.Vector3(0, 0.02, 0.04));
|
||||||
|
(footL.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||||
|
(footR.material as THREE.MeshPhysicalMaterial).opacity = alpha * 0.85;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Limb capsules — emissive reacts to motion intensity
|
||||||
|
BODY_SEGS.forEach(([ai, bi, rT, rB], idx) => {
|
||||||
|
limbMs[idx].visible = limbGlowMs[idx].visible = show;
|
||||||
|
if (show) {
|
||||||
|
positionLimb(limbMs[idx], smoothKps[ai], smoothKps[bi], rT, rB);
|
||||||
|
positionLimb(limbGlowMs[idx], smoothKps[ai], smoothKps[bi], rT * 1.6, rB * 1.6);
|
||||||
|
const limbMat = limbMs[idx].material as THREE.MeshPhysicalMaterial;
|
||||||
|
limbMat.opacity = alpha * 0.82;
|
||||||
|
// Glow brighter with more motion (direct sensor feedback)
|
||||||
|
limbMat.emissiveIntensity = 0.06 + mPow * 0.4;
|
||||||
|
const glowMat = limbGlowMs[idx].material as THREE.MeshPhysicalMaterial;
|
||||||
|
glowMat.opacity = alpha * (0.06 + mPow * 0.15);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Joint dots & skeleton lines
|
||||||
|
jDots.forEach((d, i) => { d.visible = show; if (show) d.position.copy(smoothKps[i]); });
|
||||||
|
skelLines.forEach(({ line, a, b }) => {
|
||||||
|
line.visible = show;
|
||||||
|
if (show) {
|
||||||
|
const p = line.geometry.attributes.position as THREE.BufferAttribute;
|
||||||
|
p.setXYZ(0, smoothKps[a].x, smoothKps[a].y, smoothKps[a].z);
|
||||||
|
p.setXYZ(1, smoothKps[b].x, smoothKps[b].y, smoothKps[b].z);
|
||||||
|
p.needsUpdate = true;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Heart ring
|
||||||
|
const vs = fr?.vital_signs as Record<string, unknown> | undefined;
|
||||||
|
const hrBpm = Number(vs?.hr_proxy_bpm ?? vs?.heart_rate_bpm ?? 0);
|
||||||
|
hrRing.visible = show && hrBpm > 0;
|
||||||
|
if (hrRing.visible) {
|
||||||
|
const chst = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5);
|
||||||
|
chst.y -= 0.08;
|
||||||
|
hrRing.position.copy(chst);
|
||||||
|
hrRing.lookAt(camera.position);
|
||||||
|
const bp = (t * (hrBpm / 60) * Math.PI * 2) % (Math.PI * 2);
|
||||||
|
const beat = Math.pow(Math.max(0, Math.sin(bp)), 10);
|
||||||
|
hrMat.opacity = beat * 0.5 * alpha;
|
||||||
|
hrRing.scale.setScalar(1 + beat * 0.12);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Breathing rings
|
||||||
|
brRings.forEach((ring, ri) => {
|
||||||
|
ring.visible = show && bPow > 0.01;
|
||||||
|
if (ring.visible) {
|
||||||
|
const chst = tmpA.addVectors(smoothKps[5], smoothKps[6]).multiplyScalar(0.5);
|
||||||
|
chst.y -= 0.05;
|
||||||
|
ring.position.copy(chst);
|
||||||
|
ring.lookAt(camera.position);
|
||||||
|
const bph = Math.sin(t * (0.9 + bPow * 4) * Math.PI * 2 - ri * 0.5);
|
||||||
|
(ring.material as THREE.MeshBasicMaterial).opacity = Math.max(0, bph * 0.2 * alpha);
|
||||||
|
ring.scale.setScalar(1 + bph * 0.08);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// WiFi pulse rings
|
||||||
|
wifiRings.forEach((wr, wi) => {
|
||||||
|
const phase = (t * 0.5 + wi * 0.4) % 1;
|
||||||
|
wr.scale.setScalar(0.8 + phase * 1.5 + mPow);
|
||||||
|
(wr.material as THREE.MeshBasicMaterial).opacity = (1 - phase) * 0.12 * (pres ? 1 : 0.3);
|
||||||
|
});
|
||||||
|
|
||||||
|
// ESP32 nodes
|
||||||
|
(fr?.nodes || []).forEach((n, i) => {
|
||||||
|
if (i < nodeMs.length) {
|
||||||
|
const [px, py, pz] = n.position;
|
||||||
|
nodeMs[i].position.set(px * 2, py + 0.12, pz * 2);
|
||||||
|
nodeMs[i].visible = true; nodeMs[i].rotation.y = t * 0.4 + i;
|
||||||
|
(nodeMs[i].material as THREE.MeshStandardMaterial).emissiveIntensity = 0.5 + Math.sin(t * 3 + i) * 0.3;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
for (let i = (fr?.nodes || []).length; i < nodeMs.length; i++) nodeMs[i].visible = false;
|
||||||
|
|
||||||
|
// Signal field
|
||||||
|
const sf = fr?.signal_field;
|
||||||
|
if (sf?.values?.length) {
|
||||||
|
const gx = sf.grid_size[0], gz = sf.grid_size[2];
|
||||||
|
for (let zi = 0; zi < Math.min(gz, GS); zi++) for (let xi = 0; xi < Math.min(gx, GS); xi++) {
|
||||||
|
const v = sf.values[zi * gx + xi] || 0;
|
||||||
|
if (v < 0.25) hc.setRGB(0.03, 0.05 + v * 1.8, 0.08 + v * 1.8);
|
||||||
|
else if (v < 0.5) hc.setRGB(0.03, 0.2 + (v - 0.25) * 2.4, 0.5 - (v - 0.25) * 1.2);
|
||||||
|
else if (v < 0.75) hc.setRGB((v - 0.5) * 4, 0.7 + (v - 0.5) * 0.6, 0.1);
|
||||||
|
else hc.setRGB(1, 1 - (v - 0.75) * 3, 0.05);
|
||||||
|
sigGrid.setColorAt(zi * GS + xi, hc);
|
||||||
|
}
|
||||||
|
if (sigGrid.instanceColor) sigGrid.instanceColor.needsUpdate = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Lighting follows data
|
||||||
|
rim.intensity = 0.8 + Math.abs(rssi + 50) * 0.015;
|
||||||
|
|
||||||
|
// Particles
|
||||||
|
const pp = pGeo.attributes.position as THREE.BufferAttribute;
|
||||||
|
for (let i = 0; i < NP; i++) {
|
||||||
|
(pp.array as Float32Array)[i * 3 + 1] += Math.sin(t * 0.8 + i * 0.5) * 0.0006 + mPow * 0.001;
|
||||||
|
if ((pp.array as Float32Array)[i * 3 + 1] > 3.5) (pp.array as Float32Array)[i * 3 + 1] = 0;
|
||||||
}
|
}
|
||||||
pp.needsUpdate = true;
|
pp.needsUpdate = true;
|
||||||
|
|
||||||
// Update skeleton from frame data
|
// HUD
|
||||||
const currentFrame = frameRef.current;
|
const ctx = hudC.getContext('2d');
|
||||||
if (currentFrame) {
|
if (ctx && fr) {
|
||||||
const persons = (currentFrame as any).persons || [];
|
ctx.clearRect(0, 0, 640, 128);
|
||||||
if (persons.length > 0) {
|
ctx.font = 'bold 14px "SF Mono", Menlo, monospace';
|
||||||
const kps = persons[0].keypoints || [];
|
ctx.fillStyle = '#32b8c6';
|
||||||
kps.forEach((kp: any, i: number) => {
|
ctx.fillText(`WIFI-DENSEPOSE [${(fr.source || '--').toUpperCase()}]`, 12, 20);
|
||||||
if (i < 17 && joints[i]) {
|
ctx.font = '12px "SF Mono", Menlo, monospace';
|
||||||
joints[i].position.set(
|
ctx.fillStyle = '#7799aa';
|
||||||
(kp.x - 0.5) * 4,
|
ctx.fillText(`Nodes: ${(fr.nodes || []).length} RSSI: ${rssi.toFixed(1)} dBm Motion: ${mot} Conf: ${(conf * 100).toFixed(0)}%`, 12, 42);
|
||||||
(1 - kp.y) * 3,
|
if (vs) {
|
||||||
(kp.z || 0) * 2,
|
const br = Number(vs.breathing_bpm ?? vs.breathing_rate_bpm ?? 0);
|
||||||
);
|
if (br > 0 || hrBpm > 0) {
|
||||||
joints[i].visible = kp.confidence > 0.3;
|
ctx.fillStyle = '#44ddaa';
|
||||||
(joints[i].material as THREE.MeshStandardMaterial).emissiveIntensity =
|
ctx.fillText(`Breathing: ${br.toFixed(1)} bpm Heart: ${hrBpm.toFixed(1)} bpm`, 12, 62);
|
||||||
0.3 + kp.confidence * 0.7;
|
}
|
||||||
}
|
|
||||||
});
|
|
||||||
boneLines.forEach(({ line, a, b }) => {
|
|
||||||
if (joints[a].visible && joints[b].visible) {
|
|
||||||
const pos = line.geometry.attributes.position as THREE.BufferAttribute;
|
|
||||||
pos.setXYZ(0, joints[a].position.x, joints[a].position.y, joints[a].position.z);
|
|
||||||
pos.setXYZ(1, joints[b].position.x, joints[b].position.y, joints[b].position.z);
|
|
||||||
pos.needsUpdate = true;
|
|
||||||
line.visible = true;
|
|
||||||
} else {
|
|
||||||
line.visible = false;
|
|
||||||
}
|
|
||||||
});
|
|
||||||
} else {
|
|
||||||
joints.forEach((j) => { j.visible = false; });
|
|
||||||
boneLines.forEach((bl) => { bl.line.visible = false; });
|
|
||||||
}
|
}
|
||||||
|
if (show) {
|
||||||
// Adjust light from RSSI
|
ctx.fillStyle = pres ? (mot === 'active' ? '#ff8844' : '#44bbcc') : '#556677';
|
||||||
const features = (currentFrame as any).features;
|
const mBar = Math.min(20, Math.round(mPow * 40));
|
||||||
if (features) {
|
const mBarStr = '\u2588'.repeat(mBar) + '\u2591'.repeat(20 - mBar);
|
||||||
const rssi = features.mean_rssi || -70;
|
ctx.fillText(`Motion: [${mBarStr}] ${(mPow * 100).toFixed(0)}%`, 12, 82);
|
||||||
pointLight.intensity = 1 + Math.abs(rssi + 50) * 0.02;
|
ctx.fillStyle = '#556677';
|
||||||
|
ctx.font = '10px "SF Mono", Menlo, monospace';
|
||||||
|
ctx.fillText('Pose: procedural (load NN model for limb tracking)', 12, 100);
|
||||||
}
|
}
|
||||||
|
hudT.needsUpdate = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
renderer.render(scene, camera);
|
renderer.render(scene, camera);
|
||||||
|
|
||||||
// FPS counter
|
state.fCount++;
|
||||||
state.frameCount++;
|
if (performance.now() - state.fpsT >= 1000) {
|
||||||
if (performance.now() - state.lastFpsTime >= 1000) {
|
onFps(state.fCount); state.fCount = 0; state.fpsT = performance.now();
|
||||||
onFps(state.frameCount);
|
|
||||||
state.frameCount = 0;
|
|
||||||
state.lastFpsTime = performance.now();
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
@ -280,15 +678,10 @@ export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: Gaus
|
||||||
onReady();
|
onReady();
|
||||||
|
|
||||||
return () => {
|
return () => {
|
||||||
canvas.removeEventListener('mousedown', onMouseDown);
|
cvs.removeEventListener('mousedown', () => {});
|
||||||
canvas.removeEventListener('mouseup', onMouseUp);
|
window.removeEventListener('resize', onR);
|
||||||
canvas.removeEventListener('mousemove', onMouseMove);
|
|
||||||
canvas.removeEventListener('wheel', onWheel);
|
|
||||||
window.removeEventListener('resize', onResize);
|
|
||||||
cleanup();
|
cleanup();
|
||||||
if (container.contains(renderer.domElement)) {
|
if (container.contains(renderer.domElement)) container.removeChild(renderer.domElement);
|
||||||
container.removeChild(renderer.domElement);
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
onError(err instanceof Error ? err.message : 'Failed to initialize 3D renderer');
|
onError(err instanceof Error ? err.message : 'Failed to initialize 3D renderer');
|
||||||
|
|
@ -298,19 +691,10 @@ export const GaussianSplatWebViewWeb = ({ onReady, onFps, onError, frame }: Gaus
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<View style={styles.container}>
|
<View style={styles.container}>
|
||||||
<div
|
<div ref={containerRef} style={{ width: '100%', height: '100%', backgroundColor: '#080c16' }} />
|
||||||
ref={containerRef}
|
|
||||||
style={{ width: '100%', height: '100%', backgroundColor: '#0a0e1a' }}
|
|
||||||
/>
|
|
||||||
</View>
|
</View>
|
||||||
);
|
);
|
||||||
};
|
};
|
||||||
|
|
||||||
const styles = StyleSheet.create({
|
const styles = StyleSheet.create({ container: { flex: 1, backgroundColor: '#080c16' } });
|
||||||
container: {
|
|
||||||
flex: 1,
|
|
||||||
backgroundColor: '#0a0e1a',
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
export default GaussianSplatWebViewWeb;
|
export default GaussianSplatWebViewWeb;
|
||||||
|
|
|
||||||
|
|
@ -32,11 +32,30 @@ export interface SignalField {
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface VitalsData {
|
export interface VitalsData {
|
||||||
breathing_bpm: number;
|
breathing_bpm?: number;
|
||||||
hr_proxy_bpm: number;
|
hr_proxy_bpm?: number;
|
||||||
|
// Rust sensing server uses these field names
|
||||||
|
breathing_rate_bpm?: number;
|
||||||
|
breathing_confidence?: number;
|
||||||
|
heart_rate_bpm?: number;
|
||||||
|
heart_confidence?: number;
|
||||||
|
confidence?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PoseKeypoint {
|
||||||
|
name?: string;
|
||||||
|
x: number;
|
||||||
|
y: number;
|
||||||
|
z: number;
|
||||||
confidence: number;
|
confidence: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export interface PersonDetection {
|
||||||
|
id?: number;
|
||||||
|
confidence: number;
|
||||||
|
keypoints: PoseKeypoint[];
|
||||||
|
}
|
||||||
|
|
||||||
export interface SensingFrame {
|
export interface SensingFrame {
|
||||||
type?: string;
|
type?: string;
|
||||||
timestamp?: number;
|
timestamp?: number;
|
||||||
|
|
@ -47,4 +66,8 @@ export interface SensingFrame {
|
||||||
classification: Classification;
|
classification: Classification;
|
||||||
signal_field: SignalField;
|
signal_field: SignalField;
|
||||||
vital_signs?: VitalsData;
|
vital_signs?: VitalsData;
|
||||||
|
pose_keypoints?: [number, number, number, number][];
|
||||||
|
persons?: PersonDetection[];
|
||||||
|
posture?: string;
|
||||||
|
signal_quality_score?: number;
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,8 +4,9 @@
|
||||||
* Manages the connection to the Python sensing WebSocket server
|
* Manages the connection to the Python sensing WebSocket server
|
||||||
* (ws://localhost:8765) and provides a callback-based API for the UI.
|
* (ws://localhost:8765) and provides a callback-based API for the UI.
|
||||||
*
|
*
|
||||||
* Falls back to simulated data if the server is unreachable so the UI
|
* Falls back to simulated data only after MAX_RECONNECT_ATTEMPTS exhausted.
|
||||||
* always shows something.
|
* While reconnecting the service stays in "reconnecting" state and does NOT
|
||||||
|
* emit simulated frames so the UI can clearly distinguish live vs. fallback data.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
// Derive WebSocket URL from the page origin so it works on any port
|
// Derive WebSocket URL from the page origin so it works on any port
|
||||||
|
|
@ -14,7 +15,10 @@ const _wsProto = (typeof window !== 'undefined' && window.location.protocol ===
|
||||||
const _wsHost = (typeof window !== 'undefined' && window.location.host) ? window.location.host : 'localhost:3000';
|
const _wsHost = (typeof window !== 'undefined' && window.location.host) ? window.location.host : 'localhost:3000';
|
||||||
const SENSING_WS_URL = `${_wsProto}//${_wsHost}/ws/sensing`;
|
const SENSING_WS_URL = `${_wsProto}//${_wsHost}/ws/sensing`;
|
||||||
const RECONNECT_DELAYS = [1000, 2000, 4000, 8000, 16000];
|
const RECONNECT_DELAYS = [1000, 2000, 4000, 8000, 16000];
|
||||||
const MAX_RECONNECT_ATTEMPTS = 10;
|
const MAX_RECONNECT_ATTEMPTS = 20;
|
||||||
|
// Number of failed attempts that must occur before simulation starts.
|
||||||
|
// This prevents the UI from flashing "SIMULATED" on a brief hiccup.
|
||||||
|
const SIM_FALLBACK_AFTER_ATTEMPTS = 5;
|
||||||
const SIMULATION_INTERVAL = 500; // ms
|
const SIMULATION_INTERVAL = 500; // ms
|
||||||
|
|
||||||
class SensingService {
|
class SensingService {
|
||||||
|
|
@ -26,7 +30,10 @@ class SensingService {
|
||||||
this._reconnectAttempt = 0;
|
this._reconnectAttempt = 0;
|
||||||
this._reconnectTimer = null;
|
this._reconnectTimer = null;
|
||||||
this._simTimer = null;
|
this._simTimer = null;
|
||||||
this._state = 'disconnected'; // disconnected | connecting | connected | simulated
|
// Connection state: disconnected | connecting | connected | reconnecting | simulated
|
||||||
|
this._state = 'disconnected';
|
||||||
|
// Data-source label exposed to the UI: "live" | "reconnecting" | "simulated"
|
||||||
|
this._dataSource = 'reconnecting';
|
||||||
this._lastMessage = null;
|
this._lastMessage = null;
|
||||||
|
|
||||||
// Ring buffer of recent RSSI values for sparkline
|
// Ring buffer of recent RSSI values for sparkline
|
||||||
|
|
@ -76,6 +83,16 @@ class SensingService {
|
||||||
return this._state;
|
return this._state;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Current data source label.
|
||||||
|
* "live" — frames are arriving from the real ESP32 over WebSocket
|
||||||
|
* "reconnecting" — WebSocket disconnected; actively retrying, no frames emitted
|
||||||
|
* "simulated" — max reconnect attempts exhausted; emitting synthetic frames
|
||||||
|
*/
|
||||||
|
get dataSource() {
|
||||||
|
return this._dataSource;
|
||||||
|
}
|
||||||
|
|
||||||
// ---- Connection --------------------------------------------------------
|
// ---- Connection --------------------------------------------------------
|
||||||
|
|
||||||
_connect() {
|
_connect() {
|
||||||
|
|
@ -96,6 +113,7 @@ class SensingService {
|
||||||
this._reconnectAttempt = 0;
|
this._reconnectAttempt = 0;
|
||||||
this._stopSimulation();
|
this._stopSimulation();
|
||||||
this._setState('connected');
|
this._setState('connected');
|
||||||
|
this._setDataSource('live');
|
||||||
};
|
};
|
||||||
|
|
||||||
this._ws.onmessage = (evt) => {
|
this._ws.onmessage = (evt) => {
|
||||||
|
|
@ -118,28 +136,33 @@ class SensingService {
|
||||||
this._scheduleReconnect();
|
this._scheduleReconnect();
|
||||||
} else {
|
} else {
|
||||||
this._setState('disconnected');
|
this._setState('disconnected');
|
||||||
|
this._setDataSource('reconnecting');
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
_scheduleReconnect() {
|
_scheduleReconnect() {
|
||||||
if (this._reconnectAttempt >= MAX_RECONNECT_ATTEMPTS) {
|
if (this._reconnectAttempt >= MAX_RECONNECT_ATTEMPTS) {
|
||||||
console.warn('[Sensing] Max reconnect attempts reached, switching to simulation');
|
console.warn('[Sensing] Max reconnect attempts (%d) reached, switching to simulation', MAX_RECONNECT_ATTEMPTS);
|
||||||
this._fallbackToSimulation();
|
this._fallbackToSimulation();
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const delay = RECONNECT_DELAYS[Math.min(this._reconnectAttempt, RECONNECT_DELAYS.length - 1)];
|
const delay = RECONNECT_DELAYS[Math.min(this._reconnectAttempt, RECONNECT_DELAYS.length - 1)];
|
||||||
this._reconnectAttempt++;
|
this._reconnectAttempt++;
|
||||||
console.info('[Sensing] Reconnecting in %dms (attempt %d)', delay, this._reconnectAttempt);
|
console.info('[Sensing] Reconnecting in %dms (attempt %d/%d)', delay, this._reconnectAttempt, MAX_RECONNECT_ATTEMPTS);
|
||||||
|
|
||||||
|
this._setState('reconnecting');
|
||||||
|
this._setDataSource('reconnecting');
|
||||||
|
|
||||||
this._reconnectTimer = setTimeout(() => {
|
this._reconnectTimer = setTimeout(() => {
|
||||||
this._reconnectTimer = null;
|
this._reconnectTimer = null;
|
||||||
this._connect();
|
this._connect();
|
||||||
}, delay);
|
}, delay);
|
||||||
|
|
||||||
// Start simulation while waiting
|
// Only start simulation after several failed attempts so a brief hiccup
|
||||||
if (this._state !== 'simulated') {
|
// does not immediately switch the UI to "SIMULATED DATA".
|
||||||
|
if (this._reconnectAttempt >= SIM_FALLBACK_AFTER_ATTEMPTS && this._state !== 'simulated') {
|
||||||
this._fallbackToSimulation();
|
this._fallbackToSimulation();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -148,6 +171,7 @@ class SensingService {
|
||||||
|
|
||||||
_fallbackToSimulation() {
|
_fallbackToSimulation() {
|
||||||
this._setState('simulated');
|
this._setState('simulated');
|
||||||
|
this._setDataSource('simulated');
|
||||||
if (this._simTimer) return; // already running
|
if (this._simTimer) return; // already running
|
||||||
console.info('[Sensing] Running in simulation mode');
|
console.info('[Sensing] Running in simulation mode');
|
||||||
|
|
||||||
|
|
@ -196,6 +220,9 @@ class SensingService {
|
||||||
type: 'sensing_update',
|
type: 'sensing_update',
|
||||||
timestamp: t,
|
timestamp: t,
|
||||||
source: 'simulated',
|
source: 'simulated',
|
||||||
|
// Explicit machine-readable marker so the UI can always detect simulated
|
||||||
|
// frames regardless of which code path produced them.
|
||||||
|
_simulated: true,
|
||||||
nodes: [{
|
nodes: [{
|
||||||
node_id: 1,
|
node_id: 1,
|
||||||
rssi_dbm: baseRssi + Math.sin(t * 0.5) * 3,
|
rssi_dbm: baseRssi + Math.sin(t * 0.5) * 3,
|
||||||
|
|
@ -262,6 +289,21 @@ class SensingService {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Update the dataSource label and notify state listeners so the UI can
|
||||||
|
* react without needing a separate subscription.
|
||||||
|
* @param {'live'|'reconnecting'|'simulated'} source
|
||||||
|
*/
|
||||||
|
_setDataSource(source) {
|
||||||
|
if (source === this._dataSource) return;
|
||||||
|
this._dataSource = source;
|
||||||
|
// Re-use the same state-listener channel — listeners receive the
|
||||||
|
// connection state but can read dataSource via service.dataSource.
|
||||||
|
for (const cb of this._stateListeners) {
|
||||||
|
try { cb(this._state); } catch (e) { /* ignore */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
_clearTimers() {
|
_clearTimers() {
|
||||||
this._stopSimulation();
|
this._stopSimulation();
|
||||||
if (this._reconnectTimer) {
|
if (this._reconnectTimer) {
|
||||||
|
|
|
||||||
51
ui/style.css
51
ui/style.css
|
|
@ -1754,6 +1754,11 @@ canvas {
|
||||||
background: var(--color-error);
|
background: var(--color-error);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.sensing-dot.reconnecting {
|
||||||
|
background: var(--color-warning);
|
||||||
|
animation: pulse 1.5s infinite;
|
||||||
|
}
|
||||||
|
|
||||||
.sensing-source {
|
.sensing-source {
|
||||||
margin-left: auto;
|
margin-left: auto;
|
||||||
font-size: var(--font-size-xs);
|
font-size: var(--font-size-xs);
|
||||||
|
|
@ -1761,6 +1766,52 @@ canvas {
|
||||||
font-family: var(--font-family-mono);
|
font-family: var(--font-family-mono);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.sensing-about-text {
|
||||||
|
margin: 0;
|
||||||
|
font-size: 12px;
|
||||||
|
color: #aaa;
|
||||||
|
line-height: 1.5;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sensing-about-text strong {
|
||||||
|
color: #ccc;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Data-source status banner (live / reconnecting / simulated) */
|
||||||
|
.sensing-source-banner {
|
||||||
|
display: block;
|
||||||
|
width: 100%;
|
||||||
|
padding: var(--space-8) var(--space-12);
|
||||||
|
margin-bottom: var(--space-12);
|
||||||
|
border-radius: var(--radius-md);
|
||||||
|
font-size: var(--font-size-sm);
|
||||||
|
font-weight: var(--font-weight-semibold);
|
||||||
|
font-family: var(--font-family-mono);
|
||||||
|
text-align: center;
|
||||||
|
letter-spacing: 0.06em;
|
||||||
|
text-transform: uppercase;
|
||||||
|
box-sizing: border-box;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sensing-source-live {
|
||||||
|
background: rgba(0, 204, 136, 0.15);
|
||||||
|
border: 1px solid #00cc88;
|
||||||
|
color: #00cc88;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sensing-source-reconnecting {
|
||||||
|
background: rgba(255, 180, 0, 0.12);
|
||||||
|
border: 1px solid var(--color-warning);
|
||||||
|
color: var(--color-warning);
|
||||||
|
animation: pulse 1.5s infinite;
|
||||||
|
}
|
||||||
|
|
||||||
|
.sensing-source-simulated {
|
||||||
|
background: rgba(255, 60, 60, 0.12);
|
||||||
|
border: 1px solid var(--color-error);
|
||||||
|
color: var(--color-error);
|
||||||
|
}
|
||||||
|
|
||||||
/* Big RSSI value */
|
/* Big RSSI value */
|
||||||
.sensing-big-value {
|
.sensing-big-value {
|
||||||
font-size: var(--font-size-3xl);
|
font-size: var(--font-size-3xl);
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue