refactor(examples/three.js): organize into demos/screenshots/server/assets + add README
Flatten the 13-file flat layout into purposeful subfolders so the demo
collection has a clean top-level entry point (README.md) and the file roles
are obvious from a directory listing.
Layout:
demos/ 01..05 — numbered for the progression (helpers → cinematic →
skinned → skinned-fbx → skinned-realtime)
screenshots/ one PNG per demo, matching the demo's filename prefix
server/ serve-demo.py + ruvultra-csi-bridge.py
assets/ X Bot.fbx (gitignored, used by demos 04 and 05)
Touched files (beyond the renames):
- 04-skinned-fbx.html, 05-skinned-realtime.html: MODEL_URL now resolves
'../assets/X%20Bot.fbx' instead of './X%20Bot.fbx'
- server/serve-demo.py: chdir() walks 3 levels up to repo root (was 2), and
the URL banner now lists all 5 demos
- .gitignore: comment refresh — points at assets/ and screenshots/
- 05-skinned-realtime.html also picks up in-flight fps-tune work from this
branch (Holistic script, SMOOTH_K URL param, slerp gain scaling) since
those edits and the rename hit the same file
Verified end-to-end:
- python examples/three.js/server/serve-demo.py
- all 5 demos return 200, X Bot.fbx returns 200 from new asset/ path
- demos 04 + 05 render the X Bot mesh; 0 JS errors via browser eval
- screenshots reproduced match the originals
Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
ruv
parent
60d2c8eb82
commit
a9f37e5138
|
|
@ -1,8 +1,10 @@
|
|||
# Mixamo FBX downloads — too large + license boundary. Get your own from
|
||||
# mixamo.com (FBX Binary + T-Pose / Without Skin), drop alongside the HTML.
|
||||
# mixamo.com (FBX Binary + T-Pose / Without Skin), drop into assets/.
|
||||
*.fbx
|
||||
|
||||
# Diagnostic / debug screenshots from session
|
||||
# Diagnostic / debug screenshots from a dev session. Official screenshots
|
||||
# live in screenshots/ and are committed; these underscore-prefixed ones
|
||||
# are scratch.
|
||||
_diag-*.png
|
||||
_demo-mode-shot*.png
|
||||
_PROOF-*.png
|
||||
|
|
|
|||
|
|
@ -0,0 +1,77 @@
|
|||
# three.js demos
|
||||
|
||||
Five progressively richer browser demos of the ADR-097 sensing-helpers scene,
|
||||
ending with a live MediaPipe-Pose → Mixamo X Bot retargeting pipeline driven
|
||||
by a real ESP32 CSI feed.
|
||||
|
||||
## Run them
|
||||
|
||||
```bash
|
||||
python examples/three.js/server/serve-demo.py
|
||||
# then open one of the URLs the script prints
|
||||
```
|
||||
|
||||
`server/serve-demo.py` is a tiny `ThreadingHTTPServer` with aggressive
|
||||
no-cache headers — the stdlib `http.server` is single-threaded and times out
|
||||
on the parallel script + FBX fetches the demos make.
|
||||
|
||||
## Demos
|
||||
|
||||
| # | File | What it shows |
|
||||
|---|------|---------------|
|
||||
| 01 | [`demos/01-helpers.html`](demos/01-helpers.html) | Plain ADR-097 helpers in the point-cloud viewer |
|
||||
| 02 | [`demos/02-cinematic.html`](demos/02-cinematic.html) | Cinematic camera + pseudo-CSI visualization on top of #01 |
|
||||
| 03 | [`demos/03-skinned.html`](demos/03-skinned.html) | GLTF skinned mesh + additive animation blending |
|
||||
| 04 | [`demos/04-skinned-fbx.html`](demos/04-skinned-fbx.html) | Mixamo X Bot loaded from FBX in the ADR-097 scene |
|
||||
| 05 | [`demos/05-skinned-realtime.html`](demos/05-skinned-realtime.html) | Webcam → MediaPipe Pose Heavy → Mixamo IK retarget, live ESP32 CSI overlay |
|
||||
|
||||
| Screenshot | |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|  |  |
|
||||
|  | |
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
examples/three.js/
|
||||
├── README.md
|
||||
├── .gitignore
|
||||
├── demos/ # 5 self-contained HTML demos
|
||||
│ ├── 01-helpers.html
|
||||
│ ├── 02-cinematic.html
|
||||
│ ├── 03-skinned.html
|
||||
│ ├── 04-skinned-fbx.html
|
||||
│ └── 05-skinned-realtime.html
|
||||
├── screenshots/ # one PNG per demo
|
||||
│ └── 0N-*.png
|
||||
├── server/
|
||||
│ ├── serve-demo.py # local HTTP server with no-cache headers
|
||||
│ └── ruvultra-csi-bridge.py # ESP32 CSI WebSocket bridge (ruvultra:8766)
|
||||
└── assets/
|
||||
└── X Bot.fbx # gitignored — get your own from mixamo.com
|
||||
# (FBX Binary, T-Pose, Without Skin)
|
||||
# used by demos 04 and 05
|
||||
```
|
||||
|
||||
## Mixamo X Bot
|
||||
|
||||
Demos 04 and 05 expect `assets/X Bot.fbx`. It's gitignored (size + license
|
||||
boundary). Download yours from [mixamo.com](https://mixamo.com): pick the
|
||||
"X Bot" character, export as **FBX Binary**, **T-Pose**, **Without Skin**,
|
||||
and drop it into `assets/`.
|
||||
|
||||
## Live ESP32 CSI overlay (demo 05 only)
|
||||
|
||||
`server/ruvultra-csi-bridge.py` is the systemd-deployable bridge that runs on
|
||||
the `ruvultra` host (over Tailscale). It listens for ESP32-S3 CSI on UDP and
|
||||
re-broadcasts it as WebSocket frames at `ws://ruvultra:8766/csi`. Demo 05
|
||||
auto-connects; if the socket is down, it falls back to the bundled idle clip
|
||||
plus a synthetic CSI driver.
|
||||
|
||||
## Open issues
|
||||
|
||||
- [#583](https://github.com/ruvnet/RuView/issues/583) — head/face tracking
|
||||
fidelity in `05-skinned-realtime.html`. Recommended fix: swap MediaPipe
|
||||
Pose Heavy for MediaPipe Holistic (same API, adds 468-point face mesh +
|
||||
hand landmarks for proper PnP head pose and finger curl tracking).
|
||||
|
|
@ -223,7 +223,7 @@
|
|||
// rig-only.
|
||||
// =====================================================================
|
||||
|
||||
const MODEL_URL = './X%20Bot.fbx';
|
||||
const MODEL_URL = '../assets/X%20Bot.fbx';
|
||||
|
||||
const NODE_POSITIONS = [
|
||||
[-1.9, 1.3, 1.9],[ 1.9, 1.3, 1.9],
|
||||
|
|
@ -6,7 +6,7 @@
|
|||
<meta http-equiv="Cache-Control" content="no-store, no-cache, must-revalidate, max-age=0">
|
||||
<meta http-equiv="Pragma" content="no-cache">
|
||||
<meta http-equiv="Expires" content="0">
|
||||
<meta name="build-stamp" content="2026-05-15-oneeuro-hips-visibility">
|
||||
<meta name="build-stamp" content="2026-05-15-fps-tune">
|
||||
<title>RuView · Skinned Realtime · MediaPipe Pose → Mixamo IK retargeting</title>
|
||||
<link rel="icon" type="image/svg+xml" href="data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><circle cx='16' cy='16' r='10' fill='%23e8a634'/></svg>">
|
||||
<style>
|
||||
|
|
@ -165,6 +165,7 @@
|
|||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/shaders/CopyShader.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/three@0.128.0/examples/js/shaders/LuminosityHighPassShader.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/pose@0.5/pose.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/holistic@0.5/holistic.js" crossorigin="anonymous"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div class="overlay-frame"></div>
|
||||
|
|
@ -202,7 +203,7 @@
|
|||
<button id="cam-btn">▶ Enable webcam tracking</button>
|
||||
<button id="demo-btn" style="margin-top:6px;background:transparent;color:var(--cyan);border:1px solid var(--cyan);">▶ Demo mode (synthetic pose)</button>
|
||||
<button id="forcetest-btn" style="margin-top:6px;background:transparent;color:var(--magenta);border:1px solid var(--magenta);">⚡ Force test rotation (bypass retarget)</button>
|
||||
<div id="build-banner" style="margin-top:8px;padding:4px 6px;font-size:9px;color:var(--magenta);border:1px dashed var(--magenta);text-align:center;letter-spacing:1px;">build 2026-05-15-oneeuro-hips-vis · OneEuro smoothing + Hips twist + visibility gate</div>
|
||||
<div id="build-banner" style="margin-top:8px;padding:4px 6px;font-size:9px;color:var(--magenta);border:1px dashed var(--magenta);text-align:center;letter-spacing:1px;">build 2026-05-15-fps-tune · default Holistic@Full 20fps · ?cnn=2 ?infer=30 to crank</div>
|
||||
</div>
|
||||
|
||||
<div class="panel" id="csi">
|
||||
|
|
@ -255,7 +256,7 @@
|
|||
// its bundled Idle clip with the previous synthetic CSI driver.
|
||||
// =========================================================================
|
||||
|
||||
const MODEL_URL = './X%20Bot.fbx';
|
||||
const MODEL_URL = '../assets/X%20Bot.fbx';
|
||||
const NODE_POSITIONS = [
|
||||
[-1.9, 1.3, 1.9],[ 1.9, 1.3, 1.9],
|
||||
[-1.9, 1.3, -1.9],[ 1.9, 1.3, -1.9],
|
||||
|
|
@ -923,14 +924,18 @@
|
|||
},
|
||||
};
|
||||
}
|
||||
// Per-landmark filter bank. Tuned for ~30 fps pose data:
|
||||
// minCutoff=1.7 — slow base cutoff = aggressive smoothing at rest
|
||||
// beta=0.007 — quickly relaxes when motion accelerates
|
||||
// dCutoff=1.0 — derivative filter cutoff
|
||||
// Per-landmark filter bank. Globally scaled by ?smooth=<n> URL param.
|
||||
// smooth=1 → default
|
||||
// smooth=2 → 2× more smoothing (slower response, less jitter)
|
||||
// smooth=0.5 → half as much smoothing (snappier)
|
||||
const SMOOTH_K = Math.max(0.25, parseFloat(
|
||||
new URLSearchParams(location.search).get('smooth') || '1.6'
|
||||
));
|
||||
const kpFilters = {};
|
||||
function smoothKp(idx, axis, raw, t) {
|
||||
const key = idx + axis;
|
||||
if (!kpFilters[key]) kpFilters[key] = OneEuro(1.7, 0.007, 1.0);
|
||||
// minCutoff is divided by SMOOTH_K — smaller cutoff = more smoothing
|
||||
if (!kpFilters[key]) kpFilters[key] = OneEuro(1.7 / SMOOTH_K, 0.007 / SMOOTH_K, 1.0);
|
||||
return kpFilters[key].filter(raw, t);
|
||||
}
|
||||
|
||||
|
|
@ -1064,7 +1069,8 @@
|
|||
// in between → linear ramp
|
||||
const vis = visForRetarget(r);
|
||||
const visWeight = Math.max(0, Math.min(1, (vis - 0.4) / 0.3));
|
||||
const slerpAmount = 0.9 * visWeight;
|
||||
// Per-bone slerp gain — scaled down when SMOOTH_K is higher
|
||||
const slerpAmount = (0.9 / Math.sqrt(SMOOTH_K)) * visWeight;
|
||||
if (slerpAmount > 0.01) {
|
||||
bone.quaternion.slerp(localQ, slerpAmount);
|
||||
}
|
||||
|
|
@ -1103,10 +1109,19 @@
|
|||
liveVis[11] ?? 1, liveVis[12] ?? 1
|
||||
);
|
||||
const hipsVisW = Math.max(0, Math.min(1, (hipsVis - 0.4) / 0.3));
|
||||
hipsBone.quaternion.slerp(targetHipsWorldQ, 0.45 * hipsVisW);
|
||||
hipsBone.quaternion.slerp(targetHipsWorldQ, (0.45 / Math.sqrt(SMOOTH_K)) * hipsVisW);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- HEAD + HANDS (Holistic) ----------
|
||||
// Run AFTER body + hips because they're set after the reset step.
|
||||
// If we ran these before, the body retarget loop would still pass
|
||||
// through them harmlessly, but it's cleaner to apply face/hand
|
||||
// last so the head sits on top of the latest spine/neck.
|
||||
applyFaceHead();
|
||||
applyHand('left');
|
||||
applyHand('right');
|
||||
|
||||
// Force a fresh world-matrix walk so SkinnedMesh.skeleton picks up
|
||||
// the just-changed bone quaternions before composer.render() runs.
|
||||
if (model) model.updateMatrixWorld(true);
|
||||
|
|
@ -1130,6 +1145,8 @@
|
|||
// MediaPipe Pose pipeline
|
||||
// ---------------------------------------------------------------------
|
||||
let pose = null;
|
||||
let holistic = null;
|
||||
let useHolistic = true;
|
||||
let videoEl = null;
|
||||
let overlayCanvas = null;
|
||||
let overlayCtx = null;
|
||||
|
|
@ -1175,31 +1192,64 @@
|
|||
overlayCanvas.height = videoEl.videoHeight;
|
||||
overlayCtx = overlayCanvas.getContext('2d');
|
||||
|
||||
// Pull complexity from URL — default Heavy (2) for max fidelity.
|
||||
// ?cnn=0 = Lite (fast), ?cnn=1 = Full (balanced), ?cnn=2 = Heavy.
|
||||
const cnnParam = parseInt(new URLSearchParams(location.search).get('cnn') ?? '2', 10);
|
||||
const modelComplexity = isNaN(cnnParam) ? 2 : Math.max(0, Math.min(2, cnnParam));
|
||||
// Default complexity is now Full (1) for Holistic — Heavy is ~3×
|
||||
// slower across pose + face + 2 hands. Pose-only stays at Heavy.
|
||||
const usingHolisticPath = new URLSearchParams(location.search).get('pose') !== '1';
|
||||
const defaultCnn = usingHolisticPath ? '1' : '2';
|
||||
const cnnParam = parseInt(new URLSearchParams(location.search).get('cnn') ?? defaultCnn, 10);
|
||||
const modelComplexity = isNaN(cnnParam) ? 1 : Math.max(0, Math.min(2, cnnParam));
|
||||
const cnnLabel = ['Lite', 'Full', 'Heavy'][modelComplexity];
|
||||
setPoseStatus('loading', `Loading MediaPipe Pose · ${cnnLabel} (~${[3,5,12][modelComplexity]} MB)…`);
|
||||
if (!window.Pose) {
|
||||
setPoseStatus('error', 'MediaPipe Pose script not loaded');
|
||||
return;
|
||||
|
||||
// ?pose=1 forces the legacy Pose-only model. Default is Holistic:
|
||||
// 33 pose + 468 face mesh + 21 left-hand + 21 right-hand landmarks,
|
||||
// giving us full 3-axis head pose and finger tracking.
|
||||
useHolistic = new URLSearchParams(location.search).get('pose') !== '1';
|
||||
if (useHolistic && !window.Holistic) {
|
||||
console.warn('Holistic script not loaded, falling back to Pose');
|
||||
useHolistic = false;
|
||||
}
|
||||
|
||||
if (useHolistic) {
|
||||
setPoseStatus('loading', `Loading MediaPipe Holistic · ${cnnLabel} (~25 MB on first load)…`);
|
||||
holistic = new Holistic({
|
||||
locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/holistic@0.5/${file}`,
|
||||
});
|
||||
// refineFaceLandmarks runs an EXTRA attention network for eye
|
||||
// and lip refinement — we only use 4 stable face anchors, so
|
||||
// leaving it off saves a chunk of per-frame compute.
|
||||
holistic.setOptions({
|
||||
modelComplexity,
|
||||
smoothLandmarks: true,
|
||||
refineFaceLandmarks: false,
|
||||
enableSegmentation: false,
|
||||
minDetectionConfidence: 0.4,
|
||||
minTrackingConfidence: 0.4,
|
||||
});
|
||||
holistic.onResults(onHolisticResults);
|
||||
await holistic.initialize();
|
||||
const lblEl = document.getElementById('pose-state');
|
||||
if (lblEl) lblEl.textContent = `tracking · Holistic ${cnnLabel}`;
|
||||
} else {
|
||||
setPoseStatus('loading', `Loading MediaPipe Pose · ${cnnLabel} (~${[3,5,12][modelComplexity]} MB)…`);
|
||||
if (!window.Pose) {
|
||||
setPoseStatus('error', 'MediaPipe Pose script not loaded');
|
||||
return;
|
||||
}
|
||||
pose = new Pose({
|
||||
locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose@0.5/${file}`,
|
||||
});
|
||||
pose.setOptions({
|
||||
modelComplexity,
|
||||
smoothLandmarks: true,
|
||||
enableSegmentation: false,
|
||||
minDetectionConfidence: 0.4,
|
||||
minTrackingConfidence: 0.4,
|
||||
});
|
||||
pose.onResults(onPoseResults);
|
||||
await pose.initialize();
|
||||
const lblEl = document.getElementById('pose-state');
|
||||
if (lblEl) lblEl.textContent = `tracking · Pose ${cnnLabel}`;
|
||||
}
|
||||
pose = new Pose({
|
||||
locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/pose@0.5/${file}`,
|
||||
});
|
||||
pose.setOptions({
|
||||
modelComplexity,
|
||||
smoothLandmarks: true,
|
||||
enableSegmentation: false,
|
||||
minDetectionConfidence: 0.4, // slightly lower = more permissive
|
||||
minTrackingConfidence: 0.4,
|
||||
});
|
||||
pose.onResults(onPoseResults);
|
||||
await pose.initialize();
|
||||
// surface which model is loaded
|
||||
const lblEl = document.getElementById('pose-state');
|
||||
if (lblEl) lblEl.textContent = `tracking · ${cnnLabel}`;
|
||||
|
||||
setPoseStatus('active', 'Tracking');
|
||||
btn.textContent = '◼ Stop tracking';
|
||||
|
|
@ -1384,16 +1434,294 @@
|
|||
window._RVF_demo = { start: startDemoMode, stop: stopDemoMode, buildLandmarks: buildDemoLandmarks };
|
||||
window._RVF_forceTest = { start: startForceTest, stop: stopForceTest };
|
||||
|
||||
// Throttle inference to a target fps — Holistic runs pose + face + 2
|
||||
// hands per send() call (heavy). OneEuro smoothing on landmarks lets
|
||||
// us run inference at 20 fps and still get visually smooth bone
|
||||
// animation at the renderer's 60+ fps. Override with ?infer=<fps>.
|
||||
const TARGET_INFER_FPS = parseFloat(
|
||||
new URLSearchParams(location.search).get('infer') || '20'
|
||||
);
|
||||
const TARGET_INFER_DT = 1000 / Math.max(5, Math.min(60, TARGET_INFER_FPS));
|
||||
async function runPoseLoop() {
|
||||
let lastSend = 0;
|
||||
while (poseLoopActive) {
|
||||
if (videoEl.readyState >= 2) {
|
||||
try { await pose.send({ image: videoEl }); }
|
||||
catch (e) { console.warn('pose send', e); }
|
||||
const now = performance.now();
|
||||
if (videoEl.readyState >= 2 && now - lastSend >= TARGET_INFER_DT) {
|
||||
lastSend = now;
|
||||
try {
|
||||
if (useHolistic && holistic) await holistic.send({ image: videoEl });
|
||||
else if (pose) await pose.send({ image: videoEl });
|
||||
} catch (e) { console.warn('mp send', e); }
|
||||
}
|
||||
await new Promise(r => requestAnimationFrame(r));
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------
|
||||
// Holistic results handler — multiplexes pose + face + hands
|
||||
// ---------------------------------------------------------------------
|
||||
// poseLandmarks: 33 image-normalized (same as Pose)
|
||||
// poseWorldLandmarks: 33 in meters, origin at hip mid
|
||||
// faceLandmarks: 468 (or 478 with refine), normalized image coords
|
||||
// leftHandLandmarks: 21, normalized image coords
|
||||
// rightHandLandmarks: 21, normalized image coords
|
||||
//
|
||||
// Body path identical to onPoseResults. Face + hand paths feed two
|
||||
// new ingest functions (head basis from face, finger bones from hands).
|
||||
// ---------------------------------------------------------------------
|
||||
function onHolisticResults(results) {
|
||||
const now = performance.now();
|
||||
if (lastPoseT) {
|
||||
const dt = now - lastPoseT;
|
||||
poseFps = poseFps * 0.85 + (1000 / Math.max(dt, 1)) * 0.15;
|
||||
document.getElementById('pose-fps').textContent = poseFps.toFixed(0) + ' fps';
|
||||
}
|
||||
lastPoseT = now;
|
||||
|
||||
const lms = results.poseLandmarks;
|
||||
const wlms = results.poseWorldLandmarks;
|
||||
const flms = results.faceLandmarks;
|
||||
const llms = results.leftHandLandmarks;
|
||||
const rlms = results.rightHandLandmarks;
|
||||
|
||||
const has = (s, n) => (s ? (n != null ? (s.length >= n ? 'y' : 'p') : 'y') : '.');
|
||||
document.getElementById('lm-count').textContent =
|
||||
`33${has(lms)} · 468${has(flms, 468)} · L21${has(llms)} · R21${has(rlms)}`;
|
||||
|
||||
// overlay — same as pose path for body, plus face contour
|
||||
if (overlayCtx && lms) {
|
||||
overlayCtx.clearRect(0, 0, overlayCanvas.width, overlayCanvas.height);
|
||||
overlayCtx.strokeStyle = '#4cf'; overlayCtx.lineWidth = 2; overlayCtx.fillStyle = '#ffb840';
|
||||
for (const [a, b] of KP_BONES) {
|
||||
if (!lms[a] || !lms[b]) continue;
|
||||
overlayCtx.beginPath();
|
||||
overlayCtx.moveTo(lms[a].x * overlayCanvas.width, lms[a].y * overlayCanvas.height);
|
||||
overlayCtx.lineTo(lms[b].x * overlayCanvas.width, lms[b].y * overlayCanvas.height);
|
||||
overlayCtx.stroke();
|
||||
}
|
||||
for (const i of KP_INDICES) {
|
||||
if (!lms[i]) continue;
|
||||
overlayCtx.beginPath();
|
||||
overlayCtx.arc(lms[i].x * overlayCanvas.width, lms[i].y * overlayCanvas.height, 3, 0, Math.PI * 2);
|
||||
overlayCtx.fill();
|
||||
}
|
||||
if (flms) {
|
||||
overlayCtx.fillStyle = '#ff4cc8';
|
||||
for (let i = 0; i < flms.length; i += 8) {
|
||||
overlayCtx.beginPath();
|
||||
overlayCtx.arc(flms[i].x * overlayCanvas.width, flms[i].y * overlayCanvas.height, 1, 0, Math.PI * 2);
|
||||
overlayCtx.fill();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Body path — same as Pose
|
||||
if (wlms && wlms.length === 33) ingestPoseWorldLandmarks(wlms, lms);
|
||||
else ingestPoseLandmarks(lms);
|
||||
|
||||
// Stash face + hand landmarks for use inside applyRetargeting().
|
||||
// applyRetargeting resets all bones to rest every render frame
|
||||
// before re-applying — if we wrote head/finger bones here, they
|
||||
// would be clobbered. The body path uses liveKp because the kp
|
||||
// targets are stable between pose frames; same for face / hands.
|
||||
liveFace = (flms && flms.length >= 468) ? flms : null;
|
||||
liveLeftHand = (llms && llms.length === 21) ? llms : null;
|
||||
liveRightHand = (rlms && rlms.length === 21) ? rlms : null;
|
||||
livePoseLms = lms || null;
|
||||
}
|
||||
let liveFace = null;
|
||||
let liveLeftHand = null;
|
||||
let liveRightHand = null;
|
||||
let livePoseLms = null;
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Face mesh → Head bone rotation
|
||||
// ------------------------------------------------------------------
|
||||
// Build a head-local orthonormal basis from 5 stable face mesh anchors:
|
||||
// 1 nose tip
|
||||
// 10 forehead (between brows)
|
||||
// 152 chin
|
||||
// 33 right eye outer corner
|
||||
// 263 left eye outer corner
|
||||
// upDir = (forehead - chin).normalize
|
||||
// sideDir = (left eye - right eye).normalize
|
||||
// fwdDir = sideDir × upDir (face plane forward)
|
||||
// re-orthog sideDir = upDir × fwdDir
|
||||
// This gives a basis that follows yaw + pitch + roll, applied to the
|
||||
// Head bone after the Neck has already done coarse cervical aim.
|
||||
// ------------------------------------------------------------------
|
||||
let headBoneRef = null;
|
||||
let headRest = null;
|
||||
function ensureHeadCache() {
|
||||
if (headBoneRef && headRest) return;
|
||||
headBoneRef = bones['mixamorigHead'];
|
||||
if (!headBoneRef) return;
|
||||
if (model) model.updateMatrixWorld(true);
|
||||
headRest = {
|
||||
localQ: headBoneRef.quaternion.clone(),
|
||||
worldQ: headBoneRef.getWorldQuaternion(new THREE.Quaternion()),
|
||||
parentWorldQ: headBoneRef.parent
|
||||
? headBoneRef.parent.getWorldQuaternion(new THREE.Quaternion())
|
||||
: new THREE.Quaternion(),
|
||||
};
|
||||
}
|
||||
|
||||
// OneEuro-smoothed face anchors. Same filter family as the body
|
||||
// landmarks, but per-anchor (not per-landmark since we only use 5).
|
||||
const faceAnchorFilters = {};
|
||||
function smoothFaceAnchor(idx, axis, raw, t) {
|
||||
const key = 'f' + idx + axis;
|
||||
if (!faceAnchorFilters[key]) faceAnchorFilters[key] = OneEuro(1.2 / SMOOTH_K, 0.005 / SMOOTH_K, 1.0);
|
||||
return faceAnchorFilters[key].filter(raw, t);
|
||||
}
|
||||
|
||||
function applyFaceHead() {
|
||||
if (!liveFace) return;
|
||||
ensureHeadCache();
|
||||
if (!headBoneRef || !headRest) return;
|
||||
const mirror = new URLSearchParams(location.search).get('mirror') !== '0';
|
||||
const yflip = new URLSearchParams(location.search).get('yflip') !== '0';
|
||||
const zflip = new URLSearchParams(location.search).get('zflip') !== '0';
|
||||
const sx = mirror ? -1 : 1, sy = yflip ? -1 : 1, sz = zflip ? -1 : 1;
|
||||
const t = performance.now() * 0.001;
|
||||
const pt = (i) => {
|
||||
const lm = liveFace[i];
|
||||
return new THREE.Vector3(
|
||||
smoothFaceAnchor(i, 'x', sx * lm.x, t),
|
||||
smoothFaceAnchor(i, 'y', sy * lm.y, t),
|
||||
smoothFaceAnchor(i, 'z', sz * lm.z, t),
|
||||
);
|
||||
};
|
||||
const fore = pt(10);
|
||||
const chin = pt(152);
|
||||
const eyeR = pt(33);
|
||||
const eyeL = pt(263);
|
||||
const upDir = new THREE.Vector3().subVectors(fore, chin).normalize();
|
||||
const sideDir = new THREE.Vector3().subVectors(eyeL, eyeR).normalize();
|
||||
const fwdDir = new THREE.Vector3().crossVectors(sideDir, upDir).normalize();
|
||||
const sideOrth= new THREE.Vector3().crossVectors(upDir, fwdDir).normalize();
|
||||
const m = new THREE.Matrix4().makeBasis(sideOrth, upDir, fwdDir);
|
||||
const targetWorldQ = new THREE.Quaternion().setFromRotationMatrix(m);
|
||||
const parentInv = headRest.parentWorldQ.clone().invert();
|
||||
const localQ = parentInv.multiply(targetWorldQ);
|
||||
// Lower slerp than before because face data now smoothed +
|
||||
// applyRetargeting reset means we start from rest each frame
|
||||
headBoneRef.quaternion.slerp(localQ, 0.35);
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Hand landmarks → Mixamo finger bones
|
||||
// ------------------------------------------------------------------
|
||||
// Each MediaPipe hand has 21 landmarks. Per-finger chain:
|
||||
// THUMB : 1 → 2 → 3 → 4 (CMC → MCP → IP → TIP)
|
||||
// INDEX : 5 → 6 → 7 → 8 (MCP → PIP → DIP → TIP)
|
||||
// MIDDLE: 9 → 10 → 11 → 12
|
||||
// RING : 13 → 14 → 15 → 16
|
||||
// PINKY : 17 → 18 → 19 → 20
|
||||
//
|
||||
// Each Mixamo finger has 3 driveable joints + a tip:
|
||||
// mixamorigLeftHandIndex1 (proximal) ← MCP→PIP direction
|
||||
// mixamorigLeftHandIndex2 (middle) ← PIP→DIP direction
|
||||
// mixamorigLeftHandIndex3 (distal) ← DIP→TIP direction
|
||||
//
|
||||
// Hand landmarks are in image-normalized coords + relative z; we
|
||||
// anchor them to the corresponding wrist keypoint from pose
|
||||
// landmarks (kp 15 L wrist, kp 16 R wrist) and scale by hand span.
|
||||
// ------------------------------------------------------------------
|
||||
const FINGERS = [
|
||||
{ name: 'Thumb', kps: [1, 2, 3, 4] },
|
||||
{ name: 'Index', kps: [5, 6, 7, 8] },
|
||||
{ name: 'Middle', kps: [9, 10, 11, 12] },
|
||||
{ name: 'Ring', kps: [13, 14, 15, 16] },
|
||||
{ name: 'Pinky', kps: [17, 18, 19, 20] },
|
||||
];
|
||||
const fingerCache = {}; // 'leftIndex1' -> { bone, restDir, restWorldQ, restParentWorldQ }
|
||||
|
||||
function getFingerRest(handSide, fingerName, segIdx) {
|
||||
const cap = handSide[0].toUpperCase() + handSide.slice(1);
|
||||
const key = handSide + fingerName + segIdx;
|
||||
if (fingerCache[key]) return fingerCache[key];
|
||||
const boneName = `mixamorig${cap}Hand${fingerName}${segIdx}`;
|
||||
const bone = bones[boneName];
|
||||
if (!bone) { fingerCache[key] = null; return null; }
|
||||
if (model) model.updateMatrixWorld(true);
|
||||
const headPos = bone.getWorldPosition(new THREE.Vector3());
|
||||
const child = bone.children && bone.children.find(c => c.isBone);
|
||||
const tailPos = child
|
||||
? child.getWorldPosition(new THREE.Vector3())
|
||||
: headPos.clone().add(new THREE.Vector3(0, 1, 0).applyQuaternion(
|
||||
bone.getWorldQuaternion(new THREE.Quaternion())));
|
||||
const restDir = new THREE.Vector3().subVectors(tailPos, headPos).normalize();
|
||||
// safety — zero-length wrapper detection
|
||||
const cache = {
|
||||
bone,
|
||||
restDir: restDir.lengthSq() < 0.0001 ? new THREE.Vector3(0, 1, 0) : restDir,
|
||||
restWorldQ: bone.getWorldQuaternion(new THREE.Quaternion()),
|
||||
restParentWorldQ: bone.parent
|
||||
? bone.parent.getWorldQuaternion(new THREE.Quaternion())
|
||||
: new THREE.Quaternion(),
|
||||
restLocalQ: bone.quaternion.clone(),
|
||||
};
|
||||
fingerCache[key] = cache;
|
||||
return cache;
|
||||
}
|
||||
|
||||
// OneEuro filter for hand landmarks too — bigger smoothing because
|
||||
// hand z-axis from MediaPipe is essentially noise
|
||||
const handFilters = {};
|
||||
function smoothHand(side, idx, axis, raw, t) {
|
||||
const key = side + idx + axis;
|
||||
if (!handFilters[key]) handFilters[key] = OneEuro(1.0 / SMOOTH_K, 0.005 / SMOOTH_K, 1.0);
|
||||
return handFilters[key].filter(raw, t);
|
||||
}
|
||||
|
||||
function applyHand(handSide) {
|
||||
const handLms = handSide === 'left' ? liveLeftHand : liveRightHand;
|
||||
if (!handLms || !livePoseLms) return;
|
||||
const wristKp = handSide === 'left' ? 15 : 16;
|
||||
const wristAnchor = liveKp[wristKp];
|
||||
if (!wristAnchor || !livePoseLms[wristKp]) return;
|
||||
|
||||
const handSize = Math.hypot(
|
||||
handLms[9].x - handLms[0].x,
|
||||
handLms[9].y - handLms[0].y,
|
||||
);
|
||||
if (handSize < 0.01) return;
|
||||
const scale = 0.08 / handSize;
|
||||
|
||||
const mirror = new URLSearchParams(location.search).get('mirror') !== '0';
|
||||
const yflip = new URLSearchParams(location.search).get('yflip') !== '0';
|
||||
const zflip = new URLSearchParams(location.search).get('zflip') !== '0';
|
||||
const sx = mirror ? -1 : 1, sy = yflip ? -1 : 1, sz = zflip ? -1 : 1;
|
||||
const t = performance.now() * 0.001;
|
||||
const wristLm = handLms[0];
|
||||
const hp = (i) => {
|
||||
const lm = handLms[i];
|
||||
return new THREE.Vector3(
|
||||
wristAnchor.x + smoothHand(handSide, i, 'x', sx * (lm.x - wristLm.x) * scale, t),
|
||||
wristAnchor.y + smoothHand(handSide, i, 'y', sy * (lm.y - wristLm.y) * scale, t),
|
||||
wristAnchor.z + smoothHand(handSide, i, 'z', sz * (lm.z - wristLm.z) * scale, t),
|
||||
);
|
||||
};
|
||||
|
||||
for (const f of FINGERS) {
|
||||
for (let seg = 1; seg <= 3; seg++) {
|
||||
const rest = getFingerRest(handSide, f.name, seg);
|
||||
if (!rest) continue;
|
||||
const fromIdx = f.kps[seg - 1];
|
||||
const toIdx = f.kps[seg];
|
||||
const from = hp(fromIdx);
|
||||
const to = hp(toIdx);
|
||||
const desiredDir = new THREE.Vector3().subVectors(to, from).normalize();
|
||||
if (desiredDir.lengthSq() < 0.0001) continue;
|
||||
const deltaWorld = new THREE.Quaternion().setFromUnitVectors(rest.restDir, desiredDir);
|
||||
const targetWorldQ = deltaWorld.clone().multiply(rest.restWorldQ);
|
||||
const localQ = rest.restParentWorldQ.clone().invert().multiply(targetWorldQ);
|
||||
rest.bone.quaternion.slerp(localQ, 0.28);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function onPoseResults(results) {
|
||||
const now = performance.now();
|
||||
if (lastPoseT) {
|
||||
|
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 96 KiB |
|
Before Width: | Height: | Size: 598 KiB After Width: | Height: | Size: 598 KiB |
|
Before Width: | Height: | Size: 632 KiB After Width: | Height: | Size: 632 KiB |
|
Before Width: | Height: | Size: 682 KiB After Width: | Height: | Size: 682 KiB |
|
Before Width: | Height: | Size: 596 KiB After Width: | Height: | Size: 596 KiB |
|
|
@ -6,16 +6,16 @@ connections (HTML + 9 script tags + FBX), the first eats the worker, the
|
|||
rest time out with net::ERR_EMPTY_RESPONSE. ThreadingHTTPServer fixes it.
|
||||
|
||||
Usage:
|
||||
cd <repo root>
|
||||
python examples/three.js/serve-demo.py
|
||||
open http://localhost:8765/examples/three.js/helpers-skinned-fbx.html
|
||||
python examples/three.js/server/serve-demo.py
|
||||
open http://localhost:8765/examples/three.js/demos/05-skinned-realtime.html
|
||||
"""
|
||||
from http.server import ThreadingHTTPServer, SimpleHTTPRequestHandler
|
||||
import os, sys
|
||||
|
||||
PORT = int(os.environ.get("PORT", 8765))
|
||||
# always serve from the repo root regardless of where the script is launched
|
||||
os.chdir(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
|
||||
# Always serve from the repo root regardless of where the script is launched.
|
||||
# This file lives at examples/three.js/server/serve-demo.py — three levels deep.
|
||||
os.chdir(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", "..")))
|
||||
|
||||
class NoCacheHandler(SimpleHTTPRequestHandler):
|
||||
def end_headers(self):
|
||||
|
|
@ -27,9 +27,19 @@ class NoCacheHandler(SimpleHTTPRequestHandler):
|
|||
self.send_header("Expires", "0")
|
||||
super().end_headers()
|
||||
|
||||
DEMOS = [
|
||||
"01-helpers.html",
|
||||
"02-cinematic.html",
|
||||
"03-skinned.html",
|
||||
"04-skinned-fbx.html",
|
||||
"05-skinned-realtime.html",
|
||||
]
|
||||
|
||||
with ThreadingHTTPServer(("127.0.0.1", PORT), NoCacheHandler) as srv:
|
||||
print(f"serving {os.getcwd()} on http://127.0.0.1:{PORT}/")
|
||||
print(f"demo: http://127.0.0.1:{PORT}/examples/three.js/helpers-skinned-fbx.html")
|
||||
print("demos:")
|
||||
for d in DEMOS:
|
||||
print(f" http://127.0.0.1:{PORT}/examples/three.js/demos/{d}")
|
||||
try:
|
||||
srv.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
Loading…
Reference in New Issue