wifi-densepose/vendor/midstream/plans/DASHBOARD_README.md

MidStream Real-Time Dashboard

Created by rUv

Comprehensive real-time dashboard for monitoring and analyzing LLM streaming with advanced temporal pattern detection, attractor analysis, and multi-modal stream introspection.

🌟 Features

Real-Time Monitoring

• Text Streaming: Process and analyze text messages in real-time
• Audio Streaming: Monitor audio streams with transcription support
• Video Streaming: Analyze video streams with object detection
• Multi-Modal: Simultaneous handling of text, audio, and video streams

Advanced Analysis

• Temporal Pattern Detection: Identify patterns in conversation flows
• Attractor Analysis: Detect fixed points, periodic cycles, and chaotic behavior
• Lyapunov Exponents: Measure system stability and chaos
• Meta-Learning: Adaptive learning from conversation patterns
• Behavior Classification: Classify system behavior as stable, unstable, or chaotic

Performance Metrics

• Real-Time FPS: Frames per second monitoring
• Latency Tracking: Message processing latency
• Stream Metrics: Bandwidth, bitrate, and chunk statistics
• Token Counting: Track LLM token usage
• Uptime Monitoring: System uptime and health

Streaming Support

• WebSocket: Real-time bidirectional streaming
• Server-Sent Events (SSE): Unidirectional event streaming
• WebRTC: Peer-to-peer audio/video streaming
• RTMP: Real-Time Messaging Protocol support
• HLS: HTTP Live Streaming support

📦 Installation

cd npm
npm install
npm run build:ts

🚀 Quick Start

Basic Dashboard

import { MidStreamDashboard } from 'midstream-cli';

const dashboard = new MidStreamDashboard();
dashboard.start(100); // Refresh every 100ms

// Process a message
dashboard.processMessage('Hello, world!', 5);

// Process streaming data
const audioData = Buffer.alloc(1024);
dashboard.processStream('audio-1', audioData, 'audio');

Interactive Dashboard

import { InteractiveDashboard } from 'midstream-cli';

const dashboard = new InteractiveDashboard();
dashboard.startInteractive();

Run Demo

# Full demo with all features
npm run demo

# Text-only demo
npm run demo:text

# Audio streaming demo
npm run demo:audio

# Video streaming demo
npm run demo:video

# OpenAI Realtime API demo
npm run demo:openai

📊 Dashboard Components

System Metrics Panel

Messages Processed: 150
Total Tokens: 2,340
FPS: 60
Latency: 12ms
Uptime: 0h 5m 23s

Temporal Analysis Panel

Attractor Type: PERIODIC
Lyapunov Exp: -0.0234
Stability: STABLE
Chaos: ORDERED
Avg Reward: 0.847

Pattern Detection Panel

• greeting (95%)
• question (87%)
• acknowledgment (92%)
• follow-up (78%)
• closing (88%)

Streaming Status Panel

Audio: ● ACTIVE
Video: ● ACTIVE
Streams: 3 active

Stream Metrics Panel

audio-stream-1 (audio): 150 chunks, 1.5 MB, 45.2 KB/s
video-stream-1 (video): 1800 frames, 180 MB, 3.2 MB/s

🎥 Restream Integration

WebRTC Streaming

import { RestreamClient } from 'midstream-cli';

const client = new RestreamClient({
  webrtcSignaling: 'wss://signaling.example.com',
  enableTranscription: true,
  enableObjectDetection: true,
  frameRate: 30,
  resolution: '1920x1080'
});

// Listen for frames
client.on('frame', (frame) => {
  console.log(`Frame ${frame.frameNumber}: ${frame.width}x${frame.height}`);
});

// Listen for audio
client.on('audio', (audio) => {
  console.log(`Audio chunk: ${audio.sampleRate}Hz, ${audio.channels}ch`);
});

// Listen for transcriptions
client.on('transcription', (text) => {
  console.log(`Transcription: ${text}`);
});

// Connect
await client.connectWebRTC();

RTMP Streaming

const client = new RestreamClient({
  rtmpUrl: 'rtmp://live.example.com/live',
  streamKey: 'your-stream-key',
  enableTranscription: true
});

await client.connectRTMP();

HLS Streaming

const client = new RestreamClient({
  enableTranscription: true
});

await client.connectHLS('https://example.com/stream.m3u8');

Stream Analysis

// Get real-time analysis
const analysis = client.getAnalysis();

console.log(`
  Frames: ${analysis.frameCount}
  Audio Chunks: ${analysis.audioChunks}
  FPS: ${analysis.fps}
  Bitrate: ${analysis.bitrate} Kbps
  Patterns: ${analysis.patterns.length}
`);

🤖 OpenAI Realtime Integration

import { MidStreamDashboard } from 'midstream-cli';
import { OpenAIRealtimeClient } from 'midstream-cli';

const dashboard = new MidStreamDashboard();
dashboard.start();

const client = new OpenAIRealtimeClient({
  apiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-4o-realtime-preview-2024-10-01',
  voice: 'alloy'
});

// Connect dashboard to OpenAI events
client.on('response.text.delta', (delta) => {
  dashboard.processMessage(delta, delta.length);
});

client.on('response.audio.delta', (delta) => {
  const audio = Buffer.from(delta, 'base64');
  dashboard.processStream('openai-audio', audio, 'audio');
});

await client.connect();
client.sendText('Analyze this conversation...');

🧪 Testing with Stream Simulator

import { StreamSimulator } from 'midstream-cli';

const simulator = new StreamSimulator(30); // 30 FPS

simulator.start(
  (frame) => {
    // Process video frame
    dashboard.processStream('video', frame.data, 'video');
  },
  (audio) => {
    // Process audio chunk
    dashboard.processStream('audio', audio.data, 'audio');
  }
);

// Run for 60 seconds
setTimeout(() => simulator.stop(), 60000);

🔧 Configuration

Dashboard Options

const dashboard = new MidStreamDashboard();

// Custom agent configuration
const agent = dashboard.getAgent();
// Agent is pre-configured with:
// - maxHistory: 1000
// - embeddingDim: 3
// - schedulingPolicy: 'EDF'

Refresh Rate

// Fast refresh (100ms) - smooth but CPU intensive
dashboard.start(100);

// Medium refresh (500ms) - balanced
dashboard.start(500);

// Slow refresh (1000ms) - low CPU usage
dashboard.start(1000);

📈 Advanced Usage

Custom Pattern Analysis

const agent = dashboard.getAgent();

// Detect custom pattern
const pattern = ['greeting', 'question', 'answer'];
const positions = agent.detectPattern(conversation, pattern);

console.log(`Pattern found at positions: ${positions}`);

Sequence Comparison

// Compare two conversation sequences
const similarity = agent.compareSequences(
  sequence1,
  sequence2,
  'dtw' // Dynamic Time Warping
);

console.log(`Similarity: ${(similarity * 100).toFixed(1)}%`);

Behavior Analysis

// Analyze system behavior
const rewards = [0.8, 0.85, 0.83, 0.87, 0.84];
const analysis = agent.analyzeBehavior(rewards);

console.log(`
  Attractor: ${analysis.attractorType}
  Lyapunov: ${analysis.lyapunovExponent}
  Stable: ${analysis.isStable}
  Chaotic: ${analysis.isChaotic}
`);

🎨 Customization

Color Themes

The dashboard uses chalk for colorful console output:

• Cyan: Headers and titles
• Green: Success states and positive metrics
• Yellow: Warnings and neutral metrics
• Red: Errors and negative states
• Magenta: Patterns and detections
• Gray: Secondary information

Custom Metrics

// Get current state
const state = dashboard.getState();

// Modify or extend as needed
console.log(`
  Messages: ${state.messageCount}
  Patterns: ${state.patternsDetected.length}
  Attractor: ${state.attractorType}
`);

🔐 Security Considerations

API Keys

Always use environment variables for API keys:

# .env file
OPENAI_API_KEY=sk-...
AGENTIC_FLOW_API_KEY=...

Stream Authentication

When using WebRTC or RTMP, ensure proper authentication:

const client = new RestreamClient({
  rtmpUrl: 'rtmps://secure.example.com/live', // Use RTMPS
  streamKey: process.env.STREAM_KEY,
  apiKey: process.env.API_KEY
});

Rate Limiting

Implement rate limiting for API calls:

// Limit message processing rate
let lastProcess = 0;
const minInterval = 100; // ms

function processWithRateLimit(message: string) {
  const now = Date.now();
  if (now - lastProcess >= minInterval) {
    dashboard.processMessage(message, message.length);
    lastProcess = now;
  }
}

📊 Performance Optimization

Buffer Management

The dashboard automatically manages buffers:

• Recent messages: Last 5 messages
• Frame buffer: Last 100 frames
• Audio buffer: Last 100 chunks

Memory Usage

Monitor and optimize memory usage:

// Periodic cleanup
setInterval(() => {
  if (global.gc) {
    global.gc();
  }
}, 60000);

CPU Optimization

Adjust refresh rate based on CPU usage:

// Start with fast refresh
dashboard.start(100);

// Reduce if CPU is high
if (cpuUsage > 80) {
  dashboard.stop();
  dashboard.start(500);
}

🐛 Troubleshooting

Dashboard Not Updating

• Check refresh rate is appropriate
• Verify messages are being processed
• Check console for errors

Stream Not Connecting

• Verify URL and credentials
• Check network connectivity
• Review firewall settings

High CPU Usage

• Increase refresh interval
• Reduce stream resolution
• Disable unnecessary features

Memory Leaks

• Check buffer sizes
• Verify event listeners are cleaned up
• Monitor with process.memoryUsage()

📚 API Reference

MidStreamDashboard

Constructor

new MidStreamDashboard()

Methods

• start(refreshRate: number): void - Start dashboard
• stop(): void - Stop dashboard
• processMessage(message: string, tokens?: number): void - Process text message
• processStream(streamId: string, data: Buffer, type: 'audio' | 'video' | 'text'): void - Process stream data
• getAgent(): MidStreamAgent - Get underlying agent
• getState(): DashboardState - Get current state

RestreamClient

Constructor

new RestreamClient(config: RestreamConfig)

Methods

• connectRTMP(): Promise<void> - Connect to RTMP stream
• connectWebRTC(): Promise<void> - Connect to WebRTC stream
• connectHLS(url: string): Promise<void> - Connect to HLS stream
• disconnect(): void - Disconnect from stream
• getAnalysis(): StreamAnalysis - Get stream analysis
• getStats() - Get stream statistics

Events

• connected - Stream connected
• disconnected - Stream disconnected
• frame - Video frame received
• audio - Audio chunk received
• transcription - Audio transcribed
• objects_detected - Objects detected in frame
• error - Error occurred

StreamSimulator

Constructor

new StreamSimulator(frameRate: number)

Methods

• start(onFrame, onAudio?): void - Start simulation
• stop(): void - Stop simulation
• getFrameNumber(): number - Get current frame number

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

MIT License - see LICENSE file for details

👨‍💻 Author

Created by rUv

For questions or support, please open an issue on GitHub.

🙏 Acknowledgments

• OpenAI for Realtime API
• WebRTC community
• Node.js community
• All contributors

---

MidStream Dashboard - Real-time introspection for the AI age