wifi-densepose/vendor/midstream/AIMDS/docs/ARCHITECTURE.md

AIMDS Architecture

System Overview

AIMDS (AI Memory & Defense System) is a multi-layered security gateway that combines high-performance vector search with formal verification to provide sub-10ms threat detection with mathematical guarantees.

Core Components

1. API Gateway (TypeScript/Express)

Location: src/gateway/

The Express-based gateway provides:

• RESTful API endpoints
• Security middleware (Helmet, CORS, rate limiting)
• Request validation (Zod schemas)
• Response formatting and error handling

Key Files:

• server.ts - Main gateway class
• router.ts - Route definitions
• middleware.ts - Custom middleware

2. AgentDB Client (TypeScript)

Location: src/agentdb/

High-performance vector database client with:

• HNSW indexing (150x faster than brute force)
• Reflexion memory for self-learning
• QUIC synchronization for distributed deployments
• MMR (Maximal Marginal Relevance) for diverse results

Key Files:

• client.ts - Main database client
• vector-search.ts - Search algorithms
• reflexion.ts - Memory system

3. lean-agentic Verifier (TypeScript)

Location: src/lean-agentic/

Formal verification engine with:

• Hash-consed dependent types (150x faster equality)
• Theorem proving with proof certificates
• Type checking for policy constraints
• Cache for proof reuse

Key Files:

• verifier.ts - Main verification engine
• hash-cons.ts - Hash-consing implementation
• theorem-prover.ts - Proof generation

4. Monitoring System (TypeScript)

Location: src/monitoring/

Comprehensive observability with:

• Prometheus metrics
• Winston logging
• Performance tracking
• Health checks

Key Files:

• metrics.ts - Metrics collection
• telemetry.ts - Logging and events

5. Rust Core Libraries

Location: crates/

Native Rust implementations for performance-critical operations:

• reflexion-memory - Core memory system
• lean-agentic - WASM-compiled verification
• agentdb-core - Vector operations

Request Flow

Fast Path (<10ms)

Request
  ↓
1. Express Gateway (validation)
  ↓
2. Generate Embedding (hash-based, <1ms)
  ↓
3. AgentDB Vector Search (HNSW, <2ms)
  ↓
4. Calculate Threat Level (<1ms)
  ↓
5. Low Risk? → Allow & Store Incident

Deep Path (<520ms)

Request
  ↓
1-4. Same as Fast Path
  ↓
5. High Risk?
  ↓
6. Hash-Cons Check (optional, <5ms)
  ↓
7. Dependent Type Check (<50ms)
  ↓
8. Rule Evaluation (<100ms)
  ↓
9. Constraint Checking (<100ms)
  ↓
10. Theorem Proving (optional, <250ms)
  ↓
11. Generate Proof Certificate
  ↓
12. Allow/Deny & Store with Proof

Data Flow

Vector Search Pipeline

Request → Embedding (384-dim) → HNSW Index
                                      ↓
                                  Top-K Results
                                      ↓
                                  MMR Diversity
                                      ↓
                              ThreatMatch Objects

Verification Pipeline

Action + Policy → Hash-Cons Cache? → Cache Hit: Return
                        ↓
                  Cache Miss
                        ↓
              Dependent Type Check
                        ↓
              Rule Evaluation
                        ↓
              Constraint Checking
                        ↓
              Theorem Proving?
                        ↓
              Proof Certificate

Memory Storage Pipeline

Incident → Vector Embedding
              ↓
         AgentDB Insert
              ↓
    ┌────────┴────────┐
    ↓                 ↓
Threat Patterns   Reflexion Memory
    ↓                 ↓
Update Index     Self-Critique
                      ↓
                Learning Loop

Database Schema

AgentDB Collections

threat_patterns:

{
  embedding: vector(384),
  metadata: {
    patternId: string,
    description: string,
    threatLevel: enum,
    firstSeen: timestamp,
    lastSeen: timestamp,
    occurrences: number
  }
}

incidents:

{
  id: string,
  timestamp: number,
  request: AIMDSRequest,
  result: DefenseResult,
  embedding: vector(384)
}

reflexion_memory:

{
  trajectory: string,
  verdict: "success" | "failure",
  feedback: string,
  embedding: vector(384),
  metadata: object
}

causal_graph:

{
  from: string,
  to: string,
  timestamp: number,
  weight: number
}

Security Layers

Layer 1: Express Middleware

• Helmet security headers
• CORS protection
• Rate limiting (configurable)
• Body size limits
• Request timeout

Layer 2: Input Validation

• Zod schema validation
• Type checking
• Sanitization
• Parameter validation

Layer 3: Vector Search

• Fast similarity matching
• Pattern recognition
• Historical threat detection
• Anomaly detection

Layer 4: Formal Verification

• Policy compliance checking
• Temporal logic verification
• Behavioral analysis
• Dependency validation

Layer 5: Proof Certificates

• Mathematical guarantees
• Audit trail
• Cryptographic hashing
• Dependency tracking

Performance Optimizations

1. HNSW Index

• 150x faster than brute force search
• Configurable M (neighbors) and ef (search breadth)
• Cache-friendly data structures

2. Hash-Consing

• 150x faster equality checks
• Structural sharing
• Pointer comparison

3. Caching Strategy

• Proof certificate cache (LRU)
• Hash-cons cache
• Query result cache
• Size-limited caches

4. Parallel Processing

• Concurrent database operations
• Promise.all for independent tasks
• Worker threads for CPU-intensive ops

5. Memory Management

• TTL-based cleanup
• Configurable memory limits
• Periodic garbage collection
• Efficient data structures

Scaling Strategy

Horizontal Scaling

• Stateless gateway instances
• Load balancer distribution
• Shared AgentDB via QUIC sync

Vertical Scaling

• Multi-threaded request handling
• WASM for CPU-intensive ops
• Optimized data structures

Database Scaling

• QUIC peer synchronization
• Sharding by threat pattern type
• Read replicas for queries
• Write leader for updates

Monitoring & Observability

Metrics

• Request latency (p50, p95, p99)
• Throughput (req/s)
• Error rates
• Threat detection rates
• Cache hit rates
• Database performance

Logging

• Structured JSON logs
• Log levels (debug, info, warn, error)
• Request tracing
• Error stack traces

Health Checks

• Component status
• Database connectivity
• Cache health
• Memory usage
• Uptime tracking

Deployment Architecture

Development

Local Machine
  ├── TypeScript (ts-node)
  ├── AgentDB (file-based)
  └── lean-agentic (WASM)

Production

Load Balancer
  ↓
Gateway Instances (3+)
  ↓
AgentDB Cluster (QUIC sync)
  ↓
Persistent Storage (SSD)

Docker Compose

services:
  - gateway (Express)
  - agentdb (vector DB)
  - prometheus (metrics)
  - grafana (dashboards)

Kubernetes

Deployments:
  - gateway (replicas: 3)
  - agentdb (statefulset)

Services:
  - gateway-lb (LoadBalancer)
  - agentdb-headless

ConfigMaps:
  - gateway-config
  - agentdb-config

Future Enhancements

1. GPU Acceleration: CUDA for vector operations
2. Distributed Tracing: OpenTelemetry integration
3. Machine Learning: Adaptive threat models
4. Multi-Region: Geographic distribution
5. Real-time Analytics: Stream processing
6. Advanced Proofs: More complex theorem proving
7. Auto-Scaling: Dynamic resource allocation
8. Circuit Breakers: Fault tolerance

References

• AgentDB Documentation
• lean-agentic Specification
• HNSW Algorithm
• Reflexion Memory