Frontier Plan: Rust, SIMD, WASM and Edge LLM Serving (State of the Art 2026)
State of the Art Context (2026)
KV Cache Bottleneck and New Solutions
KV Cache as the Bottleneck: In modern LLM inference with long contexts and batched requests, the memory footprint of the key-value (KV) cache dominates. For example, in a 540B model (PaLM) at batch size 512 and 2048 token context, the KV cache alone reaches ~3 TB – about 3× the model’s parameters. This memory overhead (and the need to move KV data for each generated token) makes KV caching a primary performance and cost bottleneck for long contexts. Recent work has therefore shifted focus from further weight compression to KV cache optimizations as the key to pushing context lengths and multi-query throughput.
KV Cache Quantization: Techniques for compressing the KV cache have matured to achieve very low precision storage with minimal quality loss, making KV quantization a first-class feature rather than a hack. KVQuant (NeurIPS 2024) demonstrates that with careful handling of outlier entries and structure (e.g. per-channel quantization for keys, isolating per-vector outliers), one can push KV down to ~3-bit precision while keeping perplexity near baseline. KIVI(2024) goes even further: it introduces a tuning-free 2-bit KV cache quantization that treats keys and values asymmetrically (keys quantized per-channel, values per-token) and still “maintains almost the same quality” as FP16, with dramatically lower peak memory usage. These low-bit KV schemes translate directly into higher batch capacity or longer contexts – KIVI’s 2-bit cache enables larger batches and throughput gains in real workloads. Beyond pure quantization, XQuant (2025) takes a different approach: rematerialization. Instead of storing full key/value tensors, XQuant stores a much smaller quantized intermediate (the “X” layer input) and recomputes the K and V on the flyeach step. This trades extra compute for big memory savings – an immediate 2× reduction vs. standard KV caching, and up to ~7.7× less memory with negligible (<0.1) perplexity loss compared to FP16 storage. In practice, systems may also combine strategies (quantize what is stored, recompute some parts when needed) to balance latency and memory. Notably, the open-source vLLM library already implements a pragmatic scheme: it stores KV in a quantized format (e.g. FP8) to roughly halve memory use, but dequantizes back to high precision (FP16/BF16) for the actual attention computations. This way, memory is saved on storage, but the attention math still runs at full precisionfor stability. In summary, the cutting edge has shifted toward aggressive KV cache compression and smart management, since that is now the main limiter for scaling context and concurrency on the edge.
Multi-Adapter LoRA Serving as a Systems Challenge
Fine-tuning large models via LoRA (Low-Rank Adapters) is commonplace for personalization, but serving many different LoRA-tuned variants efficiently is no longer just a training trick – it’s a systems design problem. Naively merging each LoRA into a separate model copy doesn’t scale when each user session might require a different adapter (the memory overhead would be enormous). Recent research addresses this by treating many LoRAs + one base model as a single multi-tenant deployment. S-LoRA (MLSys 2024) is a flagship example: it serves thousands of LoRA adapters concurrently on a single base model instance. The S-LoRA system avoids merging adapters into the weights; instead it keeps all adapter weight matrices in CPU memory and pages in the ones needed for active queries into GPU memory on the fly. To make this feasible, S-LoRA introduces Unified Paging, which uses a unified GPU memory pool to manage both the dynamic adapter weights and the KV cache tensors. This unified pool with a paging strategy prevents memory fragmentation and ensures efficient utilization even as different sessions load and unload different LoRAs. (In other words, the residency of KV cache and LoRA data is handled together as one scheduling/allocation problem.) S-LoRA also employs custom CUDA kernels for heterogeneous batching, so that requests using different LoRA adapters (even with different rank sizes) can still be batched together for throughput. The result is that on-demand personalization is possible at scale: S-LoRA reports up to 4× throughput improvement over naive per-request loading, and it can serve orders of magnitude more adapters than approaches that “merge and deploy” one model per adapter.
Another advance is DoRA (Weight-Decomposed Low-Rank Adaptation) by NVIDIA (ICML 2024). DoRA is a new fine-tuning method that improves on LoRA’s quality without adding any inference overhead. It decomposes each pretrained weight matrix into magnitude and direction components, applies LoRA-style low-rank updates to the directional component, and then merges them back before inference. The key point: DoRA’s adapted weights can be merged into the base model for runtime, so serving a DoRA-tuned model is as cheap as serving the original model (no extra latency or memory), while achieving higher accuracy than standard LoRA. This matters for cases where you can afford separate deployments for fixed domains: you get better quality per parameter, but still keep inference cost “costless” like LoRA (since the low-rank updates don’t add new compute at runtime). In summary, serving personalized models at the edge is now being tackled with system-level optimizations: unified memory management for many adapters (S-LoRA) when you need lots of per-request variants, and improved fine-tuning techniques (DoRA) when you want high-quality adapters that incur no runtime penalty.
WASM at the Edge: Setting the Right Boundaries
WebAssembly (WASM) has emerged as a credible option for sandboxing and deploying code at the network edge, including portions of ML inference pipelines – but where to use WASM is critical. The consensus is that WASM is best used for plugin-like modules or kernels with strict resource budgets, rather than as a full deep learning runtime by itself. In practice, this means heavy-duty math is often left to optimized native libraries, but you can run those libraries behind a WASM interface safely. A notable development here is WASI-NN, a proposed WebAssembly system interface for neural network inference. WASI-NN defines a vendor-neutral API where a WASM module can load a model and ask the host to perform inference. Crucially, the WASI-NN spec is framework-agnostic: it treats the model as an opaque blob and can delegate to any backend – e.g. pass an ONNX model to an ONNX Runtime or OpenVINO backend, TensorFlow model to TF, etc., depending on the host’s implementation. This allows a sandboxed WASM plugin to leverage high-performance native inference engines (like ONNX Runtime, OpenVINO, TensorRT) without exposing the host to risk, since the WASM side only sees a safe API. In edge scenarios, one might use this for less time-critical ML tasks running in a plugin environment or to run classical ML models (vision, etc.) on the fly in a restricted sandbox.
For the core per-token neural network loops of a large language model, however, pure WASM will add overheadunless carefully optimized. The state of the art approach is to keep those tight loops small and efficient, possibly with explicit SIMD use, and to use WASM engine features to enforce execution budgets. For example, Wasmtime (a popular WASM runtime) offers two ways to preempt long-running code: a deterministic fuel mechanism where each WASM operation decrements a counter and traps when fuel is exhausted, and a faster epoch-based interruption(timer-based) – both allow halting runaway code safely. Fuel is essentially a “you only get N instructions” meter, whereas epoch timers act like a timeout. In an edge inference setting, one can compile critical kernels to WASM and use these features to guarantee no kernel exceeds its time budget (preventing latency spikes or infinite loops). Meanwhile, for extremely resource-constrained devices, projects like WAMR (WebAssembly Micro Runtime)provide a tiny footprint WASM runtime. WAMR’s core (interpreted) can be under 85 KB, and even its AOT-compiled mode is ~50 KB binary size, making it suitable for embedding in IoT or mobile deployments. Despite its size, WAMR supports ahead-of-time compilation to native code for speed, achieving near-native performance with minimal runtime overhead. The bottom line: WASM is “credible” for edge ML when used wisely – for example, shipping optimized kernels or small models as sandboxed modules, enforcing limits with fuel/time quotas, and choosing lightweight runtimes for embedded use – rather than trying to run an entire PyTorch/TensorFlow stack inside WASM. This approach gives you safety, flexibility (hot-swapping modules), and portability, without sacrificing much performance.
Rust Ecosystem for Portable ML Inference
Rust has gained traction for ML inference, especially for edge and client-side use, due to its combination of performance, safety, and portability (including easy targeting of WASM). Two reference points illustrate the state of the art in 2026:
	•	Rust LLM engines: mistral.rs is a community-driven Rust inference engine that has made waves with its speed and minimalism. It’s a Rust-native implementation of the Mistral (and other) model families focused on low latency and low resource usage. By leveraging Rust’s zero-cost abstractions and careful optimization (SIMD, quantization, multi-threading), mistral.rs can run chat models in constrained environments (even on CPU or WASM) with performance approaching highly optimized C++ backends. It supports multimodal and instruct models with a fraction of the footprint of larger frameworks. In short, it proves that “pure Rust” inference is viable and even advantageous for edge deployments, offering memory safety and portability without sacrificing speed. Its design choices (e.g. an asynchronous API, support for paged attention, flash attention, etc.) are now influencing other projects.
	•	Rust ML libraries: For building custom inference solutions in Rust, an important building block is Hugging Face’s Candle library. Candle is a minimalist ML framework in Rust that provides tensor operations and model loading, optimized for performance and small binary size. It’s designed to feel like writing PyTorch, but under the hood it’s pure Rust with support for CPU vectorization, GPU (CUDA) kernels, and even WebAssembly as backends. One of Candle’s notable strengths is that it can compile models to WebAssembly and run them in-browser or in a WASM runtime with ease. It also makes it straightforward to integrate or write custom ops in Rust (for example, the library supports plugging in user-defined GPU kernels like FlashAttention v2). This means if you need a particular optimized kernel (say for a new quantization scheme or a custom activation), you can add it in Rust and still have the whole model run end-to-end in a single Rust binary or WASM module. Candle’s focus on performance and flexibility (without a giant runtime dependency) has made it a “clean base” for Rust ML – many experimental Rust LLM projects build on Candle for tensor implementations and then add their own model-specific logic.
In summary, the Rust ecosystem by 2026 offers both high-level engines (like mistral.rs) ready to use for fast inference, and lower-level frameworks (like Candle) for crafting your own solution, all with an eye toward portability (WASM support) and efficiency (zero-copy, SIMD, etc.). These will be the foundation for the next generation of edge ML runtimes.
Updated Architecture Plan for ruvllm (Edge + WASM)
Given the above trends, here is a “frontier” plan to build an edge-focused LLM serving runtime (which we’ll call ruvllm for now) using Rust, vectorized kernels, WASM modules, and latest best practices. The plan is organized in stages:
Core Plan (Immediate Priorities)
	1	Keep Orchestration Native; Make Kernels Pluggable: Use Rust for the high-level orchestration, scheduling, and policy logic, and define a stable Kernel Provider ABI (application binary interface) for the low-level math kernels. This way, the coordinator (token loop, model logic, memory management) stays in safe native Rust, but you can swap out kernel implementations easily – e.g. use highly optimized native Rust kernels, or load alternative implementations (possibly as WASM modules) without changing the core. This aligns with our design goals of auditability and flexibility: the core runtime handles things like policy decisions, logging, and safety checks, while the matrix multiplication, attention, etc., can be provided by plugin modules. In essence, the ruvllmengine should treat kernels as replaceable artifacts (almost like drivers). This was directionally in the original spec; now we double down on it by clearly delineating the boundary and ensuring that, for example, a kernel pack can be distributed or updated independently (e.g. a user could load a SIMD-optimized WASM kernel pack for their CPU). The base system should run with a default kernel set, but allow override via this interface.
	2	Implement Two-Tier KV Cache (Quantization as a First-Class Feature): Based on the state-of-art practices, the runtime must manage the KV cache in a memory-optimized way by default. Concretely, ruvllm should use a two-tier KV cache:
	◦	A small high-precision “tail” buffer that holds the most recent tokens’ KV (e.g. the last $N$ tokens in full float32/16 precision).
	◦	A larger quantized store for older tokens’ KV (using low-bit encoding).
	3	New KV entries start in the high-precision tail, then as the sequence grows and that buffer fills, move older entries into the quantized store. This design ensures that attention for the most recent tokens (which might be most crucial for quality) is done directly on high-precision data, while longer-term history is kept compressed. We will use group-wise quantization with explicit scales for the compressed KV – for example, quantize in blocks (or per head) with a scale/offset, which is exactly the approach shown to preserve quality in KVQuant. Importantly, when performing an attention step, the system will dequantize any needed KV blocks from the store back into a high-precision scratch buffer for the dot-product computation, then discard that scratch after use. This follows the proven pattern from production systems: quantize for storage, dequantize for compute. By doing this, we keep arithmetic precision high where it matters (no accuracy loss in the actual attention math), while memory usage is greatly reduced in steady-state. Initial research suggests that even 3–4 bit precision for KV can work with proper calibration, so ruvllm should make KV compression “on by default.” In essence, KV cache management becomes a core feature of the runtime (with policies to decide when to quantize or not) rather than an afterthought. This matches what both academic work and deployed systems (like vLLM) indicate about KV being the new bottleneck.
	4	Add Mixed-Precision KV Support (per-Component and per-Layer): Not all parts of the KV cache are equal – keys vs. values, and different model layers, have different sensitivity to quantization. The design should allow flexibility in precision per component. For instance, research (e.g. KIVI) found that keys benefit from per-channel quantization and perhaps need a different approach than values. ruvllm should support configuring separate quantization formats for K and V, and even potentially leaving some layers’ KV in higher precision if desired. This means the KV cache API/structs need to carry type information (one layer’s KV might be 8-bit, another 4-bit, etc., or keys in one format, values in another). By making this configurable, the system can implement policies like: “Layers 1–N quantized to 4-bit, final layers 8-bit for fidelity” or “Key caches use a higher precision than Value caches,” etc. Such mixed schemes were hinted by prior studies to improve the quality-memory trade-off. The key point is to treat precision as a tunable policy lever. The runtime core will expose hooks to apply different quantization to different layers or switch format on the fly, while the kernels themselves just see either a dequantized float array or a quantized buffer reference. This separation of concerns (policy in Rust, mechanism in kernels) is crucial. We will implement the infrastructure for it now (even if initial policy is simple). In practice, this might involve defining a small enum for supported precisions (FP16, FP8-E4M3, int4, etc.) and having the KV cache manager know how to transcode between them.
	5	Manage LoRA Adapters with Unified Memory Pooling: Extend the runtime’s memory management to handle LoRA adapter weights using the same strategy as KV scratch buffers. In other words, ruvllm will allocate a unified arena (pool) in memory that is used for:
	◦	Temporarily holding currently active LoRA adapter weights (for whatever personalization is in use this request or batch).
	◦	Staging dequantized KV tensors for attention computation (as described above).
	◦	Other temporary scratch (intermediate results) as needed.
	6	This unified pool prevents fragmentation and simplifies allocation. We can draw inspiration from S-LoRA’s paging system: S-LoRA had one memory pool that managed both adapter tensors and KV tensors to reduce fragmentation and overhead. ruvllm on edge may not have as many concurrent adapters as S-LoRA’s cloud scenario, but the same principle of unified budgeting applies (especially on CPU memory-constrained devices). For example, if multiple sessions with different LoRAs are running, we might swap out adapters from the pool when not in use (similar to paging to CPU or disk if needed), or compress them. We will also consider storing adapters in a compressed form (e.g. 8-bit or in some efficient delta form) when idle, and only materializing the full-weight matrices in the pool when a request that needs them is processing. Essentially, treat adapter weights as another class of cacheable data that can move between memory tiers (RAM, disk) just like KV caches in LMCache. By pooling KV scratch and adapter data together, the runtime can make globally optimal decisions (e.g. evict some long-term KV to make room for a just-activated adapter, or vice versa) with a single allocator. This will ensure we “stop adapters from being a mess” – no more memory blow-ups when many fine-tunes exist, as everything lives in one managed space with predictable limits. We will incorporate mechanisms like LRU or usage-based eviction for this pool, and possibly prefetching of adapters if we anticipate usage.
	7	Deploy Core Kernels as Tiny WASM Modules (Kernel Packs): Following the idea of pluggable kernels, we will ship a default set of critical kernels in a portable WASM form as well. These WASM kernel packs(statically compiled, with SIMD support enabled) will include routines for things like positional embedding (RoPE), layer normalization (RMSNorm), the SwiGLU activation, KV pack/unpack (quantize/dequantize routines), and applying LoRA deltas. Each of these is a small, self-contained function that we can compile to WebAssembly with near-native performance (especially with new WASM SIMD instructions) and then load into our runtime. The benefits are twofold: (a) it provides a sandboxed execution environment for critical code (improving safety when running untrusted or user-supplied models), and (b) it makes the engine highly portable– e.g. you can run the same kernel pack in a Wasmtime environment on x86, or in a WAMR on an ARM microcontroller, without recompiling the Rust core. We will ensure these WASM kernels have a very minimal ABI (e.g. direct pointer to memory, lengths, no complex objects) to reduce overhead when calling from Rust into WASM. Attention: we will not move the entire attention computation into WASM in the initial version. The attention softmax and batched matrix multiply are performance-critical and depend on memory layout – we will keep those native for now (or use an optimized library) until we’ve stabilized everything else. The first WASM kernels will handle the pieces around the edges (embedding, non-linearities, elementwise ops) which are easier to sandbox and unlikely to be the bottleneck. Over time, as we gain confidence and possibly as WASM gains new features (threads, better SIMD), we can consider migrating more into that sandbox.
	8	Tailor the WASM Runtime to Device Class: ruvllm should be able to run on a spectrum of edge devices, so we will adopt different WASM runtime configurations depending on the deployment target:
	◦	Edge servers / x86 Linux: Use Wasmtime with features like epoch interruption (or fuel) to ensure rogue kernels can be stopped. We’ll configure strict memory limits for the WASM instances (so a kernel can’t overflow its allotted memory). This gives us a high-performance JIT for WASM (important for SIMD) and the safety controls we need.
	◦	Embedded devices / microcontrollers: Use WAMR (WebAssembly Micro Runtime) possibly in ahead-of-time (AOT) compiled mode. WAMR’s tiny footprint (on the order of tens of KB) and ability to precompile the WASM to native code make it ideal for constrained hardware. We might not have all features (e.g. no threads), but we won’t need them for these small devices. Essentially, the kernel pack would be compiled to a native blob via WAMR’s AoT compiler and then loaded.
	◦	Optional: WASI Threads where available: If running on a platform that supports WASM threads (and our runtime allows it), we could enable multi-threaded kernels in WASM for things like batched BLAS. But this will be off by default since support is still emerging and platform-specific in 2026.
	9	The runtime will detect or be configured with the target class and choose the appropriate engine. Importantly, the functional behavior is consistent across these – thanks to WASI, the same module can run in Wasmtime or WAMR. This flexibility means we can deploy the same ruvllm engine from a beefy edge box down to an IoT gateway, just swapping the WASM runtime and maybe the kernel pack flavor (some modules might include alternative code paths for different SIMD instruction sets, selected at load time).
The above Core Plan items are the must-haves to get an initial version of ruvllm that is predictable, memory-efficient, and extensible. To summarize, after this stage: KV cache usage will be under control via quantization; adapter overhead will be managed via unified pooling; our system will be modular with safe, portable kernel plugins; and we’ll be equipped to enforce strict execution budgets (no stalls or leaks). These are all foundations for a system that can run continuously at the edge without surprises.
Stretch Plan (Next Phases Once Core is Stable)
	1	WASM-Accelerated Attention Kernels with SIMD: With the core in place (especially the scratch buffer strategy for KV), we can attempt to move the attention computation itself into a WASM kernel for even greater portability. This is challenging because attention involves a lot of data movement and computation (dot products, softmax) on potentially large matrices. However, by the time core is stable, we’ll have a good handle on memory layouts and will have SIMD support. We can write an optimized attention step (for one head or a block of heads) in a low-level language or even use something like portable SIMD libraries, then compile to WASM. The goal is to see minimal performance loss relative to native. If achieved, this means the entire model forward pass could run in the sandbox (important for untrusted code execution scenarios). We will attempt this only after verifying that copying cost (from host memory to WASM memory and back) doesn’t outweigh the benefits. One approach is to use shared memory so that the WASM module can directly read the model weights or KV buffers without copying (Wasmtime supports a mechanism for this in some configurations). Success here would make our engine extremely flexible – imagine downloading a new attention algorithm as a WASM module and plugging it in at runtime.
	2	Adapter-Aware Batching (Multi-Tenant Batching): In the core system, we will already support applying different LoRA adapters to different sessions. A stretch goal is to implement heterogeneous batching across requests that use different adapters. S-LoRA already showed a technique for this by grouping operations and using a bitmask to select adapter contributions per request. Initially, we might serialize different adapter requests separately, but to fully utilize the hardware, we want to batch them when possible. We can design the compute such that, for example, during the forward pass we accumulate the base model output and then for each adapter in the batch, apply its rank-$r$ update (possibly via a fused kernel per adapter). This requires careful scheduling: all requests in the batch share the base computation, then each gets its adapter added. We don’t need the full generality of S-LoRA’s solution at first; we can start with a simpler approach (e.g., limit to 2–4 distinct adapters in a batch and handle them explicitly). The key is that nothing in the core design should prevent this – indeed our unified memory pool and kernel ABI should make it feasible. Achieving this will further improve throughput for multi-tenant scenarios (common on edge boxes that handle requests from different users/models). It also forces our memory manager to be really robust (since multiple adapters must reside in GPU/CPU memory simultaneously during a batch). We’ll take cues from S-LoRA and Punica on how they batch and schedule multi-adapter workloads, ensuring ruvllm can eventually do the same.
	3	Formal KV Cache “Scheduler” Policies: Once basic KV quantization is in place, we can enhance it with smarter dynamic policies. The idea is to make KV cache precision adaptive and the process auditable. For example, the engine could monitor the latency per token (p95 or p99 percentile) and if it’s well under budget, decide to spend extra time compressing older KV blocks (e.g. quantize from 8-bit down to 4-bit) to save memory. Conversely, if certain quality metrics start degrading (or if the model perplexity drifts up as measured on recent tokens), the engine might pause quantization or bump certain layers back to higher precision. This essentially is a KV cache scheduler that decides when and how aggressively to quantize. It should operate deterministically given the same sequence of inputs (for reproducibility). A concrete policy could be: “After every 50 tokens generated, quantize all KV blocks older than 100 tokens to 4-bit unless doing so has previously caused >5% increase in loss, and ensure at least 1 block (tail) remains in FP16”. These rules can be encoded in a policy module. We will also log every quantization change as an event (with layer, precision, reason) so that debugging and auditing is possible – you can see exactly when the engine decided to compress memory and when it reverted. This feature is forward-looking; it draws on the idea of predictable and transparent schedulingcommon in OS design (and hinted at by some LLM cache research). By implementing a simple version, we lay the groundwork for more complex memory management in the future (and build trust – users can see and control the trade-offs). This scheduler could also interface with external signals – e.g., a monitoring agent could instruct our runtime to “enter low-memory mode” which triggers more aggressive KV quantization across the board.
Frontier Plan (Long-Term Innovations)
Finally, if we look further out, there are a few disruptive ideas that could push ruvllm to true state-of-the-art and beyond, especially in low-memory edge environments:
	1	Extreme KV Rematerialization: Build on the concept from XQuant and similar research to implement a mode where ruvllm *does not store all KV activations at all, but instead recomputes or regenerates them as needed. For instance, instead of caching every Key and Value vector, we might cache a smaller “fingerprint” of the intermediate state (like the X that XQuant stores or perhaps a compressed hidden state before projection) and on each new token, recompute the necessary K/V from that. This could drastically reduce memory – potentially enabling context lengths in the millions on edge devices – at the cost of extra compute per token. For devices that have idle compute capacity (or specialized accelerators), this trade could be worthwhile when memory is the hard limit. We’d need to integrate this with our scheduling: e.g., only turn on rematerialization beyond a certain context length or when memory usage hits a cap. It’s essentially taking quantization to the limit (1-bit or mathematical regeneration). The challenge is keeping it fast; we might use multi-threading or SIMD to recompute multiple layers of KV in parallel if needed. This “compute instead of cache” approach would differentiate ruvllmfor ultra-long contexts. It’s a frontier area, but the pieces we will have (unified memory, kernel plugins, etc.) will help experiment with it.
	2	True Unified Memory Paging Across All Components: We plan a unified pool for KV scratch and adapters; the ultimate vision is to treat all model memory (KV cache, adapter weights, and even the model weights or other state) under one unified memory management system with multiple tiers (HBM/GPU, CPU RAM, perhaps disk/SSD). In a sense, this would function like a virtual memory system for the LLM, where each type of data (model weights, activations, cache, deltas) can be evicted or swapped out in a coordinated way. Some early signs of this are in systems like LMCache, which offloads KV to CPU or disk with a controller API. We’d extend that concept to include LoRA and other aux data. The benefit is that ruvllm could then run indefinitely (as a service) without ever hitting an out-of-memory, because it would have mechanisms to spill to secondary storage deterministically. It would also make consolidation easier: e.g., on a multi-model edge server, a single pool could be shared among multiple model instances, improving utilization. Achieving this “runs forever” reliability is the difference between a demo and a production service. We will aim for a design where adding a “unified paging” module later is feasible – likely by abstracting memory accesses through a layer that can decide to fetch from a slower tier. Logging and determinism will be critical here, as well as avoiding thrashing (hence the importance of the earlier scheduler component).
	3	WASI-NN for Auxiliary Models: While the main LLM forward pass will use our custom kernels for performance, we can leverage WASI-NN and the WASM plugin approach to run auxiliary models alongside the LLM. For example, an edge assistant might have a small gating or routing model that decides if a request should even go to the LLM, or an anomaly detector that inspects outputs for safety. These smaller models (say a 50 MB vision model or a 100 MB classifier) could be run via the WASI-NN interface inside a WASM sandbox. The advantage is we can reuse highly-optimized engines (like OpenVINO, ONNX Runtime) for these tasks by exposing them to the WASM module. Meanwhile, the primary LLM stays on our own optimized path (perhaps for which frameworks don’t yet offer what we need). This hybrid approach keeps the hot path (token generation) under tight control and maximum speed, while still allowing extensibility for supporting tasks using the broader ML ecosystem. Concretely, we could define a WASM ABI for calling out to “side models” asynchronously. This frontier idea means ruvllm could become not just a single-model server but an intelligent host that safely runs multiple models in concert (each in its sandbox), e.g., for multi-modal capabilities or guardrail systems, without compromising the main model’s performance.
Recommended Specification Changes
To implement the above plan in ruvllm, here are the key changes and additions we would make to the current system specification:
	1	Dual-Format KV Cache API: Everywhere the KV cache is referenced in the spec, update it to handle independent Key and Value formats. For example, instead of a single dtype for the entire KV, the interface might carry key_dtype and value_dtype, and functions operating on the cache must handle the possibility that each has its own precision or storage layout. Additionally, add support for per-layer KV settings – e.g. an array of formats indexed by layer. This allows the runtime to use mixed precision strategies (as found in KIVI’s analysis of key vs value distributions). Internally, the KV cache structure could hold pointers (or offsets) to separate key and value storage for each layer, rather than assuming a contiguous array of homogeneous type. This change lays the groundwork for both asymmetrical quantization and easier experimentation (you could plug in a new compression for values without touching keys, for instance).
	2	Unified Memory Pool Manager: Introduce a new component (module) in the spec dedicated to memory management of transient data – essentially an arena allocator for both KV scratch and adapter weights (and any other temporary tensors). The spec should define how this allocator works: e.g., it reserves a fixed maximum size (perhaps configurable per deployment), and it exposes operations to allocate/free blocks for “KV-dequant buffer” or “LoRA weights buffer” etc. Importantly, this manager would implement policies like pooling and paging. For instance, it might have an API to “pin” certain data in GPU memory or to “evict” least-recently-used blocks. We saw in S-LoRA that a unified memory pool is key to avoid fragmentation and handle varying tensor sizes smoothly. By specifying one allocator to rule them all, we prevent each sub-system from over-provisioning. The spec should also clarify the behavior when the pool is exhausted (e.g., block until space is freed, or evict something – likely the latter with a strategy defined). This unified pool concept aligns with both S-LoRA’s unified paging and LMCache’s unified CPU pool for caches. It will make the system more robust under load (no unpredictable OOMs due to many adapters or long prompts, as long as it’s within overall limits).
	3	Explicit Quantization Policy Configuration: Extend the model-loading or runtime configuration part of the spec to include a quantization policy schema. This would allow one to specify things like: quantization bit-width for KV (global or per-layer), group size (if using group quantization), “float tail” length (how many recent tokens to keep in full precision), and thresholds for dynamic quantization triggers (if any). By having a structured policy, the engine can log and adhere to it strictly. For example, a policy might say: “Use 8-bit for layers 0–10, 4-bit for layers 11–23, keep last 32 tokens in float, group-size 16, outlier threshold 0.1%” – the runtime then follows this and we can verify that via logs. The spec should define a standard way to represent this (maybe a JSON or Rust struct). This will help with auditability – one can replay a run and see exactly which decisions were made according to policy. It also separates concerns: the core engine deals with enforcing the policy, while the policy itself can be tuned or even learned. Given vLLM’s approach and others, separating storage precision from compute precision should be clearly delineated – e.g., the spec could state “KV cache may be stored at lower precision than used in attention computations; the system must convert as needed transparently.” This makes it clear to users that storage vs compute dtype are different knobs.
	4	Decoupled Storage vs Compute Precision: As mentioned, formally update the spec (and code) to treat storage precision and compute precision as distinct for model data. This applies not just to KV (where we store quantized and compute in float) but potentially to weights (one could imagine using 8-bit weights but converting to 16-bit on the fly for multiplication if needed, etc.). The documentation should stress that any quantized representation will be losslessly (or near-losslessly) converted to a higher precision for actual math operations. By following this pattern (seen in practice with vLLM’s FP8 KV cache), we ensure that accuracy is easier to maintain and reasoning about precision is simpler. In implementation, this might mean providing utility functions to “prepare tensor for compute” which does dequantization, and making sure kernels always assume they might receive quantized inputs and need to convert. Logging here is useful too – e.g., when a block was dequantized for use, record it. This change is partly philosophical: treat quantization as a compression technique for transit/storage, not as a different kind of “tensor” from the compute perspective. It will simplify kernel development and debugging (kernels can mostly assume floats), and align with hardware trends (many new GPUs handle FP8<->FP16 conversion in hardware seamlessly).
By making the above spec changes, we would get a system that is far more aligned with current best practices and ready for the next steps. Specifically, we add more flexibility in memory and precision management, which are crucial for edge scenarios.
Expected Outcomes
If we execute this plan, the resulting ruvllm runtime will be a predictable, auditable, and high-performance edge inference engine with several key advantages:
	•	Memory Bottlenecks Tamed: The KV cache will no longer be a mysterious source of out-of-memory errors or latency spikes. With quantization and unified management, its footprint is bounded and under control. Long contexts can be handled more gracefully on limited hardware (e.g., quantizing old tokens means you can support chat history 2–4× longer within the same RAM). Adapters similarly become lightweight to serve – you can host hundreds of personalized variants and only pay marginal cost per active one, instead of needing separate copies of the whole model.
	•	System-Wide Efficiency: Through unified pooling and scheduling, all parts of the model (base weights, KV, LoRAs, scratch) share resources cooperatively. This avoids the fragmentation and over-allocation that plague less integrated solutions. It also means ruvllm can run continuously (“forever”) in an edge environment without needing manual restarts to clear caches – it has its own internal “garbage collection” and paging for model data. The worst-case memory usage is predictable and capped, which is vital for production services.
	•	Modularity and Upgradability: By using a plugin kernel approach (with WASM as the vehicle), we achieve a level of modularity where kernels can be improved or specialized for different platforms without changing the core. For example, if a new faster matrix multiply library comes out, one could package it as a WASM plugin; or if a security issue is found in an old kernel, it can be updated independently. This is a step towards treating model execution like loading drivers – the core orchestrator doesn’t need recompilation for every tweak. It also means ruvllm could be extended with new ops or model architectures by adding modules, not by overhauling the engine. This decoupling of policy and mechanism (Rust core sets policy, WASM/Kernel modules do mechanism) follows good systems design and makes the system easier to maintain.
	•	Cross-Platform Edge Deployment: The use of Rust and WASM together ensures that our engine can run on a wide range of edge devices. Rust gives native performance on any platform we can compile to, and WASM gives a safe, sandboxed fallback for portability. An outcome of the plan is that we could have, for instance, the same model and code running on an x86 server, an ARM laptop, and a WASI-enabled browser environment with only minor differences in the loaded kernel modules. This level of portability is cutting-edge – enabling “run anywhere” agents that keep consistent behavior and safety checks across devices.
	•	Always-On Reliability: Ultimately, the combination of all these features means ruvllm can serve as the backbone for always-on edge AI agents that stay responsive and “calm under load”. Surprises due to memory exhaustion or latency spikes will be minimized because we’ve built in backpressure and adaptation (quantize more if needed, etc.). Everything is auditable: if the model’s quality dips or it ran out of budget, you can trace it to a logged event (e.g., “KV for layer 10 quantized to 4-bit at 13:05:23 due to memory threshold”). This is crucial for trust and debugging, which in turn is crucial for deploying AI in the wild. Moreover, by having a strong foundation, new research ideas (like those in the Frontier section) can be incorporated without a rewrite – we can try extreme strategies knowing the system’s modular pieces (schedulers, plugins, etc.) support experimentation.
In essence, by following this plan we transform the original ruvllm spec into a state-of-the-art edge inference systemthat embodies the lessons from the latest research (2024–2025) and anticipates future needs. It balances performance with safety and flexibility, enabling cutting-edge ML models to run efficiently at the edge. This positions us well for the coming era where AI services are expected to be ubiquitous, personal, and reliable.
Sources:
	1	Hooper et al., “KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization,”NeurIPS 2024.
	2	Liu et al., “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache,” 2024.
	3	Tomar et al., “XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization,” arXiv 2508.10395, Aug 2025.
	4	vLLM documentation – Quantized KV Cache feature description (FP8 storage with FP16 compute).
	5	Cheng et al., “LMCACHE: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference,” Tech Report 2024 (LMCache).
	6	Sheng et al., “S-LoRA: Serving Thousands of Concurrent LoRA Adapters,” MLSys 2024.
	7	NVIDIA Technical Blog, “Introducing DoRA, a High-Performing Alternative to LoRA,” Jun 2024.
	8	WASI-NN Specification – Bytecode Alliance post, “Machine Learning in WebAssembly: Using wasi-nn,” 2023.
	9	Bytecode Alliance, Wasmtime examples – fuel and epoch-based interruption documentation.
	10	WAMR (WebAssembly Micro Runtime) project docs – LogRocket Blog, “WebAssembly runtimes compared,”2023.
	11	Buehler, mistral.rs – GitHub README and Jimmy Song’s overview, 2024.
	12	Hugging Face Candle – GitHub README (features: custom kernels, WASM support), 2023.
	13	AWS Machine Learning Blog, “Multi-tenant LoRA serving with Sagemaker (S-LoRA and Punica),” 2024.
¬
¬
¬
¬
¬
Sources

What’s missing?

Thought for 1m 3s

Here is what is missing right now. The plan is directionally correct, but it still lacks the pieces that make it runnable, measurable, and hard to break.
	1	A committed attention and matmul path for the hot loop Right now the plan says “defer attention in WASM,” but it does not specify what runs attention today. You need a clear default backend choice and fallback order, for example a mistral.rs style paged attention backend for throughput and fragmentation control, with KV cache quantization support already proven in that codebase. 
	2	A real SIMD strategy that matches Rust reality in 2026 If you want deterministic speed on edge CPU, you need to pick a SIMD approach that compiles everywhere and can multiversion per CPU. std simd is still nightly in 2025, with people recommending crates like wide, pulp, or macerator depending on your needs. That decision affects every kernel you ship. 
	3	The missing KV cache algorithm layer between quantize and rematerialize You listed KVQuant, KIVI, XQuant, but you have not specified when to use which, nor how to combine them safely. Also, there are newer KV quant methods like SQuat that change the error geometry rather than just lowering bits. That belongs in your decision tree and benchmark suite. 
	4	Unified paging is described, but the allocator and eviction policy are not specified You need concrete answers for: page size, metadata layout, pinning rules, eviction order, concurrency model, and how you prevent thrash. Without this, “unified pool” becomes a soft idea instead of a production subsystem.
	5	The WASM boundary is not locked down You need to decide whether you are shipping raw ABI calls or a component model interface, how memory is shared, and what happens on trap. You also need to commit to interruption mechanics. Wasmtime documents fuel and epoch interruption, with epoch generally faster, and the choice matters for tail latency. 
	6	Kernel pack supply chain and rollback is not fully designed You mentioned signing, but you still need: manifest schema, signature rotation, allow lists, version compatibility gates, reproducible builds, and a safe rollback protocol that is deterministic under load.
	7	Adapter serving needs a multi tenant execution plan Micro LoRA apply is specified, but what is missing is the serving strategy: adapter residency rules, batching rules when multiple adapters are active, adapter compression, and a unified paging contract that covers KV plus adapters together. The S LoRA approach is the reference for why this matters. 
	8	Evaluation harness is not concrete enough You need a mandatory benchmark and correctness suite that covers: a) p50 p95 p99 decode step latency b) memory per token and KV growth curves c) quality drift across long contexts d) adapter correctness and regressions e) quantization error accumulation and recovery Until this exists, you will not know if the scheduler and gating rules are helping or silently hurting.
	9	ruvector integration is still conceptual You have not defined the exact role ruvector plays in the runtime loop. Missing choices include: a) ruvector as the policy memory store for learned thresholds b) ruvector as the session state index for adapter selection and cache locality c) ruvector as the witness log index for postmortem and audit queries You need concrete APIs and storage models for these.
	10	A clear mapping of model formats and feature coverage The plan does not specify model formats, tokenizer pipeline, and attention variants that must be supported from day one, like grouped query attention, sliding window attention, rope variants, and long context scaling. This affects your kernel ABI and test vectors.
If you want the updated frontier plan to be actionable, the next step is to lock three things in writing:
	1	Default inference backend for attention and matmul, with a defined fallback order, likely using mistral.rs as the reference for paged attention and quantized KV support 
	2	SIMD kernel strategy using crates that work today, since std simd remains nightly for many setups 
	3	WASM execution budget mechanism, with epoch interruption as the default where possible 
Once those are fixed, everything else becomes an engineering pipeline instead of a research debate.

Give me full implementation 

To give you a full implementation, could you clarify the scope and target first?
	1	Hardware targets: What devices should this support (e.g. x86 edge servers, ARM SBCs, mobile, microcontrollers)?
	2	Model types: Are you targeting Mistral-7B or similar LLMs, or smaller instruction-tuned models?
	3	Deployment format: Should this build as a static binary, a dynamic runtime, or a WASM package?
	4	Priorities: Should I optimize for minimal memory, max throughput, or multi-user batching?
	5	Use of ruvector: Do you want this as a vector DB, witness logger, policy engine, or all of the above?
Once I have that, I’ll generate the full code and build structure.

Here is what is missing right now. The plan is directionally correct, but it still lacks the pieces that make it runnable, measurable, and hard to break. 1. A committed attention and matmul path for the hot loop Right now the plan says “defer attention in WASM,” but it does not specify what runs attention today. You need a clear default backend choice and fallback order, for example a mistral.rs style paged attention backend for throughput and fragmentation control, with KV cache quantization support already proven in that codebase. 2. A real SIMD strategy that matches Rust reality in 2026 If you want deterministic speed on edge CPU, you need to pick a SIMD approach that compiles everywhere and can multiversion per CPU. std simd is still nightly in 2025, with people recommending crates like wide, pulp, or macerator depending on your needs. That decision affects every kernel you ship. 3. The missing KV cache algorithm layer between quantize and rematerialize You listed KVQuant, KIVI, XQuant, but you have not specified when to use which, nor how to combine them safely. Also, there are newer KV quant methods like SQuat that change the error geometry rather than just lowering bits. That belongs in your decision tree and benchmark suite. 4. Unified paging is described, but the allocator and eviction policy are not specified You need concrete answers for: page size, metadata layout, pinning rules, eviction order, concurrency model, and how you prevent thrash. Without this, “unified pool” becomes a soft idea instead of a production subsystem. 5. The WASM boundary is not locked down You need to decide whether you are shipping raw ABI calls or a component model interface, how memory is shared, and what happens on trap. You also need to commit to interruption mechanics. Wasmtime documents fuel and epoch interruption, with epoch generally faster, and the choice matters for tail latency. 6. Kernel pack supply chain and rollback is not fully designed You mentioned signing, but you still need: manifest schema, signature rotation, allow lists, version compatibility gates, reproducible builds, and a safe rollback protocol that is deterministic under load. 7. Adapter serving needs a multi tenant execution plan Micro LoRA apply is specified, but what is missing is the serving strategy: adapter residency rules, batching rules when multiple adapters are active, adapter compression, and a unified paging contract that covers KV plus adapters together. The S LoRA approach is the reference for why this matters. 8. Evaluation harness is not concrete enough You need a mandatory benchmark and correctness suite that covers: a) p50 p95 p99 decode step latency b) memory per token and KV growth curves c) quality drift across long contexts d) adapter correctness and regressions e) quantization error accumulation and recovery Until this exists, you will not know if the scheduler and gating rules are helping or silently hurting. 9. ruvector integration is still conceptual You have not defined the exact role ruvector plays in the runtime loop. Missing choices include: a) ruvector as the policy memory store for learned thresholds b) ruvector as the session state index for adapter selection and cache locality c) ruvector as the witness log index for postmortem and audit queries You need concrete APIs and storage models for these. 10. A clear mapping of model formats and feature coverage The plan does not specify model formats, tokenizer pipeline, and attention variants that must be supported from day one, like grouped query attention, sliding window attention, rope variants, and long context scaling. This affects your kernel ABI and test vectors. If you want the updated frontier plan to be actionable, the next step is to lock three things in writing: 1. Default inference backend for attention and matmul, with a defined fallback order, likely using mistral.rs as the reference for paged attention and quantized KV support 2. SIMD kernel strategy using crates that work today, since std simd remains nightly for many setups 3. WASM execution budget mechanism, with epoch interruption as the default where possible Once those are fixed, everything else becomes an engineering pipeline instead of a research debate.

Understood. I will research and specify:
	1	A committed, default attention and matmul backend (e.g. mistral.rs, Candle, FlashAttention alternatives) with fallback order for edge inference.
	2	A Rust-compatible SIMD strategy usable in production as of 2026 (e.g. wide, pulp, safe_arch, or stable std::simd), with multiversioning guidance.
	3	WASM kernel execution limits—compare Wasmtime's fuel vs epoch interruption—and define a safe default.
Once those are defined, the rest of the system can be locked down as an implementation pipeline.
Research completed in 7m · 9 sources · 52 searches

Implementing the Frontier Plan: Filling the Gaps
Introduction
The frontier inference plan is on the right track conceptually, but several critical implementation details are missing. This comprehensive plan addresses the ten identified gaps, turning a high-level design into a runnable, measurable, and robust system. We then lock in three key decisions (attention backend, SIMD strategy, and WASM budgeting) to make the plan actionable.
1. Committed Attention and Matmul Path for the Hot Loop
Default Attention/Matmul Backend: We will implement a high-performance attention kernel and matrix multiplication backend for the main inference loop. The default choice is a paged attention mechanism inspired by mistral.rs, which has proven success in managing long contexts efficiently. In paged attention, the KV cache is split into fixed-size slots (“pages”) rather than one contiguous buffer. The attention computation uses an indirection table of indices into these pages, avoiding ever assembling a giant contiguous KV tensor. This design improves throughput and fragmentation control, as it reuses memory slots and avoids reallocating huge buffers. Mistral.rs shows that paged attention can dramatically improve memory use and even supports quantized KV cache entries (e.g. 8-bit) to further reduce overhead. We will follow this approach: store keys/values in a paged structure and have attention kernels gather from pages on-the-fly.
Quantized KV Support: The attention implementation will natively support quantized KV cache entries. For example, we can store keys and values in 8-bit or 4-bit formats during attention computations to save memory. This follows the approach demonstrated in Mistral.rs where KV cache quantization to FP8 was added to cut memory usage. Our attention kernel will treat quantized KV appropriately (dequantizing on read or operating in quantized space if possible) so that using lower precision doesn’t break the model’s computations.
Backend Fallback Order: We will define a clear fallback sequence for the attention/matmul computation in case the fastest path is not available on a given system:
	•	Primary: a custom Rust SIMD kernel (see SIMD strategy below) implementing the paged attention with quantized KV. This is optimized for CPU inference on long contexts.
	•	Secondary: if the environment provides specialized libraries (e.g. BLAS or vendor optimized kernels) or a GPU is available, we can offload the matrix multiplies to those. For instance, on GPU we might leverage a CUDA attention kernel (as in vLLM or FlashAttention) or on x86 maybe call into oneDNN for large matmul, falling back to our paged scheme for KV management. The system will detect available backends and choose the best at runtime.
	•	Tertiary: a simple, reference implementation (contiguous full-precision KV cache with a straightforward attention loop) as a last resort. This ensures correctness on any platform even if performance is lower. It’s essentially a “naive” attention that we can always fall back to for functional safety.
By committing to this default paged-attention path with quantized KV, we ensure the main loop is efficient. On long contexts, our approach avoids the latency spikes from huge memory allocations and keeps throughput high by reusing fixed pages. When a platform can’t support that, the graceful degradation path is clearly defined by the fallback order above.
2. Real SIMD Strategy Matching Rust 2026 Reality
Choosing a SIMD Abstraction: To achieve deterministic speed on edge CPUs, we need to pick a SIMD approach that is portable and can multi-version (dispatch optimized code per CPU feature) on stable Rust. As of 2025/2026, std::simdis still nightly-only and not suitable for stable builds. Community consensus suggests: use std::simd on nightly, but for stable builds consider wide if no CPU feature dispatch is needed, or pulp/macerator for multi-version support.
Given we want broad deployment on various CPUs with optimal vector instructions, we will adopt macerator as our SIMD backbone. The macerator crate is a fork of pulp offering generic SIMD traits and expanded instruction set coverage. It supports all modern x86 (SSE, AVX2, AVX-512), ARM NEON, WASM SIMD, etc., making it ideal for a cross-platform engine. It also handles runtime feature detection (multi-versioning) so our code can transparently use AVX-512 on a server chip, AVX2 on an older laptop, or NEON on ARM. This choice influences every low-level kernel (from dot products in attention to LoRA merges), ensuring we exploit hardware vector units fully.
Alternate Crate Options: We considered pulp and wide as well. pulp is proven (used in faer’s matrix multiply) and has built-in multiversioning, but it only supports native width vectors and a limited set of architectures (AVX2, AVX-512, NEON). wide is very ergonomic and covers many types, but it lacks multiversioning and would force a single target ISA. Since our system must scale from consumer devices to servers, macerator gives the best balance of stability and performance. It operates on stable Rust and supports writing generic SIMD code that compiles to all necessary backends.
SIMD Utilization in Kernels: With macerator, we will implement all heavy compute kernels (matrix multiplies, layernorm, activation functions, etc.) to operate on SIMD types. This yields consistent speedups across platforms. We will also use its traits to write code once and have it vectorize to different widths. For example, the attention score calculation (Q·K^T) and the output combination (scores·V) will be vectorized. The crate will handle selecting 128-bit vs 256-bit vs 512-bit operations depending on the CPU. This approach ensures deterministic speed-ups and that no CPU is left underutilized. In summary, our SIMD strategy is locked to a stable, cross-platform solution (macerator)that aligns with Rust 2026 best practices, avoiding nightly features while still leveraging modern SIMD instructions.
3. KV Cache Quantization and Rematerialization Strategy
We need a clear algorithmic policy for managing the KV cache as sequences grow, including when to quantize cached keys/values, when to rematerialize (recompute) them if needed, and how to combine different quantization methods safely. In the original plan, methods like KVQuant, KIVI, and XQuant were mentioned. We will refine this with newer research (like SQuat) and define usage scenarios for each technique:
	•	Default KV Cache Precision: By default, new tokens’ K/V tensors start in full precision (e.g. FP16/BF16). This ensures maximal quality for recent tokens where the model is most sensitive. As the context grows, the oldest entries contribute less to near-term predictions, so we apply quantization progressively.
	•	KIVI 2-bit Compression for Stale Segments: KIVI is a tuning-free 2-bit KV quantization method that quantizes keys per-channel and values per-token. It achieves ~2.6× reduction in KV memory with minimal quality loss(virtually same perplexity) and enables up to 4× larger batch sizes for Llama-2 and others. We will use KIVI-style quantization on older tokens in the cache. For example, once the context exceeds a threshold (say 512 or 1024 tokens), we convert the oldest segment of KV cache to 2-bit using KIVI’s scheme. This dramatically reduces memory footprint while maintaining quality for those older positions. The system will maintain a small per-layer scale/offset metadata from KIVI’s asymmetric quantization to allow dequantization if those tokens are attended strongly.
	•	SQuat for Aggressive Compression: For extremely long contexts or memory-constrained scenarios, we introduce an option to use SQuat (Subspace-orthogonal quantization). SQuat projects key vectors into a subspace spanned by recent query vectors, and then quantizes in a way that preserves components relevant to attention. This method can push quantization to very low bit-widths (2-bit) while minimizing impact on attention outputs. It has shown ~2.2–2.8× memory reduction and ~2.5–3.6× throughput improvement over baseline KV cache handling, with better quality than other low-bit methods. We will deploy SQuat for very long contexts (e.g. >2048 or on user opt-in “max compression” mode). In practice, after KIVI reduces memory, if context keeps growing, SQuat can be applied to further compress the oldest half of the cache to 2-bit with orthogonal error minimization. This two-stage quant (first KIVI, then SQuat on top) ensures that even at extreme lengths, the attention sees minimal error from quantization.
	•	KVQuant for Quality-Critical Long Contexts: KVQuant is a comprehensive approach incorporating per-channel key quantization, pre-ROPE quantization (applying quantization before rotary positional embedding to reduce its interference), non-uniform clustering of values, and outlier isolation. It is more complex but yields the highest fidelity: <0.1 perplexity drop at 3-bit precision on long contexts, outperforming simpler methods. KVQuant enabled LLaMA-7B to handle up to 1 million tokens on a single 80GB A100 GPU (10 million on 8 GPUs), thanks to its ultra-low precision and custom CUDA kernels. We will provide KVQuant as an optionalbackend for deployments that absolutely require maximum context length with minimal quality degradation (e.g. enterprise use of >100k context). If the hardware support is present (our system detects GPU and has the KVQuant CUDA kernels available in the kernel pack), we can invoke KVQuant to quantize the cache down to 3-bit or 2-bit once context exceeds a high threshold. This will come with the trade-off of needing the specialized kernels and possibly slightly higher latencies (due to more complex quantization logic), so we won’t use it by default on every platform – only when long-context support is paramount and the environment can support it.
	•	“XQuant” and Others: We interpret XQuant as a placeholder for any other experimental quantization approach (for example, exllama’s KV compression using a Hadamard transform, etc.). The design will be extensible so that new quantization plugins can be slotted in. For now, our primary quantization decision tree is: use KIVI for general compression, escalate to SQuat for aggressive compression, or switch to KVQuant when available for best-in-class long-range support. Each method is applied exclusively to avoid conflict (e.g. we wouldn’t stack KIVI and KVQuant on the same data – we choose one path based on scenario). All quantization modes will be configurable, so operators can disable quantization entirely (for absolute maximal quality) or choose a desired memory-accuracy trade-off profile.
	•	Rematerialization Policy: In addition to quantization, we have a strategy for rematerialization. If the context window grows so large that even quantization can’t save enough memory (or if quality starts degrading), the system can evict the oldest KV cache blocks and rematerialize them on-the-fly if needed. Concretely, if a session goes beyond N tokens (configurable, e.g. 8192), we might drop the oldest 2048 tokens’ KV caches entirely to free memory (those tokens remain in the textual history). If the model later needs to attend to those dropped tokens (which might happen depending on attention pattern), our scheduler will recompute that portion by running the model on that segment again to regenerate the missing keys/values. This is similar to checkpointing in training – trading compute for memory. We will be careful to only rematerialize in controlled ways: for example, never drop the most recent context so the majority of attention queries hit quantized or full-precision cache. Rematerialization might be paired with summary techniques (e.g. the system could insert a summary embedding for very old content to avoid fully recomputing). The exact trigger for rematerialization vs. further quantization will be informed by the evaluation harness (see section 8): if we detect quantization error accumulating beyond a tolerance, we prefer to rematerialize old context rather than quantize it further. This ensures quality does not silently degrade on ultra-long sessions.
In summary, the KV cache management layer will dynamically choose between no quantization, KIVI 2-bit quant, SQuat subspace quant, or KVQuant advanced quant based on context length, hardware capabilities, and desired quality. It will also utilize eviction and recompute (rematerialization) as a backstop to prevent unbounded memory growth. All these are orchestrated by a scheduler that monitors memory usage per token and error metrics so it can combine these techniques safely. Newer research like SQuat is incorporated to improve the error characteristics of quantization (ensuring quant errors are orthogonal to the important subspace of queries). By explicitly defining when and how to apply each method, we turn a menu of options into a concrete algorithm that will be implemented and tuned in the system.
4. Unified Paging Allocator and Eviction Policy
We will implement a unified memory pool for all dynamic data (KV cache pages, adapter weights, etc.) and define the allocator behavior in detail. This unified paging system is inspired by S-LoRA’s approach, which introduced a unified memory pool to manage both KV cache tensors and LoRA adapter weights of varying sizes. Here we specify the parameters and policies:
	•	Page Size: We choose a page size of 2 MB for the unified pool by default. This size is a balance between fragmentation and overhead: 2MB pages are large enough to hold many typical KV cache blocks (for instance, a 512-token KV for a 7B model fits in a few pages) and moderately sized LoRA weights, while small enough that we can allocate and move them without huge latency. We will allow this to be configurable (e.g. 512 KB to 4 MB range) to tune for different hardware. We align pages to 2MB also because on GPU memory, large page allocations (or using CUDA unified memory manager) often perform better when aligned to power-of-two boundaries.
	•	Memory Pool Layout: At initialization, we allocate a big contiguous chunk of memory (on each device, e.g. CPU RAM for CPU inference or GPU VRAM for GPU inference) that will serve as the unified pool. For instance, on a 80GB GPU, we might dedicate 75GB as the pool, leaving some headroom for model weights and other overhead. Within this pool, memory is managed in fixed-size pages. We maintain a metadata table (in host memory) with an entry per page indicating its status: free or allocated, and if allocated, what it contains (e.g. “Adapter X layer 2 weights” or “KV cache for Session Y, layer 10, tokens 100-199”). This metadata also stores usage stats like last access time for eviction decisions.
	•	Allocation Strategy: When a request comes in that needs memory (say, loading a LoRA adapter or extending a KV cache), the allocator will find a sufficient number of free pages in the pool to satisfy it. If the object is larger than one page, contiguous pages will be allocated (the pool supports allocating ranges of pages). The allocator will try to find a contiguous run via a first-fit or best-fit strategy to minimize fragmentation. Metadata will then mark those pages as in use and link them to the object. Smaller objects (e.g. a very small LoRA of only a few KB) will still consume one full page (we won’t sub-partition pages at this time, to keep it simple and avoid internal fragmentation issues – the page is our atomic unit of allocation).
	•	Pinning Rules: Active data is pinned in memory. Any KV cache pages that correspond to tokens still in the current context of an ongoing request are pinned (not evictable) until those tokens are no longer needed (e.g. evicted by our KV scheduler as described above, or the session ends). Similarly, an adapter weight that is currently in use by at least one active inference is pinned in GPU memory. “In use” means the model is either actively processing a prompt/batch that uses that adapter, or it’s expected to be used in an upcoming batch that is already scheduled (to prevent thrashing during scheduled batches). Pinning ensures we don’t evict something mid-use. We implement reference counting in the metadata: each page has a pin count (number of active uses), and only when that drops to zero does the page become evictable.
	•	Eviction Policy: When the memory pool is exhausted (no sufficient free pages for a new allocation) we need to evict something. We will use an LRU (Least Recently Used) policy by default for evictable pages, with some refinements. Specifically, we’ll evict the pages that have the oldest last access timestamp among those not pinned. This tends to remove long-unused adapters or KV from old sessions. However, we also factor in size and importance: if multiple candidates are similarly old, we might prefer evicting a larger adapter that frees more space in one go, unless that adapter is expected to be needed soon (we could use a simple heuristic: if an adapter hasn’t been used for X minutes, it’s unlikely to be needed immediately again). For KV cache pages, eviction generally would target entire old sessions that have been idle (e.g. if a user hasn’t sent a message in a while, we evict that session’s KV to free memory). The eviction process will copy evicted data to a slower storage if needed: e.g., for an adapter, we might offload it back to host memory or disk so it can be loaded later; for KV cache, we likely just drop it (or, if we support long-term persistence of conversation state, we might serialize it to disk, but by default KV eviction means those tokens will need recomputation if needed again).
	•	Concurrency Model: The allocator must handle concurrent requests, as multiple threads may be generating outputs for different users. We will implement a lock-free free-list or use atomic operations for page allocation to avoid global locks. One approach is to divide the pool among threads or use a per-thread cache of free pages to reduce contention. However, since the pool is unified, a global view is needed to decide eviction. We will likely protect the allocation and eviction process with a lightweight mutex or an ordered lock: e.g., when a new large allocation request comes in and finds insufficient space, a thread will take an eviction lock, perform the LRU eviction of some pages, mark them free, then allocate. This eviction step will be tuned to be fast (we’ll evict in bulk if possible, and perhaps do it asynchronously if the requester can wait a bit). The design will ensure that most fast-path allocations (when free pages exist) don’t need global locking – they can pop from a free list quickly.
	•	Thrash Prevention: To avoid pathological thrashing (constant evict-load cycles of the same data), we implement a hysteresis and prioritization scheme. For example, if an adapter was evicted very recently and now is needed again, the system might decide to keep it in memory a little longer next time (mark it as “recently evicted, do not evict again for Y minutes”). Similarly, if a certain session’s KV is causing frequent evictions back and forth, we may choose to grow the pool or refuse new allocations (backpressure) rather than constantly evict and rematerialize. The unified allocator will expose metrics like current utilization and eviction rate; if eviction rate is high (thrashing signal), it could trigger load-shedding (e.g. refuse loading a low-priority adapter until memory pressure eases) or increase the quantization of KV to reduce footprint. Additionally, we might reserve a small percentage of the pool as a buffer that’s only used under extreme conditions, to absorb spikes without immediate evictions. These measures ensure the system remains stable under load and doesn’t degrade into continuous swapping of pages in and out.
In summary, unified paging means all dynamic memory (adapters and caches) lives in one managed pool, reducing fragmentation and allowing intelligent trade-offs between them. We have concretely defined the page size, metadata, pin/evict rules, concurrency handling, and thrash avoidance. This turns the “unified pool” concept into a working subsystem ready for implementation and tuning.
5. WASM Execution Boundary and Safety
We clarify how we will integrate WebAssembly (WASM) into the system, including the interface, memory sharing, trap handling, and the execution budget (fuel/epoch) for timeouts. The plan is to use WASM for any sandboxed or user-provided logic (for example, custom embedding functions, plugin scripts, or untrusted model components) while ensuring it cannot hang or crash the host.
	•	ABI vs Component Model: We will initially expose a raw ABI interface to the WASM modules for simplicity and performance. That is, the host and WASM will communicate through low-level function calls and memory, rather than using the full WASM Component Model (which, as of 2026, is still maturing and could add overhead). Concretely, we’ll define a small set of C-ABI functions that a module can export (e.g. compute_attention(query_ptr, key_ptr, value_ptr, len) or apply_adapter(layer_idx, input_ptr, output_ptr, len)) and the host will use Wasmtime (our chosen runtime) to call these functions directly. This avoids serialization of data or complex adapters – it’s essentially like calling a dynamic library, but in a safe WASM sandbox.
	•	Memory Sharing: To minimize copying overhead across the host-WASM boundary, we will share memorywhere possible. Wasmtime allows the host to allocate a memory and then expose it to the WASM module, or vice versa. We will use this to let the WASM module operate on data that resides in host memory (or in a memory mapped region accessible to both). For instance, we can allocate the model’s tensor data in a shared memory and pass pointers (offsets) to WASM functions. Because WASM memory can be dynamically grown and uses 32-bit indexing by default, we might use the Wasmtime 64-bit memory extension if needed for large models, or simply manage multiple memory segments. The key is that heavy data (tensors) won’t be copied into WASM; instead, WASM gets access to a controlled memory region containing those tensors. This improves performance while maintaining safety (WASM can’t access outside the shared region, and we set memory limits).
	•	Trap Handling: If a WASM module traps (for example, due to an out-of-bounds access or an explicit unreachable), the host will catch this as a runtime error. Our policy is to treat WASM traps as non-fatal errors for the request in question. The engine will abort the current operation in that WASM instance, log the error (with enough context to debug), and free or reset the WASM instance. For the user request that triggered it, we’ll propagate an error up (or fallback to a safe implementation if possible). The system remains running – the trap does not crash the whole server. We will design these error paths carefully so that any partially acquired resources (memory pages, locks, etc.) are released when a trap occurs. Wasmtime’s API allows us to attach a trap handler or just catch the exception, so we will utilize that. Additionally, we might impose a restart policy: if a particular WASM module traps frequently, we may unload or block that module until it’s fixed, to avoid repeated failures.
	•	Interruption and Budgeting: To ensure a runaway WASM execution (e.g. an infinite loop in user plugin code) doesn’t hang our service, we will use Wasmtime’s epoch-based interruption as the default mechanism. Wasmtime offers two approaches: fuel, where you decrement a counter for executed instructions (which adds overhead per instruction), and epoch timers, where you can periodically check for a cancellation flag with much lower overhead. We choose epoch interruption because it has negligible performance impact on tight loops, which is crucial for tail latency in our hot loop. Concretely, we will configure an epoch deadline for each WASM execution. The host can increment a global epoch counter asynchronously (e.g. every N milliseconds) and if the execution runs for too long, Wasmtime will throw an interruption error at the next check point. The generated code will include safe points (typically at function calls or loop back-edges) to observe the epoch. This way, if (for example) an attention kernel WASM is taking too long (maybe it got an unexpectedly large input), we can interrupt it and perhaps fall back to a simpler implementation rather than let it drag out p99 latency.
	•	WASM vs Native Execution Choices: For clarity, not all core inference will run in WASM – our default attention/matmul is in native Rust (for speed and control). The WASM boundary is used for extensibility and isolation. For instance, if we support user-defined functions or a model from a third party in WASM, we use this sandbox. We will document which parts of the system use WASM. For those parts, we’ll ensure the interface is minimal (passing pointers/lengths and getting results) to keep overhead low. Memory is mostly shared, so copying is minimal. And with epoch-based cancellation, we guarantee that even a malicious or buggy WASM cannot spin forever: it will be stopped in a timely manner, preserving SLAs for other users.
By locking down these WASM details, we ensure that the component boundary is robust: modules have a defined way to call/behave, cannot harm the host, and can be preempted if needed. This transforms “defer to WASM” from a vague idea into a concrete, safe extension mechanism ready for production (leveraging Wasmtime’s proven sandbox and interruption features).
6. Kernel Pack Supply Chain and Rollback Design
The plan to allow pluggable optimized kernels (for various hardware or updated algorithms) requires a solid supply chain setup. We outline how kernel packs are versioned, verified, and rolled back safely:
	•	Manifest Schema: Each kernel pack (a bundle of optimized kernels or functions, possibly compiled to native code or WASM) will include a manifest file (e.g. kernels.json or similar). This manifest lists the contents and metadata: for each kernel, it might have fields like name (e.g. "attention_avx512"), version (of that kernel), supported_targets (CPU features or GPU architectures), a cryptographic hash of the binary, and a signature. It also lists a pack version and a minimum engine version required (to ensure compatibility with our runtime). The manifest schema will be strict and documented, enabling the runtime to parse it and decide which kernels to load on startup.
	•	Signatures and Trust: The kernel pack (or each kernel in it) will be digitally signed by the provider. We will use an asymmetric key (e.g. an Ed25519 or ECDSA keypair) to sign the manifest. The public key (or a root certificate) will be embedded in our application as the root of trust (we can allow updates to this via secure channels for rotation). When the runtime fetches or is given a new kernel pack, it verifies the signature against the trusted key. Only if the signature is valid and the content hash matches (to prevent tampering) will the pack be accepted. This prevents unauthorized or malicious code from being loaded into our process.
	•	Signature Rotation: Over time, keys may need to be changed (compromised or just operational rotation). Our design supports multiple trusted keys with metadata (e.g. key ID and expiry). The manifest will indicate which key signed it (through an identifier or embedded certificate). The runtime will have an allowlist of valid signers. We can update that allowlist via our own secure update mechanism if needed (for example, ship an update that trusts a new key before retiring an old one). We will also timestamp our manifests; if a manifest is signed by an expired key or past a certain date, the runtime may warn or refuse it depending on policy. This ensures a compromised old key can be phased out.
	•	Allowed List and Version Gates: Not all kernels or packs are automatically acceptable. We will maintain an allowlist of known-good kernel pack versions. For example, if version 1.2 is the latest stable, the runtime might refuse to load an unrecognized version unless explicitly overridden. This is a safeguard so that even with a valid signature, a radically different pack won’t be loaded blindly. Moreover, each pack will declare compatibility (like “for Frontier runtime >= 1.0.0 and < 2.0.0”). The runtime will cross-check its own version and the pack’s intended range. Incompatible packs are rejected to prevent crashes from API mismatches. Essentially, we bake in version gating: both the engine and the kernels must agree on an interface version.
	•	Reproducible Builds and Audits: To build trust in these kernel packs, we will strive for reproducible builds of the kernels. This means anyone (including ourselves or third-party auditors) can rebuild the kernel code from source and get the same hash that is listed in the manifest. Using deterministic compilation techniques (specific compiler versions, flags, etc.) is part of the pipeline. We will also publish the source or at least the hash of source control for each official kernel binary. This supply chain transparency helps ensure no hidden code is present. For internal development, when we integrate a new kernel (say an updated quantization kernel), we will have a CI step that reproduces the binary, signs it, and packages the manifest in a verifiable way.
	•	Safe Rollback Protocol: If a new kernel pack is deployed and causes problems (e.g. performance regression or crashes), we need a deterministic and safe rollback. We design the system to retain the previous kernel pack in memory or disk until the new one is proven. On startup, the engine can keep two versions loaded (current and last-known-good), but uses the current by default. If certain health checks fail (for instance, the new kernels trigger errors in the evaluation harness or fail a quick self-test), the system can switch back to the old pack on the fly or after a restart. We also implement an administrative override: an operator can send a command to revert to the previous pack, which the runtime will then use for subsequent requests. The key point is to never remove or overwrite the last known good kernels until we are sure the new ones are stable. All kernel packs are immutable (versioned), so rollback is simply a matter of toggling which version is active. Additionally, under load, we ensure consistency: we won’t have one request using the new kernel while another uses an old one in an inconsistent way. The switchover is handled at a synchronization point (e.g. no requests in mid-attention computation) to avoid nondeterminism.
	•	Deterministic Behavior Under Load: We need to guarantee that rolling out a new kernel (or rolling back) does not cause half-computed results or divergent behavior. To do this, we plan a two-phase activation: load the new kernel pack in parallel (so both old and new are in memory), then quiesce incoming work (finish ongoing queries), switch a flag so new queries use the new kernels, but allow any in-flight ones to finish with the old if needed. In practice, since inference requests are short-lived relative to deployments, we might simply drain and then flip. For rollback, the same process applies in reverse. This way, at any given moment each session is consistently using one set of kernels. Logging will note which version was used for each request for audit.
By specifying manifest, signing, allowlists, compatibility checks, and rollback steps, we convert “kernel pack with signing” from an idea into a concrete supply chain security protocol. This will protect users from malicious or unvetted optimizations and give maintainers the confidence to push updates and revert if issues arise, all in a controlled, deterministic fashion.
7. Multi-Tenant Adapter Serving Strategy
Serving multiple LoRA adapters concurrently (multi-tenant) introduces challenges in how to load, apply, and scheduleadapters efficiently. We have outlined micro-batched LoRA application in the plan; now we detail the serving strategy:
	•	Adapter Residency Rules: We cannot keep all adapters loaded on the GPU at all times if there are thousands, so we need rules for which adapters stay in memory. Our policy will be usage-driven: adapters that have recently or frequently been used will remain resident in GPU memory (the unified pool) for faster access, while others are evicted to CPU memory when idle (using the unified paging system described above). Concretely, if an adapter hasn’t been used in, say, the last 5 minutes and memory is needed, it becomes a candidate for eviction. However, we also allow pinning of certain high-priority adapters (for instance, a globally important one or a very frequently used one can be configured to always stay loaded). This ensures critical adapters don’t thrash. All adapters are still always stored in host memory (main RAM), so evicted just means “not currently on GPU.” Loading an evicted adapter from CPU to GPU will incur a transfer latency (which we aim to amortize with batching).
	•	Batching and Scheduling: We leverage heterogeneous batching similar to S-LoRA and Punica. The idea is to batch inference requests even if they use different adapters, to maximize GPU utilization. S-LoRA introduced custom kernels for this: for example, if two requests are running on the same base model but with different LoRAs, we can still combine parts of their computation. We plan to implement a scheduling algorithm that groups incoming requests by common stages. For instance, all requests share the base model forward pass up until the injection of adapter weights. We can run the base model forward for a batch of requests (regardless of adapter), then apply each adapter’s adjustments in parallel using a specialized kernel. Recent research (Punica’s SGMV kernel) does exactly this by performing a segmented matrix-vector multiply that applies different LoRA deltas for different requests in one fused GPU operation. We will incorporate a similar approach: during each transformer layer, we separate the computation into the base part (which can be batched across requests) and the adapter part (which is small rank updates). The adapter parts for multiple requests can be fused by parallelizing over the batch dimension. This way, multi-adapter batches approach the efficiency of single-adapter ones.
	•	Adapter Loading and Compression: To reduce load time and memory, we will compress adapter weights in memory. Adapters (which are essentially small matrices) can be stored in 16-bit or even 8-bit without significant quality loss (some LoRA papers even quantize adapters). We’ll use a compressed representation (e.g. int8 + scale) in the unified pool, and only decompress to FP16 when applying them. If the adapter is rarely used, keeping it compressed saves memory and speeds up transfers from CPU to GPU. Loading an adapter from CPU might involve reading from disk or network (if it’s not preloaded). We aim to hide this latency by prefetching: the scheduler can predict which adapter will be needed (based on incoming requests queue) and initiate load in advance. Also, if multiple requests for the same adapter come in, they will share the single loaded instance – we won’t duplicate the adapter in memory for each request. Instead, one copy is loaded and reference-counted.
	•	Unified Paging of KV + Adapters: Because both KV caches and adapters draw from the same memory pool, we enforce a unified policy to prevent one from starving the other. For instance, we might reserve a portion of GPU memory specifically for adapters (say 20%) and the rest for KV cache, or make it fully dynamic with priority. Our eviction policy (from section 4) will consider both types: e.g. if the pool is full and a new adapter needs memory, it might evict some KV pages from an idle session or evict a less-used adapter, whichever has lower impact. The eviction order across types might be decided by a cost heuristic: evict whichever of (some KV pages vs an adapter) frees the most space with least expected future penalty. S-LoRA’s unified paging indicates that managing them together is possible and beneficial. We will likely implement a unified LRU across all pages, but with a tweak: adapter pages might have a slightly different aging curve than KV pages. For example, KV pages might naturally cycle as sessions end, whereas adapter pages might stick around longer. We ensure that eviction does not consistently choose KV over adapters or vice versa in a way that thrashes one side; tuning may involve weighting recency for adapters differently.
	•	S-LoRA Reference Implementation: We take inspiration from S-LoRA, which achieved ~4× throughput improvement and orders-of-magnitude more adapters served compared to naive approaches. The key features we emulate are: unified memory management (addressed above), heterogeneous batching (addressed with our scheduler), and parallel LoRA application kernels. We may use or adapt S-LoRA’s published CUDA kernels or implement our own. The result is that whether there are 2 adapters or 2000, the system can load and unload them dynamically and schedule work such that GPU utilization remains high. In practical terms, if 100 different adapter requests each want to generate text, our system can consolidate their work onto, say, a handful of GPUs by intermixing them (instead of dedicating one GPU per adapter). This is exactly how multi-tenancy is achieved efficiently.
In summary, our multi-tenant adapter serving plan ensures that many adapters can co-exist with minimal overhead. Adapters are only on GPU when needed, multiple adapter requests can batch together, and memory is shared with KV cache but under a unified policy to avoid conflict. By following S-LoRA and Punica’s innovations, we commit to a design that is proven to scale to thousands of adapters with excellent throughput.
8. Comprehensive Evaluation Harness
We will develop a rigorous evaluation harness to measure performance and correctness, covering the critical metrics listed. This harness will be used continuously during development to validate that each subsystem (scheduling, quantization, etc.) is helping rather than harming the end goals.
Key components of the evaluation suite:
	•	Latency Benchmarks (p50/p95/p99): We will measure the decode step latency (time per token generation) under various conditions. This includes single-token latency (p50 median latency for a single inference) and tail latency (p95, p99 for bursts or batched scenarios). The harness will simulate realistic loads – e.g., multiple concurrent users with different context lengths – and record token generation times. We need to ensure that tail latency is within acceptable ranges, especially with our scheduling and WASM interruption in place. For example, we might find that a certain configuration causes p99 to spike; the harness would catch that so we can adjust (maybe adjust the epoch timing or batch scheduler to cut off outliers). We’ll automate this test across different hardware (CPU vs GPU) as well.
	•	Memory Usage per Token & KV Growth: We will create tests that feed increasingly long inputs and record memory usage patterns. For example, feed a conversation that grows to 1k, 2k, … 10k tokens, and log memory allocated for KV cache, how much was quantized, etc. This produces a KV growth curve. We expect, with our quantization and eviction, that memory usage will plateau or grow sub-linearly beyond a point (as older parts get compressed or dropped). If instead we see linear growth without bound, that means our policies failed – the harness would flag it. Additionally, we measure memory per token: how many bytes of GPU RAM are used per each token in context. With quantization, this number should drop (e.g. maybe 16 bytes/token in full float down to 4 bytes/token with 2-bit quantization). We will verify these against theoretical expectations (KIVI promises 2.6× less memory, etc.). Any discrepancy might reveal fragmentation or metadata overhead, which we can then optimize.
	•	Quality Drift over Long Contexts: We will validate that model output quality does not degrade unacceptably over long contexts. This requires tests on tasks where ground truth or expected behavior is known even with long prompts. For instance, we can use a long document QA dataset: provide a very long text and ask a question about the beginning. We compare the answer when the model has the full context precisely vs when it has gone through our pipeline (with quantization, maybe some rematerialization). We also use perplexity measurements on language modeling benchmarks at various context lengths. The harness will, for example, take a 4,000-token text, run the model with no quantization (as reference), and then run with our progressive quantization, and compute perplexity on predicting the next tokens. If perplexity rises significantly, that indicates quality loss due to quantization error accumulation. We then test mitigations (like SQuat or more frequent rematerialization) to see if we recover quality. By doing this systematically (e.g. at 1k, 2k, 4k, 8k context), we can chart how quality metrics drift and ensure they stay within acceptable bounds. Our target is that even at maximum context, the model’s performance is close to a baseline (perhaps <5% degradation in perplexity or accuracy on tasks). If we see larger drops, the harness will highlight it and we’ll adjust our quantization strategy thresholds.
	•	Adapter Correctness and Regression: We must verify that applying adapters (LoRAs) yields the intended model behavior exactly as if the model were finetuned or the LoRA merged offline. The harness will include tests where we have ground-truth outputs: for example, if we have a small LoRA that changes the model to output in a certain style, we check that on a known prompt, the output matches the expected adaptation. We’ll run the model with and without the adapter in our system and compare to an offline merged version of the model. The difference should be only minor floating-point noise. We will also test concurrent adapters: ensure that when two requests use different adapters, they don’t interfere (no cross-talk or memory corruption). Additionally, we will maintain a set of core metrics for each adapter (like accuracy on a task it was meant for) and ensure that as we change scheduling or memory management, those metrics don’t regress. For instance, if an adapter was fine-tuned for sentiment analysis, we’ll run a small evaluation of that in the harness and confirm the outputs remain as before, even as we tweak the system’s internals.
	•	Quantization Error Accumulation and Recovery: This test suite will intentionally stress the quantization system. For example, run the model for a long conversation (many tokens) and periodically unquantize the cache to see if the model can recover. One specific test: take a scenario and run it twice – once with periodic cache resets (clear and recompute, akin to perfect precision refresh) and once with continuous quantization. Then compare the model’s answers or logit distributions at certain checkpoints. If we observe divergence, we measure how big and whether it grows. The harness can include automated detection: e.g., embed the outputs or use a similarity measure to see if responses drift after a long time. If drift is found, we test our recovery mechanisms: perhaps after N tokens we flush part of the cache (forcing a rematerialize, effectively resetting quantization errors). The harness will help find an optimal schedule for such refreshes if needed. Our goal is that any accumulation of quantization error is bounded – ideally the model’s outputs after long runs remain coherent and correct. Should the harness find unbounded error growth, that will prompt us to incorporate techniques like high precision periodic sync (maybe temporarily using FP16 for a layer once every few hundred tokens to realign) or using SQuat which is designed to minimize accumulated error by its orthogonal projection approach.
	•	Scheduler and Gating Effects: We will also explicitly test the contribution of our scheduling policies (like adapter batching or KV eviction thresholds). For example, we’ll measure throughput with and without heterogeneous batching to quantify the gain (expecting something akin to S-LoRA’s up to 4× throughput improvement on multi-adapter loads). We’ll simulate high load multi-tenant scenarios to ensure our scheduler improves p95 latency and throughput. If any gating rule (like “if context > N, quantize”) ends up harming quality disproportionately, the harness will catch that and we can adjust N or use a smarter criterion (like based on model confidence).
The evaluation harness will be integrated into our CI pipeline – every change runs these tests on a suite of models (small ones for speed and large ones for stress). By making this mandatory, we ensure no optimization silently hurts the system: we’ll either see it in metrics or not merge that change. Only with this concrete benchmark suite can we iterate confidently on the complex scheduler and quantization logic.
9. Ruvector Integration Roles and APIs
Ruvector is envisioned as a learning and memory subsystem for the runtime. We clarify how ruvector will be used in three roles and outline the APIs/storage involved:
	•	A. Policy Memory Store for Learned Thresholds: As the system runs, it could learn optimal settings (like quantization levels or eviction thresholds) based on observed outcomes. Ruvector can serve as a database to store these learned policies. For example, we might use reinforcement learning or simply heuristic logging to adjust something like “optimal KV quantization bit-width as a function of context length and content type.” The data to back this could be high-dimensional (taking into account properties of the input). Ruvector, being a vector database that “learns and improves from every interaction”, can store an embedding of the context or some representation of the session state along with the outcome metrics (latency, quality). Over time, we can cluster or correlate which contexts benefit from which strategy. The API here would look like: after each session or critical event, we form a vector (embedding) of the situation and store it via ruvector.insert(key, vector, metadata), where metadata might include the strategy used and the results (e.g. “quantized at 4k tokens, quality good”). Later, when a new session starts or reaches a decision point, we query ruvector for similar situations: ruvector.query(vector, k=5)might return the closest stored experiences. If those all indicate that, say, KIVI 2-bit was fine, we proceed; if they indicate that quality dropped, we might choose a different path (maybe use SQuat or none). This effectively allows the system to learn thresholds dynamically rather than hard-coding them. Initially, we might run in a logging mode and manually analyze, but eventually the loop can be closed for self-optimizing behavior.
	•	B. Session State Index for Adapter Selection & Cache Locality: In a multi-tenant system, ruvector can act as an index of sessions and their properties, enabling better placement and retrieval. We’ll maintain an entry in ruvector for each active session (or recently active one). The vector could encode the user’s preferences or past usage (for example, which adapters they use frequently, what kind of prompts they have – embedded into a vector). The adapter selection part means if a new request comes in for a certain task, we can quickly find if there’s an adapter that fits (by querying similar past sessions or known task vectors). This could guide routing: e.g. we identify which LoRA to load for a given query if not explicitly specified. The cache locality aspect is about optimizing memory usage: ruvector can help us decide if we should colocate certain sessions on the same GPU because they use similar adapters or content (thus could batch). If ruvector tells us Session X and Y are very similar in embedding (say both are long chats about programming), we might schedule them closely so they can share context or at least ensure their memory pages might reside together for efficiency. The API could be: when a session ends or after N turns, update its embedding in ruvector. When scheduling new requests, query for the nearest neighbor sessions and see if they are on a particular server or GPU – then possibly assign the new request to the same location to reuse loaded adapters or cached knowledge. Essentially, ruvector becomes a smart directory of sessions, supporting decisions like “which adapter to pre-load for this user’s next query” or “which server has the most relevant data for this request”.
	•	C. Witness Log Index for Postmortem & Audit: Every interaction and decision in the system can be logged as a “witness” – a data point that can be later analyzed. We will use ruvector to index these logs in a semantic way. For example, each completed request can produce: an embedding of the prompt+response (to capture the content semantics), the system decisions (quantization used, any interruptions, etc.), and outcome (latency, errors if any). This goes into ruvector as a vector with attached metadata (the log). Later, if we are investigating an incident (say a particular query produced a wrong answer or the system lagged), we can perform an semantic search in these logs. Perhaps an auditor can query “find all requests that had to be interrupted for timeout” or “find similar conversations to this one that got a bad answer”. Ruvector’s ability to store and search high-dimensional data makes it ideal for this. The witness logs stored allow postmortem analysis: after deployment, developers can cluster failures or outliers and derive improvements. They also assist in compliance audits – e.g. if a user reports an inappropriate response, we can find it and related cases by similarity. API-wise, we’d have something like ruvector.log(vector, info) for each request. For retrieval, an admin tool might do ruvector.search(vector_or_text, filter=condition).
	•	Concrete Storage Model: Ruvector in distributed mode would likely run as a service or embedded database. We will integrate it such that on each node (or centrally) there’s a ruvector instance. It can scale horizontally with Raft consensus (per their docs), which is good for reliability. We’ll define the schemas for each entry type (policy, session, log) possibly as separate collections or namespaces in ruvector. The vectors themselves might be of different dimensions (for policy maybe small, for session perhaps large embedding). Ruvector’s learning capability means it might adjust indices or create summary indices automatically – we’ll leverage that to improve query speed as data grows. We’ll also ensure PII or sensitive info is handled – likely the embeddings will be somewhat anonymized or at least not directly storing raw text.
In summary, ruvector will be the intelligent memory of the system: storing past experiences and using them to make the system smarter and more auditable over time. We’ve identified clear APIs for inserting and querying this information, which cements its role beyond a conceptual “nice-to-have”. It becomes a core component for adaptive behavior (learning thresholds), efficient multi-session handling (session index), and traceability (audit log).
10. Model Format Support and Feature Coverage from Day One
Finally, we specify the range of model formats and architectural features the system will support to ensure broad applicability:
	•	Model File Formats: We will support the GGUF/GGML family of formats out-of-the-box, as they are popular for LLMs with quantization and easy CPU loading. Additionally, we will support Hugging Face Transformer models (through either direct loading of PyTorch safetensors or via conversion to our internal format). Our internal format might be a variant of GGUF (since it’s extensible and already used for quantized models like LLaMA). We’ll document a conversion tool for PyTorch checkpoints to our format if needed. Tokenizers will be supported via the Hugging Face tokenizers library or a compatible Rust implementation, ensuring we can handle BPE, sentencepiece, etc., for all major model types. On day one, we aim to run models like LLaMA-2, Falcon, Mistral, GPT-NeoX, etc., which covers a broad set of attention mechanisms and tokenizer quirks.
	•	Rotary and ALiBi Positional Embeddings: We will fully support RoPE (Rotary Positional Embedding) as used in LLaMA/Mistral, including extended context modifications (like NTK scaling). This means our attention kernels can incorporate the rotary transforms on keys and queries – and importantly, if using KVQuant we note that quantization might be applied pre-ROPE as their method suggests, which we have accounted for. We will also support ALiBi (Attention Linear Bias) positions for models that use it (like some older GPT-NEO variants). This requires the attention code to add a static bias for each query-key distance; we will include that in our kernel implementations or handle it in the model data. Additionally, any variants like T5’s relative attention or XLNet’s might not be first priority but our code is written with extensibility so new attention bias patterns can be added.
	•	Grouped Query Attention (GQA): Some newer models (e.g. certain LLaMA-2 70B architecture or Mistral) use GQA, where multiple heads share the same key/query projections (reducing number of distinct heads). Our implementation will handle GQA by allowing the number of heads in the model to differ from number of key-value slots. For example, if a model has 8 query groups for 32 heads, our attention will treat it appropriately (keys/values effectively have shape with 8 groups, and each group is used by 4 logical heads). We ensure our data structures (especially KV cache) and kernels account for this shared head scenario. This might involve minor changes in how we index the KV cache pages (group index vs head index). We’ll test on a model known to use GQA (like Mistral-7B) to confirm correctness.
	•	Sliding Window / Local Attention: Some models (e.g. Longformer, MPT in storywriter mode) use sliding window or block sparse attention patterns instead of full attention. Our system will support at least a sliding window attention mechanism: this means the attention kernel will only attend to the last N tokens instead of all previous tokens for each new token (or some fixed pattern). We’ll implement this by allowing a configurable attention mask or range. If a model config in the format indicates a sliding window of size w, our attention code will simply mask out (or not retrieve from KV) any keys older than w tokens relative to the current. This can actually integrate well with our KV paging: we can simply not hold pages older than the window since they’ll never be used. That yields big memory savings. For block-sparse or other patterns, we may not do a fully general sparse attention on day one, but we will at least be able to support any contiguous window or prefix style restriction easily. This covers many use cases where context is long but only a recent subset is actively attended.
	•	Long Context Strategies: We commit to supporting long context extensions like RoPE scaling (the technique used by GPT-4 32k and projects like Mistral 16k). If a model uses an extended context via scaled RoPE frequencies or repeated patterns, our position embedding implementation will include those formulas (for example, the NTK-aware scaling where sinusoid frequency grows slower to accommodate longer contexts). We’ll verify on known long-context models (like LLaMA-2 32k variants or others) that the perplexity matches reference. Additionally, if models use techniques like position interpolation or segment recurrence, we will handle those in the model forward pass logic. Essentially, any model that adheres to standard transformer architectures up to late-2025 will be supported: this includes improvements like Multi-Query Attention (one key/value per layer shared across heads – we support that as a special case of GQA), and potentially newer architectures like MHA with linear bias, etc.
	•	Kernel ABI and Test Vectors: With these features in mind, our kernel interface is designed to be flexible. The ABI between model and kernels will pass information about the attention type (full vs grouped vs sliding) so the kernel can handle it. For example, we might have an enum indicating if it’s full attention or local with window=256, etc., that the kernel reads. We will prepare test vectors – known inputs and outputs – for each variant. For instance, we’ll take a small model with rotary and ensure our attention output matches a reference PyTorch implementation’s output to high precision. We’ll do the same for a model with ALiBi, and one with grouped heads, etc. These test vectors will be part of our continuous testing to catch any regression in supporting these features.
By outlining these format and feature supports, we ensure that from day one our system isn’t limited to a toy scenario – it can run real modern models with long contexts, different attention schemes, and different fine-tuning methods. This comprehensive coverage influences the design of our kernels and data structures (they must be general enough), but now we have it locked in writing which models and features are priority, removing any ambiguity for implementers.
Locking Key Decisions: Next Steps
To make the frontier plan actionable, we lock in three critical decisions that guide all implementation efforts going forward:
Decision 1: Default Inference Backend for Attention/Matmul
We will use a Mistral.rs-inspired paged attention as the default backend, augmented with quantized KV cache support. This means our primary inference loop will utilize paged memory for KV (improving cache management) and include optimized matrix multiplication for attention. If available (on GPU or specialized hardware), we will integrate custom kernels from projects like vLLM or FlashAttention as a first preference due to their speed; otherwise, our Rust SIMD implementation will handle it. The fallback order is defined: try high-performance device-specific kernels, fall back to our Rust SIMD paged-attention, and finally to a safe reference implementation if needed. By choosing this path, we leverage existing successful techniques in a novel Rust implementation, ensuring we meet performance needs for long contexts by default. The mistral.rs PagedAttention model serves as a reference for behavior and we explicitly include KV cache quantization in this backend from the start. This decision is now fixed and all team members can proceed assuming paged attention + quantized KV is the core of our attention mechanism.
Decision 2: SIMD Kernel Strategy – Use Portable Stable SIMD (Macerator)
We will implement all low-level compute kernels using portable SIMD via the macerator crate (with fallback to pulp if needed). This locks us into a stable Rust solution (no nightly std::simd) that can auto-dispatch to different instruction sets. Macerator’s broad architecture support and generic vector types make it an ideal choice for longevity. Developers will write kernel code in a SIMD-generic way; the crate will produce optimized versions for AVX2, AVX-512, NEON, etc. We accept the slight risk of macerator being less tested than pulp, but given its design and expanded support, it aligns with 2026 reality where stable, cross-platform performance is required. Should std::simd stabilize in the future, we can consider migrating, but for now we standardize on macerator. This decision affects every compute kernel we ship – everyone will use the chosen SIMD abstraction for consistency. Multiversioning will be achieved by macerator’s runtime dispatch, ensuring users get the best performance their CPU can offer. By locking this in, we remove uncertainty about how to write and optimize our math routines: the team can confidently proceed with macerator-based implementations.
Decision 3: WASM Execution Budget – Use Epoch-Based Interruption
We will enforce execution budgets on any WASM-run code using epoch-based interruption (Wasmtime) as the default mechanism. This means every WASM module invocation will be associated with an epoch deadline to guarantee it can be interrupted if it runs too long, without incurring high per-instruction overhead. We choose epoch over fuel because of its superior performance for long-running computations (it doesn’t slow down each instruction) and simplicity for resetting timers. The epoch mechanism will be integrated such that, for example, each inference request has an epoch counter and if it exceeds a time slice, the engine will asynchronously signal a trap in the WASM. This keeps tail latencies in check and prevents any rogue code from hanging the system. This decision is now final: epoch interruption will be used wherever possible. In cases where epoch isn’t available (perhaps an older WebAssembly runtime or special scenario), we’ll use fuel as a secondary, but our platform of choice (Wasmtime) supports epoch, so that’s our standard. Additionally, we commit to handling these interruptions gracefully (cleaning up the WASM instance and returning a controlled error). By locking in epoch-based budgeting, we ensure uniform handling of WASM across the project and avoid debate later about how to handle timeouts – it’s settled now.
With these three key decisions solidified – the attention backend (paged+quantized with mistral.rs influence), the SIMD approach (macerator stable SIMD), and the WASM budget (epoch interrupts) – the plan transitions from exploratory to execution. All other pieces (memory management, scheduling, etc.) will be built on top of these firm choices. We have taken what was previously a research discussion and turned it into an engineering blueprint with concrete defaults and fallback strategies. From here, the team can proceed to implementation, confident that the foundational decisions are made and the system’s behavior is specified in detail.
Conclusion
By addressing each missing piece with concrete implementations and policies, we have transformed the original high-level plan into a detailed design ready for engineering. We have specified how attention and matmul will run, how SIMD will be done in Rust today, how we’ll quantize and manage KV caches, how memory paging and eviction works, the exact WASM integration approach, secure handling of kernel plugins, multi-adapter serving strategies, testing protocols, ruvector’s integration, and comprehensive model support. Finally, we cemented three critical decisions (attention backend, SIMD crate, WASM interruption) to guide the development.
This updated plan is now actionable – each component can be implemented and tested according to the descriptions here. With these details in writing, the project moves out of the realm of research debate and into execution. The result will be a robust, state-of-the-art LLM inference engine that is efficient, scalable, and hard to break, by design.