Breaking the KV Wall
Offload the KV cache to petabyte scale storage colocated with your own GPUs, and cut the GPU time spent on recomputing, on hardware you already own.
Modern LLM use cases are evolving to multi turn conversations, each conversation spanning millions of tokens and hundreds of turns. However each new turn forces the GPU to recompute the previous turn's KV cache. At a 1M context window, this becomes a time consuming process taking several seconds of precious GPU compute burned in a request's prefill phase.
Within a single request, each token generates the K/V tensors that later tokens attend to collectively, making up the KV cache. Inference engines attempt to keep a local prefix cache to improve throughput, but it does not scale due to 2 reasons:
Caches don't cross hosts: Large scale LLM deployments run across many GPU hosts which don't share KV cache between them, therefore the same prefix arriving on a different GPU sees does not benefit from local prefix cache leading to KV recomputation and wasted GPU time.
GPU RAM is not enough: Production deployments see enormous amounts of requests in a short period of time. The limited on device RAM and disk hold only a small set of KV cache, a prefix that recurs after many intervening requests has already been evicted, forcing GPU recomputation.
We introduce OpenLake, a high throughput distributed store that can retrieve KV state at low latency and high throughput. For such a store to beat recomputation, it has to clear 2 bars:
Low latency and high throughput serving: The time to first token must be in the order of microseconds, to keep inference times low, and the store sustains a high throughput of KV blocks to serve concurrent requests.
Durable and Cheap: It must serve petabytes of data on non-GPU/ heterogeneous hardware configuration (at a fraction of the GPU cost it displaces).
To address the above requirements, latency on the hot path must be minimized, by removing memory copies at (kernel and nic) encountered during typical TCP movement. OpenLake uses RDMA (Remote direct memory access) over Infiniband and RoCEv2 settings to store and serve petabyte scale data. Further data is replicated and striped across several nodes making sure cheaper non GPU nodes can serve data at high throughput/low latency to GPU nodes on demand.
KV Requirements:
To understand the effectiveness and cost efficiency of KV offload, we take an example of the Gemma 4 31B model. Assuming a 256K context window:
Per Layer per token: 2 (K and V) × 16 KV heads × 256 head dims × 2 bytes = 16,384 bytes = 16 KiB.
Across 10 FA layers (60 Layers total): 10 × 16 KiB = 160 KiB per token (FA)
Across 256K context window: (160 KiB × 256,000) + (800 MiB) = 42.8 GB.
To serve single full length conversation on Gemma 4-31B, 43GB of KV cache is required. Modern H100 features only 80 GB of HBM ram. Despite model sharding strategies like tensor parallelism, this means the GPU can hold only 1 full length conversation in the GPU prefix cache. Offloading this to external storage scales up this number to tens of thousands of requests.
For repeating conversations and high number of users, recomputing the request on GPU requires 8.9s on an A100 SXM 80GB. But fetching the same request from RDMA based storage takes 1s. Therefore it is always beneficial to retrieve over recomputing on GPU. Over a PCIe of ~64 GB/s, this transfer reduces to under <1s. RDMA helps eliminate additional TTFB latency and CPU utilization overhead on the data copy. The benefits are further amplified when the model is tensor sharded across several GPUs, where each GPU retrieves a smaller KV cache. With OpenLake enabled, the GPU can retrieve the KV state from local disk or remote disk at millisecond level latency. Making KV retrieval with OpenLake 10x-80x faster than recomputation on long horizon tasks and high throughput settings.
Figure 1) TTFT: Recompute vs OpenLake in seconds
Deferred Materialization:
We introduce deferred materialization, a lossless client side fused compression for KV blocks. The KV retrieval latency from the external storage is very critical since it blocks the prefill stage of inference and affects the overall inference latency. Conventional systems are bound by the NIC transfer rate (typically 400 Gib/s or 50GB/s) on high end systems. Therefore the KV retrieval time is bound by this number. For example a KV cache of size 50GB will take 1 second due to the 50GB/s NIC bandwidth. This means the user must wait at least 1s for prefill, in addition to the decode latency.
OpenLake beats this and introduces “Logical Throughput” (many multiples of the physical throughput) by deferring the materialization of the KV cache on the client.
The OpenLake connector on the client side, performs a lossless compression of the KV cache while streaming the data out from the GPU HRAM into the remote storage (leverages the client GPU by firing a cuda kernel for compression). This compresses the cache upto the order of 1.72x (BF16 in our tests) and reduces the amount of bytes sent over the wire. With 50GB/s NIC rate, this reduces the compressed payload from 50GB to ~29GB requiring only 610ms of time to retrieve compared to the original 1 second. Given the highly parallel nature of GPUs, OpenLake achieved a decompression throughput of 600 GB/s on an H100 GPU (making decompression essentially free, since the decompression throughput is significantly higher than the network throughput). Overall, the client GPU defers the materialization of KV blocks on the client, ensuring that a) Storage requirements reduce by storing compressed KV blocks and b) TTFB and data retrieval time improves by 1.7x.
Deferred materialization is able to improve the physical bandwidth of NIC like resources (in this scenario 50GB/s) to a logical throughput of 80GB/s in this scenario. This is purely lossless compression and does not affect the quality of the outputs. Based on our testing on NVIDIA H100 with 50 GB/s NIC bandwidth (400Gib/s), we have been able to achieve a logical bandwidth of 80 GB/s and a compression ratio of 1.72x. Deferred materialization is only available on selective Nvidia GPUs, and we are working on adding support for AMD GPUs.
OpenLake Architecture
To avoid multi hop round trip latency of consulting a centralized metadata store, OpenLake does not use a central metadata storage and avoids single point of failure for small block I/O. The client computes the target node of the request based on a consistent hash ring of the storage targets participating in the cluster. In distributed settings, a quorum is maintained via erasure coding and writes are acknowledged when the EC majority is met. This makes the reads fast, by reading from a single node when the data size is small (which is common for KV blocks). Further, OpenLake follows a tiered storage architecture with hot blocks in node local memory, and older blocks flushed to disk. The network uses Paced RDMA, which is developed by OpenLake which implements credit based RDMA congestion control and ensures bursts of requests do not impact tail latencies on data retrieval. The entire architecture is built on pinned thread per core model and uses io_uring. In unison this provides open lake 2 fold benefits: 1) OpenLake can be run on pure GPU nodes, creating a shared unified cache pool across nodes. 2) KV block read latency is minimized, accelerating inference and freeing GPU seconds for other requests.
Efficiency:
To evaluate the cost improvements, we created a benchmark dataset with a variety of prompts, that repeats the same prefix after N requests. N controls the new requests which the inference engine sees before a prefix is repeated. With a small cache size like that of a GPU VRAM and the local RAM, N is forced to be a small number, since the KV space is limited to hold only the most recent N requests (which can be fulfilled by order of tens of GB).
Offloading the KV to an external store increases the N by many multiples. We then repeat the same prompt sequence with and without OpenLake KV enabled and measure the per request latency, throughput, cost and overall CT (completion time). Based on this we measure the effectiveness of OpenLake on inference latencies.
We found the following improvements with OpenLake with and without OpenLake on Gemma 4-31B.
Figure 2a) Effective Overall Throughput (Amortized)
Figure 2b) Total GPU Seconds Used
Conclusion and Future:
With this release, we are also open-sourcing the OpenLake vLLM and SGLang connectors which can be enabled by passing additional configs to the Inference engine of your choice. The future of OpenLake is opensource, If you would like to discuss issues or would like to get involved, please join us on our discord.
Get started
You can get the benefits of KV offloading with vLLM with 3 simple commands.
For single GPU, non RDMA:
1. Install the connector.
2. Start the OpenLake daemon (run this in the background or a second terminal).
3. Point vLLM at the daemon.
* This runs vLLM against a local OpenLake cluster and does not require InfiniBand. For IB-connected instances, pass an RDMA config (kv_rdma.toml) to openlaked and set openlake_device to your IB device instead of local.
For any additional support, feel free to open an issue on the OpenLake github or send us a message on discord.
GitHub: github.com/openlake-project/openlake
Discord: Invite
Get Involved: See more
Contact Us: Drop a message