LLM KV Cache Systems Engineer

About the role We are looking for an LLM KV Cache Systems Engineer to design and build KV Cache subsystems for large-language-model inference. You will work on memory allocation, lifecycle management, reuse, compression, transfer, tiering, and KV-aware scheduling across single-node and distributed serving systems. This is a systems role. Direct source-level experience with an LLM KV Cache subsystem is more important than familiarity with serving-framework APIs alone. What you will do - Design block/page allocators, KV Cache lifecycle management, fragmentation control, eviction, prefix caching, cross-request reuse, and preemption/recovery mechanisms. - Optimize KV layouts, GQA/MQA handling, quantization, compression, or sparsity while balancing accuracy, capacity, memory bandwidth, and compute overhead. - Build KV transfer and disaggregated-serving mechanisms, including prefill/decode separation, RDMA/NVLink/CXL transport, and GPU/CPU/SSD memory tiering or offload. - Develop KV-aware request routing, batching, admission control, scheduling, and multi-tenant isolation under latency and SLA constraints. - Integrate and productionize KV technologies in systems such as vLLM, SGLang, TensorRT-LLM, LMDeploy, Mooncake, or DistServe, with observability and failure handling. What we are looking for - Master’s degree or Ph.D. in Computer Science, Computer Engineering, Distributed Systems, High-Performance Computing, or a related field. - Deep understanding of Transformer attention, prefill/decode execution, memory behavior, and modern LLM serving architectures. - Direct source-level design, implementation, or research experience in an LLM KV Cache subsystem. - Proficiency in C++/Python, Linux, CUDA or equivalent accelerator development, and distributed-system performance engineering. - Demonstrated results in at least one KV-specific metric: capacity, hit or reuse rate, fragmentation, transfer bandwidth, TTFT, TPOT, throughput, latency, memory cost, or service-level efficiency. Preferred qualifications - Experience with disaggregated inference, distributed KV transfer, cache-aware routing, or GPU/CPU/SSD tiered KV storage. - Core open-source contributions, publications, patents, or large-scale production deployment in LLM serving or KV Cache systems.