Running Disaggregated LLM Inference on IBM Fusion HCI

Prefill–Decode Separation, KV Cache Affinity, and What the Metrics Show Getting an LLM to respond is straightforward. Getting it to respond consistently at scale, with observable performance, that’s where most deployments run into trouble. Traditional LLM deployments often struggle with scaling inefficiencies, high latency, and limited visibility into where time is spent during inference. Red Hat OpenShift AI 3.0 introduces a new inference architecture built around llm-d (LLM Disaggregated Inference), which separates the Prefill and Decode phases of LLM inference into independently scalable pod pools. This approach addresses key challenges by isolating compute-heavy and memory-bound workloads, improving KV cache reuse across requests, and enabling fine-grained observability into each stage