Why LLM Inference Slows Down with Longer Contexts

A systems-level view of how long contexts shift LLM inference from compute-bound to memory-bound You send a prompt to an LLM, and at first everything feels fast. Short prompts return almost instantly, and even moderately long inputs do not seem to cause any noticeable delay. The system appears stable, predictable, almost indifferent to the amount of text you provide. But this does not scale the way you might expect. As the prompt grows longer, latency does increase. But more importantly, the system itself starts behaving differently. What makes this interesting is that nothing external has changed. The model and hardware is same. But the workload is not. As sequence length grows, the way computation is structured changes. The amount of data the model needs to access changes. And the balanc