What Happens When You Call Any LLM API: The Full Journey in 400ms

You type a prompt. You hit send. About 400ms later, you get your first token back.

Most engineers treat that 400ms as a black box. But understanding what's actually happening inside it makes you a dramatically better AI engineer — because it explains why latency varies, why costs work the way they do, why some prompts are slower than others, and where failures actually occur.

Here's the complete journey.

Stage 1: API Gateway (~5ms)

Your request arrives at the API gateway. Three things happen immediately:

• TLS termination — the HTTPS connection is established
• Authentication — your API key is validated
• Request validation — the schema is checked, max_tokens is verified

The gateway also hands your request to the Rate Limiter, which checks your TPM (tokens per minute) and RPM (requests per minute) limits. This is where you get a 429 Too Many Requests error — it happens before your request even reaches a GPU.

Also at this stage: the billing meter starts. The moment your request passes the gateway, you're being charged for input tokens.

Tip: the ~5ms gateway time is why identical API calls can have slightly different latencies — network routing, authentication cache hits, and rate limiter state all vary.

Stage 2: Load Balancer (~2ms)

Your validated request hits the load balancer, which routes it to one of multiple GPU clusters using geographic routing and a least-connections algorithm. This is why:

• Identical prompts can have different response times — you may land on a different cluster with different current load
• Latency varies by region — the cluster may not be in your geographic region

Stage 3: Tokenization (~3ms)

Your text gets converted to token IDs. "Hello world" becomes [15339, 1917].

Two important facts here:

• Different providers use different tokenizers — the same text may tokenize differently on OpenAI vs Anthropic vs Google
• Token count = your cost — the tokenizer output determines exactly how many input tokens you're billed for
• Context window limit is checked here — if your prompt exceeds the model's context window, the request fails at this stage

Stage 4: Model Router (The Hidden Layer)

This is the layer most engineers don't know exists. Every provider with multiple models has a model router that inspects your request and directs it to the appropriate cluster:

• Large model request → Heavy inference cluster (multi-GPU)
• Small model request → Optimized cluster (single GPU)
• Embedding request → Dedicated embedding cluster

Capacity-aware routing, queue management during peak load, and model version pinning all happen here. This is why model version matters more than you might think — routing logic can change between versions.

Stage 5: Inference Engine (~300–800ms — this is 95% of your wait time)

This is where inference actually happens. It has four sub-stages:

Prefill Phase

All input tokens are processed in parallel. Attention matrices are computed. The KV Cache is generated — a snapshot of the attention patterns that will be reused during generation.

This is why long prompts have higher TTFT (time to first token) — more tokens to prefill before generation begins.

Decode Phase (Autoregressive)

The model generates one token at a time. Each step: KV Cache from previous tokens + current token → Attention Layer → FFN Layer → Softmax → Sample Next Token.

The loop: The → capital → of → France → is → Paris

This autoregressive loop is why streaming exists — each token is sent to you as it's generated, giving you faster perceived response even though the total time is the same.

Temperature controls randomness at the sampling step. top_p and top_k sampling also apply here.

Attention Mechanism

32–128 attention heads running in parallel. Multi-head attention with Flash Attention for memory efficiency. KV Cache avoids recomputing attention for past tokens. GQA/MQA in newer models further reduces memory usage.

Hardware Layer

A100/H100/H200 GPUs. HBM memory (90GB+) holds the KV Cache and model weights. Tensor parallelism splits the model across multiple GPUs for large models. NVLink/NVSwitch handles high-speed GPU communication.

GPU compute costs $2–3/hour — this is why inference costs what it does.

Stage 6: Post-Processing (~5ms)

Generated token IDs are converted back to text (detokenization). Stop sequences are checked. The content moderation safety classifier runs — and it can block the response here. Every major provider has this, which is why some outputs trigger content filters while logically similar ones don't.

Stage 7: Response & Billing (~5ms)

The response is packaged as JSON. Usage metadata is attached (input tokens, output tokens, model version, latency). The response travels back through the load balancer and TLS.

The billing math:

• Input cost: input tokens × price/1K
• Output cost: output tokens × price/1K (usually 3–5x more expensive than input)
• Total = input cost + output cost

Prompt caching reduces cost at most providers. Batch APIs offer 50% off for non-urgent async workloads.

The Latency Breakdown

Inference is 95% of your wait time. Everything else is infrastructure tax.

What This Means for You

• Latency variance isn't random — it's routing, cluster load, and sequence length
• Prompt length matters — longer prompts mean longer prefill, higher TTFT
• Output length matters more for cost — output tokens are 3–5x more expensive
• Prompt caching is one of the highest-ROI optimizations available
• Streaming doesn't reduce total time — it reduces perceived latency

You type a prompt. ~400ms and 14 infrastructure layers later, you get your answer. Now you know why.

Up Next

Article 09: Claude Code vs Codex CLI vs Gemini CLI — The Agentic CLI Wars

Three CLIs. Three philosophies. One shared pattern underneath. April 2026's practitioner breakdown of the tools competing to become the standard agentic coding interface.

Jason Newell is an AI practitioner, builder, and writer covering agentic systems, developer tooling, and the future of AI engineering.