Design a Streaming Chat Completions API

Replies stream over server-sent events (or chunked HTTP) with typed events (content deltas, tool calls, usage, done, errors) and event ids; a dropped client resumes from its last event id, and a cancellation propagates all the way to the worker that is generating.

Stateless gateways hold the long-lived connections and do authentication and rate limiting, a scheduler places requests on inference workers with free capacity (in another cluster when one fails), and overload is signalled with 429 or queueing rather than timeouts.

Per-key request and token budgets live in a shared fast store (token buckets), tokens are reserved from the prompt size up front and corrected when the reply ends, and the hottest keys are not a single hot shard.

Clients send an idempotency key; usage is emitted once per request (on completion or cancellation) as an event with the request id into a durable log, and aggregation deduplicates on that id.

3 billion requests a month is about 1,200 a second on average and 4,600 at peak; at 8 seconds per reply that is roughly 37,000 concurrent streams at peak and almost 2 million output tokens a second, which sizes the gateways and the GPU fleet.

Every functional requirement in the brief is visibly served by something on the board, and the non-functional targets are addressed rather than ignored.

Components are labelled, data flows are drawn as connections between them, and the direction of each flow is unambiguous.