Latency, throughput and cost move in different directions

Time to first token and tokens per second

Time to first token is dominated by how much input must be processed before generation starts, so it grows with prompt length and with anything prepended by retrieval. Generation rate is largely independent of input length and dominates the tail of a long answer. Which one a user perceives depends entirely on the interface: streaming output makes time to first token the felt latency, while a batch job that waits for a complete response cares only about the total.

Throughput is a different measurement entirely

A single-request timing tells you nothing about behaviour under concurrency. Systems queue, batch and shed load, and the curve of latency against concurrent requests is rarely flat and rarely linear. Measure at the concurrency you expect, and measure the tail rather than the mean: the slowest few per cent of requests set the timeout you will have to configure and the experience your least fortunate users will report.

Cost is charged on both halves of the request

Input and output are typically priced separately and at different rates, so the shape of your traffic decides which rate dominates. A summarisation workload with long inputs and short answers is priced almost entirely on input; a generation workload with a short instruction and a long answer is priced almost entirely on output. Comparing candidates on a single blended figure reverses the ranking as soon as the traffic shape changes.

Measure all three on the same requests

The only comparison that supports a decision is one where the same fixed set of realistic requests is sent to every candidate and all three axes are recorded per request: time to first token, total time, and tokens consumed in each direction. Record the timestamps too. Shared infrastructure varies by hour and by day, and a benchmark run without a date is a measurement of an afternoon.