Cached prompts distort a latency comparison

Caching operates at several layers: an exact-match response cache, reuse of a processed prefix shared by many requests, and whatever the surrounding application does on its own. All of them reward repetition, and a latency test is repetition by construction. The second run of an identical prompt can be dramatically faster than the first without a single thing about the model having changed.

Test the state you will actually be in

Decide which condition your deployment lives in and measure that one deliberately. If most real traffic shares a long fixed prefix, warm the prefix first and report warm numbers - that is your reality. If every request is distinct, vary the input on every run so nothing can be reused, and report cold numbers. Reporting whichever you happened to get is how a system acquires a reputation for speed it will not reproduce in production.

Say which you measured

Cold and warm figures for the same candidate can differ by more than the gap between candidates, so a timing without a stated cache condition is not comparable to anything. Record it beside the date, the concurrency and the input lengths.