Making a large language model 40% faster on a single GPU

On 25 September our inference server, a single RTX PRO 6000 Blackwell GPU running a language model of about 27 billion parameters, answered one user at 111.4 tokens per second. On 5 October it answered at 158.4. Same GPU, same model, same quality. This post is about what changed in the results, measured the same way every time.

The starting point

On 25 September we took a baseline with a fixed benchmark: one user, then eight users at once, then a load test to see how many conversations the GPU could hold. That baseline was already a good setup. It was well ahead of the standard FP8 configuration for the same model, which ran at 83.5 tokens per second on this card.

We wanted to know how far one GPU could really go. Hardware is the expensive part of running a model, so every extra conversation per card matters to whoever pays for it.

The results

Measure	Before (25 Sep)	After (5 Oct)	Change
One user, tokens per second	111.4	158.4	+42%
Eight users together, tokens per second	708.5	957.6	+35%
Conversations held at about 4k context (latest step)	56	88	+57%

Compared with the standard FP8 setup, a single user now gets almost 1.9 times the speed. In practice, answers stream noticeably faster and the same card serves more people before anyone has to wait.

The capacity figure comes from our most recent change, deployed on 4 October. On its own it lets the GPU hold 88 conversations of around 4,000 tokens at the same time instead of 56.

Quality stayed where it was

Speed is easy to get if you are willing to give up accuracy, so we did not let ourselves. Every change had to pass the same quality gate before it went live:

• Grade-school maths (GSM8K) stayed within 0.3 points of the previous version, which is inside the normal noise for that test.
• Long-context recall passed 9 out of 9 times, with a hidden fact placed in documents of up to 250,000 tokens.
• Answers with and without caching matched in every consistency check.

If a change made the model faster but failed any of these, it did not ship. Several did not.

How we got there, briefly

We wrote our own GPU code for the parts of the model where the time goes, and tuned how the server runs the model on this particular card. The rule was simple: every change is tested against the version before it, on the same benchmark, with nothing else changed. If it is faster and the quality checks pass, it goes in. If not, it is rolled back.

That discipline matters more than any single trick. It means the numbers in the table are comparable, and we can tell you which change did what.

Long conversations, without the wait

We are also working on what happens when someone comes back to a very long conversation. In our tests, a 150,000-token conversation now resumes in about one second instead of around 51 seconds, with the same answer it would have given before. That part is still in testing and not yet in production.

Why it matters

A faster model on the same hardware means lower cost per conversation, more users per GPU and less reason to buy a second card before you need one. For teams running their own models, on their own hardware, that is often the difference between a project that pays for itself and one that does not.

We are continuing this work on the same GPU and will share the next results the same way: before, after and how it was measured.

Running a model on your own hardware and want it to go further? Talk to us about it.