The setup was fairly straightforward: one NVIDIA L40S serving the model through vLLM, LiteLLM in front of it for authentication and per-user API keys, and an OpenAI-compatible endpoint that people could plug into whatever client they wanted.

It worked.

The problem was that it was generating at around 19 tokens per second.

On an L40S.

19 tok/s is usable. Nobody was sitting there wondering whether the server had died. But once you start using the model for coding or agentic workflows, it gets painful very quickly. An agent might call the model repeatedly, run a tool, come back to the model, run another tool, and so on. Slow generation gets multiplied across the entire loop.

So I started looking into why it was so slow.

A quick disclaimer before I get into it: I’m not an inference engineer. I don't work on inference performance for a living, and this was my first time properly digging into model quantization, kernels and speculative decoding like this. I did it because the system I was responsible for needed it, and I learned a lot along the way.

If you spot something wrong in my methodology or explanation, please leave a comment. I’d much rather correct the post than confidently keep a bad explanation in it.

First, was it actually slow?

Before changing anything, I wanted a number I could reproduce.

Our original production deployment used the official Qwen3.8-27B FP8 checkpoint.

I used a deliberately simple serving benchmark:

• 1,024 input tokens

• 1,024 output tokens

• concurrency 1

• temperature 0

• forced generation with --ignore-eos

• three requests

Later, when I was preparing this post, I reran the benchmark against the old production container, which was still alive on our second GPU for rollback.

It gave me:

18.66 output tok/s

with a mean TPOT of 53.42 ms.

So the ~19 tok/s we had been seeing wasn't just the UI feeling sluggish. The backend really was that slow under this test.

At this point I was starting to suspect we had bought a very expensive space heater.

Why was FP8 this slow?

This was the first useful rabbit hole.

I had assumed that an FP8 model on an L40S should naturally be a good combination. The model fit comfortably in memory, the L40S supports FP8, and nothing looked obviously broken.

Then I started paying more attention to what vLLM was actually doing.

The official checkpoint was using block-scaled FP8. On our L40S, vLLM selected a Triton block-scaled FP8 matrix multiplication path. The startup logs were also warning that there wasn't a tuned L40S W8A8 configuration available for that path.

That made the next experiment pretty obvious: try a checkpoint that could use a better kernel on Ada.

I went with:

dbirks/Qwen3.8-27B-W4A16-AutoRound

Most of the big weight matrices are quantized to 4-bit while activations stay in BF16. Some parts of the model, including things like the MTP head and lm_head, remain BF16.

The interesting bit for our L40S was that vLLM could run the quantized linear layers through Marlin.

Same GPU. Still Qwen3.8-27B.

Throughput jumped to roughly 43 tok/s.

That's more than twice what we were getting before, before touching speculative decoding at all.

The first big improvement came from changing the checkpoint and kernel path.

Then I started playing with MTP

Qwen3.8 has native MTP support, so I wanted to see how far speculative decoding could take it.

The rough idea is simple: instead of having the target model move forward exactly one token at a time, you speculate several future tokens and then verify them. If enough guesses are accepted, generation gets a lot faster.

I started with three speculative tokens.

That jumped straight to around 90.5 tok/s.

Which was already kind of ridiculous compared with where this started.

Then I did what any reasonable person would do after getting a 2× improvement.

I changed the number from 3 to 4.

Then 5.

Then 6.

Then 7.

MTP sweep

The gains were tapering off as I increased the speculative length, but every step still helped.

MTP7 got us into the 116–118 tok/s range.

If I hadn't found DFlash2 shortly afterwards, that probably would have been the production configuration.

Unfortunately, I found DFlash2

DFlash2 uses a separate draft model to propose a block of future tokens for the target model to verify.

There was already a Qwen3.8-27B DFlash2 drafter available:

incoai/Qwen3.8-27B-DFlash2

Great.

Except our version of vLLM didn't support the Qwen3 DFlash2 path.

The model card pointed to vLLM PR #52816, which meant I was now building a custom vLLM image.

This part took longer than the benchmark.

My first attempt was essentially a full source build. I let it compile for more than 2,700 seconds before deciding I had probably proved my commitment to the cause and killing it.

I eventually got the PR build working using vLLM's precompiled path. The final image build took around two minutes.

Then I had to check that CUDA was working, that the Qwen3 DFlash2 speculator was actually present, that DFlash2DraftModel resolved correctly, and download another ~3.6 GB draft checkpoint.

At some point later, one of the benchmark environments also failed because it didn't have pandas.

Nothing says cutting-edge inference work like getting stopped by a dataframe library.

Eventually, though, it ran.

And the first warm results were around 184 tok/s.

That was high enough that I didn't trust the comparison yet.

Making the comparison fairer

There was an obvious problem with comparing that ~184 number directly with my earlier MTP results.

DFlash2 required the newer PR52816 vLLM build. The earlier experiments had been run on the older environment.

So I reran the configurations I actually cared about using the same vLLM runtime.

Same W4A16 target.

Same L40S.

Same 1K input / 1K output benchmark.

Same concurrency.

Three measured runs for each setup.

Same-runtime comparison

DFlash2 was about 61.9% faster than MTP7 in this test.

That was the result that convinced me to take it seriously for production.

The whole journey

There are two sets of numbers in this post that are easy to mix up, so here is everything together.

The headline 18.66 → 192.40 tok/s covers everything we changed between the original deployment and the final one.

That includes the switch from FP8 to W4A16, the Marlin kernel path, a newer vLLM runtime and DFlash2.

If you specifically want to measure what DFlash2 did in a controlled comparison, use the same-runtime numbers:

42.93 → 191.47 tok/s

And if you want DFlash2 versus the best MTP configuration I tested:

118.27 → 191.47 tok/s

I think that distinction matters, especially when putting a big 10× number in the title or a tweet.

Before and after

Before writing this post, I still had both deployments running.

The old FP8 backend was sitting on one L40S as a rollback option, while the new W4A16 + DFlash2 backend was running on the other.

That gave me a nice opportunity to rerun the exact same benchmark against both live containers.

Live before/after test

The TTFT is actually worse on the new configuration:

230.70 ms → 324.37 ms

But once generation gets going, the difference is huge.

TPOT went from:

53.42 ms → 4.89 ms

And the DFlash2 run had a 94.07% acceptance rate with an average acceptance length of 7.58.

For this workload, the drafter was doing very well.

The fresh DFlash2 run also landed at 192.40 tok/s, which was reassuringly close to the 191.47 tok/s average from the earlier controlled experiment.

Then Hermes told me I was only getting 40 tok/s

After all this, I added a little tokens-per-second indicator to the Hermes client I was using.

It started showing around 38–42 tok/s during real agent sessions.

For a moment I thought something had gone horribly wrong again.

It hadn't.

The ~192 tok/s benchmark and the ~40 tok/s agent number measure very different things.

The serving benchmark is one long controlled generation.

A real agent turn can involve:

• model generation

• time to first token

• multiple separate API calls

• shell commands

• file reads and writes

• searches

• tool execution

• network round trips

• waiting between one agent step and the next

One of my agent turns, for example, ran for roughly 66 seconds and generated around 2,700 output tokens across the whole process. That works out to around 41 tok/s end to end.

A large chunk of those 66 seconds isn't model decoding at all.

This also made me appreciate how painful the original backend must have been for agents.

At ~19 tok/s, every model call was slow. Then the agent would call a tool and hit the model again. Then again.

A slightly slow chatbot is annoying.

A slow model inside a long agent loop can make the whole thing feel broken.

That was ultimately why I cared about this in the first place.

I also tried benchmarking this more properly and accidentally benchmarked my patience

The 1K → 1K benchmark is intentionally synthetic.

It's useful because I can hold the workload constant and compare inference configurations cleanly, but obviously people do not normally send 1,024 random tokens to a model and demand exactly another 1,024 tokens back.

So I wanted a larger real-prompt benchmark as well.

I tried NVIDIA SPEED-Bench at concurrency 1.

Eventually the baseline reached roughly:

994 / 1,536 prompts

after:

22 hours

It still wasn't done.

I killed it.

At some point your benchmark stops measuring the server and starts measuring your willingness to give up a GPU for several days.

So when I use 192.40 tok/s in this post, I'm talking about this specific C1, 1,024-input / 1,024-output benchmark.

I would not expect 192 tok/s from every coding session, agent run or company chat.

Prompt length changes things. Reasoning changes things. Cache state matters. Concurrency matters. Speculative acceptance matters.

The number is useful because the conditions stayed fixed while I changed the inference setup.

Getting it into production

A fast benchmark wasn't enough for me to swap the backend.

Our production configuration has a bunch of things the synthetic test deliberately ignores:

• 131K context

• FP8 KV cache

• prefix caching

• CPU KV offload

• Qwen reasoning parsing

• automatic tool calling

• the Qwen coder tool parser

• our normal chat template

I also changed the default reasoning effort from medium to xhigh.

Generation had become fast enough that I was much happier spending some of that gain on better reasoning quality.

Before moving traffic, I tested the new backend directly.

Normal chat worked.

xhigh reasoning worked.

The reasoning content stayed separate from the final answer.

Tool calling returned proper OpenAI-compatible tool_calls.

Structured JSON parsed successfully.

Multi-turn conversation worked.

Once I was happy with that, I brought the new backend up alongside the old one.

Our architecture already looked roughly like this:

Users
  ↓
Traefik
  ↓
LiteLLM
  ↓
vLLM

Users talk to LiteLLM using the same public model alias:

qwen3.8-27b

So the migration was pleasantly boring.

The old backend remained available for rollback.

The new backend came online separately.

Then LiteLLM's internal upstream changed from the old vLLM container to the new one.

Nobody needed a new API key.

Nobody needed a new model name.

Nobody needed to change Hermes or any other client.

That's probably the least exciting part of this entire experiment, which is exactly what I wanted from the production part.

A few things I didn't expect

I went into this thinking I would probably find a configuration flag or two that we had set badly.

The first huge improvement came before speculative decoding was even involved.

Moving from the official FP8 checkpoint to W4A16 + Marlin took us from around 19 tok/s to 43 tok/s.

That surprised me.

I had mentally put "FP8 model + datacenter GPU" into the bucket of things that should already be fast. I hadn't paid enough attention to the exact quantization format and which kernel vLLM was actually using on our hardware.

Then speculative decoding took over:

Official FP8                  18.66 tok/s

W4A16 + Marlin                42.93 tok/s

W4A16 + MTP7                 118.27 tok/s

W4A16 + DFlash2              191.47 tok/s

Fresh DFlash2 retest          192.40 tok/s

All on one L40S.

No H100.

No second GPU needed to make a single request faster.

Just a lot of experiments and quite a few wrong turns.

I suspect someone who actually does inference engineering every day could have arrived at some of these decisions much faster than I did.

But that was also the fun part of doing it for the first time. I now understand far more about what is actually happening between "I downloaded an FP8 model" and "tokens are coming out of the GPU."

And unfortunately, after all of this, I can no longer blame the L40S.

Reproducing the benchmark

For anyone who wants to reproduce the basic controlled test, this was the shape of it:

vllm bench serve \
  --backend openai \
  --base-url http://127.0.0.1:8000 \
  --endpoint /v1/completions \
  --model <MODEL> \
  --served-model-name qwen3.8-27b \
  --tokenizer <MODEL> \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 3 \
  --max-concurrency 1 \
  --temperature 0 \
  --ignore-eos

For my autoregressive/MTP7/DFlash2 comparison, I kept the vLLM runtime, W4A16 target and benchmark conditions fixed.

If you run this on another GPU, another model, or even another kernel implementation, I would expect different results.

Which is why this whole thing started in the first place.

I assumed I knew roughly how fast our stack should be.

Then I measured it.