Kimi K3 and the New Open-Weights Moat: Inference Efficiency

Moonshot AI released Kimi K3 [1] this month, a 2.8-trillion-parameter open-weights model with a one-million-token context window. The headline number is the parameter count. The number that matters is how few of those parameters move on any given token, and how the model keeps its memory footprint from exploding at long context. I talked through the launch with the panel on IBM's Mixture of Experts podcast, and I wanted to write down the systems reading, because the interesting part of Kimi K3 is not that it is big. It is how it is built to be cheap to run.

Open weights did not stay a generation behind

For most of the past two years, the working assumption was that open-weights models trailed the closed frontier by roughly a year. You could run something good locally, but the best model in the world was always behind an API. That assumption is now wrong, and Kimi K3 is the clearest evidence yet.

On the reasoning and agentic-coding benchmarks Moonshot published, K3 trades blows with GPT-5.6 Sol and Claude Fable 5 rather than trailing them by a release cycle [1]. It leads on some, including a 93.5 on GPQA-Diamond and a 91.2 on BrowseComp, and it falls behind Fable 5 on a few others. I will get to why single benchmark numbers deserve suspicion later. The point for now is directional: the gap between open weights and the frontier has collapsed from a generation to a coin flip, and it happened faster than most enterprise roadmaps priced in.

This is the same arc I wrote about with DeepSeek V4. Kimi K3 is the next data point on that curve, and it pushes the argument further, because it is not just competitive on quality. It is engineered around the cost of serving.

What is actually inside the model

Start with the shape. Kimi K3 is a Mixture-of-Experts model with 2.8 trillion total parameters spread across 896 routed experts, but it activates only 16 of them per token, plus a shared expert [1]. That works out to roughly 104 billion active parameters on any given forward step [2]. The other 96 percent of the weights sit idle for that token.

Fig. 1: Kimi K3 stores 2.8 trillion parameters but routes each token to only 16 of 896 experts. Total capacity sets what the model can know. Active parameters set what each token costs. The gap between the two is the whole economic argument for sparsity.

That gap between stored capacity and active compute is the reason a 2.8T model is servable at all. You pay for 2.8 trillion parameters in memory, but you pay for about 104 billion in arithmetic per token. Capacity is cheap to store and expensive to move, so MoE decouples the two: keep the knowledge large, keep the per-token work small. None of that is new with Kimi K3. What is new is how far they pushed it, and what they did about the two problems sparsity does not solve: the KV cache, and expert routing.

Kimi Delta Attention and the KV cache problem

Here is the part that should interest anyone who has watched a long-context deployment fall over. As I argued in The KV Cache Is the New Bottleneck, once your context runs to hundreds of thousands of tokens, the attention cache, not the weights, is what fills your accelerator memory. A standard transformer stores a key and value vector for every token at every layer, so the cache grows linearly with sequence length and never stops.

Kimi K3 attacks this with Kimi Delta Attention, or KDA. KDA is a linear-attention mechanism that comes out of Moonshot's earlier Kimi Linear work [3]. Instead of keeping a per-token key and value for the whole history, a KDA layer carries a fixed-size recurrent state that it updates as tokens arrive, using a finer-grained gating rule that extends Gated DeltaNet. The state does not grow with the sequence, so the memory cost of long context stops being linear.

The catch with pure linear attention has always been quality: a fixed-size state has to forget things, and full attention does not. So Kimi K3 does not go all-in on KDA. It is a hybrid. Most layers use KDA, and a full Multi-Head Latent Attention layer is inserted every fourth layer across the 93-layer network [4]. The linear layers keep memory and throughput flat, and the periodic full-attention layers preserve the exact recall that linear attention gives up.

Fig. 2: In a full-attention model the KV cache grows with every token. Kimi K3 runs Kimi Delta Attention on most layers, each carrying a fixed-size recurrent state, and inserts a full-attention layer every fourth layer to keep exact recall. The Kimi Linear research reported up to 75 percent less KV-cache memory and up to 6x faster decoding at a one-million-token context.

The numbers Moonshot reported are the kind that change deployment math. The Kimi Linear research measured up to 75 percent less KV-cache memory and up to six times faster decoding at a one-million-token context [3]. Kimi K3 carries this forward and reports up to 6.3x faster decoding in million-token contexts [1]. That speedup is not a bigger GPU or a better kernel. It is an architectural decision about what to store.

There is a systems cost worth naming, because it is not free. A KDA recurrent state is physically much larger than a single token's KV entry, so serving Kimi K3 well means managing two different kinds of memory at once: paged KV caches for the full-attention layers and large recurrent-state blocks for the KDA layers. The vLLM team had to decouple physical block size from the prefix-match unit just to get partial prefix-cache hits to work [4]. The efficiency is real, but it moves complexity into the serving stack rather than removing it.

Routing, residuals, and the training bill

Two more choices are worth calling out, because they are where a lot of the practical gains hide.

The first is how experts get chosen. Large MoE models usually need an auxiliary loss to stop the router from sending every token to the same few popular experts, and that loss comes with a balancing coefficient you have to tune. Kimi K3 replaces it with quantile balancing, which sets each expert's allocation from the quantiles of the router's own scores and removes the hyperparameter entirely [1]. One fewer knob to tune on a model this expensive to train is not a footnote.

The second is Attention Residuals, which let a layer read directly from representations written by earlier layer blocks instead of only from the layer just below it. Moonshot reports this buys roughly 25 percent higher training efficiency for under 2 percent additional cost [1]. Combined with the rest, they claim about 2.5 times better scaling efficiency than Kimi K2, their previous flagship. These are training-side wins, and they matter because they are what let a lab outside the largest Western players reach 3T-class scale at all.

The moat moved, and it is not the benchmark

Put the pieces together and a thesis falls out. Raw capability is no longer where open weights lose. Kimi K3 proves you can reach the frontier on quality with open weights. So if the model quality is a commodity, where is the durable advantage?

It is in the cost of serving, and that cost is set by architecture and hardware, not by leaderboard position.

Fig. 3: Open-weights quality has caught the closed frontier, so quality is converging toward a commodity. The lasting differentiator is the cost of serving a token at long context, which is decided by sparsity, attention memory, and quantization. Kimi K3 competes on that axis, not on a single benchmark.

Look at what Kimi K3 actually optimized. Sparse activation to cut per-token arithmetic. Linear attention to cut per-token memory. Quantization-aware training from the supervised-fine-tuning stage onward, shipping MXFP4 weights with MXFP8 activations so the deployed model is cheap in bits, not just cheap in theory [1]. Every one of those is a decision about the cost of a served token at long context. None of them is a decision about a benchmark score.

This is the same conclusion I reached in The Real AI Cost Problem: the gains that compound come from the full stack, from the attention algorithm down through the number format to the kernel, not from any single layer. Kimi K3 is that argument shipped as a product. It also lines up with why quantization won the compression debate. The winning techniques are the ones that reduce what you move at inference time, and Kimi K3 stacks several of them at once.

Benchmarks, read honestly

The published numbers are strong. On the agentic and reasoning suites, Kimi K3 leads on Program Bench, SWE Marathon, BrowseComp, and document understanding, and it posts a 93.5 on GPQA-Diamond and 88.3 on Terminal Bench 2.1 [1].

Selected results at maximum reasoning effort, as reported by Moonshot [1]. K3 trails Claude Fable 5 on FrontierSWE and HLE-Full and GPT-5.6 Sol on DeepSWE, so this is parity with the frontier, not a clean sweep of it.

I would not read too much into any single figure. These are self-reported, run at maximum reasoning effort, and the field has learned the hard way that benchmark scores are gamed, contaminated, and weakly correlated with the work you actually need done. BrowseComp's 91.2, for instance, used context compaction that triggers at 300K tokens, which is a serving choice bundled into the score [1]. The honest reading is not "K3 is the best model." It is "an open-weights model is now inside the error bars of the closed frontier," which is the only claim that changes anyone's plans.

My take, from the systems side

If your strategy assumed open weights would stay a step behind and you could always pay for the closed frontier when it mattered, Kimi K3 breaks that assumption. The quality is here, in the open, and it is engineered to be cheap to serve. So the question for an enterprise is no longer "which model is smartest." It is "which model gives me the lowest cost per useful outcome on my hardware, with my data, under my control." That is a hardware-software co-design question, and it is the one I keep coming back to. A model you can run on your own accelerators, quantize to your own budget, and keep your data inside of is worth more than a few points on a leaderboard you did not design. The moat was never the model. It is the efficiency of the system you serve it with.

What I would watch next

The architectural bet in Kimi K3 is that linear attention plus periodic full attention is the right long-context design, and that the extra serving complexity is worth the memory it saves. The vLLM preview makes clear that this bet lands most of its cost on the serving stack, which now has to juggle two memory systems and large expert-parallel deployments across 64-plus accelerators [4]. Whether that complexity is worth it outside the largest clusters is the open question, and it is exactly the kind of question that decides which architectures survive contact with production.

What I am sure of is the direction. The competition among frontier models, open and closed, is moving away from who has the most parameters and toward who can serve a token at long context for the least money and power. Kimi K3 is a serious entry in that race, and it is open. For anyone building on top of these models rather than selling access to them, that is the development of the month.

---

References

[1] Marktechpost, "Moonshot AI Releases Kimi K3: A 2.8 Trillion Parameter Open MoE Model With Kimi Delta Attention and 1M Context," July 16, 2026. Source for parameter counts, expert routing (16 of 896), Kimi Delta Attention, Attention Residuals, quantile balancing, MXFP4/MXFP8 quantization, and benchmark results.

[2] Moonshot AI, "Kimi K3 Quickstart," Kimi API Platform documentation, 2026. Model overview and context-window configuration.

[3] Zhang et al., "Kimi Linear: An Expressive, Efficient Attention Architecture," arXiv:2510.26692, October 2025. Introduces Kimi Delta Attention (KDA), the KDA-plus-MLA hybrid, and reports up to 75% KV-cache reduction and up to 6x decode throughput at 1M context. Code: MoonshotAI/Kimi-Linear.

[4] vLLM, "A Preview of Production-Scale Kimi K3 Support on vLLM," July 22, 2026. Source for the 93-layer network, MLA every fourth layer, recurrent-state vs paged-KV serving, and 64+ accelerator deployment.

[5] IBM, "Hugging Face breach: OpenAI's model breaks containment," IBM Mixture of Experts, IBM Technology, 2026. Panel discussion covering the Kimi K3 launch.