Kimi K3 and the New Open-Weights Moat: Inference Efficiency

Moonshot AI's release of Kimi K3 [1] has attracted considerable attention, in part because of its reported scale of 2.78 trillion parameters. Parameter count alone, however, provides an incomplete picture of the model's computational profile and architectural significance. Two design characteristics are more informative: Kimi K3 activates approximately 104.2 billion parameters for each token, and its backbone combines 69 Kimi Delta Attention (KDA) layers with 24 Gated Multi-Head Latent Attention (Gated MLA) layers. Together, these choices reduce per-token computation relative to a comparably sized dense model while moderating the growth of the attention state at long context lengths. They do not, however, make a model of this scale inexpensive or straightforward to deploy. I discussed the initial release with the panel on IBM's Mixture of Experts podcast [6]. This article extends that conversation through a more detailed systems-level analysis, supported by architectural diagrams.

Open weights did not stay a generation behind

For most of the past two years, the working assumption was that open-weights models trailed the closed frontier by about a release cycle. You could run a capable model locally, but the strongest model available was accessible only through a proprietary API. Kimi K3 weakens that assumption.

On Moonshot's reported evaluation suite, K3 is frontier-level on several coding, reasoning, and agentic tasks, and it leads on some, with a 93.5 on GPQA-Diamond and a 91.2 on BrowseComp [1]. K3 does not lead on every benchmark. Moonshot's own report places it behind Claude Fable 5 and GPT-5.6 Sol overall. Even so, an open-weights model competing this closely on difficult tasks is enough to change where the deployment decision is made.

DeepSeek V4 established the first part of this shift, which is open-weight models reaching competitive quality. Once open-weight capability converges with proprietary systems on selected tasks, model selection depends less on a single leaderboard score and more on the cost of serving the model: the memory footprint, the hardware required, and the throughput and latency the deployment can sustain. Kimi K3 is designed around that serving cost.

The spec sheet

Here is the model architecture and configuration.

Two quantities determine most of the model's serving cost. The first is the activated parameter count: only 104.2 billion of the 2.78 trillion parameters are used per token, which keeps per-token compute far below a dense 2.78-trillion-parameter forward pass. The second is the attention structure: linear attention on most layers with periodic global attention, an arrangement that reduces the rate at which the attention state grows at long context.

Sparse computation: latent experts and routing

Kimi K3 is a Mixture-of-Experts (MoE) model. The 2.78 trillion parameters are spread across 896 routed experts per MoE layer, and any single token activates only 16 of them, plus two always-on shared experts [1]. That comes to 104.2 billion active parameters per forward step, about 3.7 percent of the total.

The parameter count hides a detail that shapes serving cost. The two shared experts operate at the full 7,168-dimensional model width. The routed path is narrower: it projects each token into a 3,584-dimensional latent space before the experts run, then projects the combined result back to the model width. Moonshot calls this Stable LatentMoE. The narrower routed path reduces expert-weight bandwidth and the all-to-all communication that expert parallelism needs, which is what makes a pool of 896 experts practical to serve.

Fig. 1: K3 stores 2.78 trillion parameters but routes each token through 16 of 896 experts plus two shared experts. The total is the stored parameter count. The activated number is a rough indicator of the computation one token triggers.

So sparsity is why a model this large can be served at all, but the framing "16 of 896" understates the cost that remains. Keeping all 2.78 trillion weights resident is still expensive in memory, bandwidth, and interconnect. Sparsity limits the subset of the network each token activates, and the latent path shrinks the data that subset must move.

Sparsity does not solve routing on its own. The router must select experts without routing most tokens to the same few, and it must do so without a fragile training penalty.

How the router picks 16 of 896

For every token, the router produces a sigmoid score for each of the 896 experts. Top-k selection runs on each score plus an expert-specific bias, and the 16 winners are kept. Their raw sigmoid scores are then normalized by their sum to form the mixture weights. The selection bias determines which experts receive the token, but it is excluded from those final weights. The routed experts and the two shared experts run as parallel branches, and their outputs are summed into the token's result (Fig. 2).

Fig. 2: The router scores every expert with a sigmoid, adds an expert-specific selection bias, keeps the top 16, and normalizes their raw scores into mixture weights. The routed branch and two full-width shared experts are summed. Quantile Balancing updates each expert's selection bias so its routed token count approaches a target load.

The hard part of routing is balance. Without a balancing mechanism, a router tends to send most tokens to a few popular experts, which wastes the rest of the network and destabilizes training. The usual fix is an auxiliary load-balancing loss with a coefficient that must be tuned, and that coefficient is an additional failure point on an expensive run. Kimi K3 eliminates it. During training, Quantile Balancing updates each expert's selection bias from the distribution of its score margins, so the expert's routed token count drifts toward the target load [1]. This removes a term from the training objective. It does not remove load balancing or the systems tuning that expert parallelism still needs.

Long-context state: KDA and Gated MLA

As I argued in The KV Cache Is the New Bottleneck, once the model weights are provisioned, the attention state becomes the incremental constraint on context length and concurrency. The crossover depends on architecture, cache precision, batch size, and topology, but at a million tokens it is the dominant constraint. So for a model like this, a central design question is how the attention layers hold their memory.

A standard transformer uses softmax attention. During cached decoding, each autoregressive attention layer stores a key and value (KV) for every token seen so far, and each new token attends over all of them. That gives direct global interaction with all cached positions, but the cache grows linearly with sequence length. At a million tokens, it is very large.

Linear attention adopts a different mechanism. Instead of keeping a key and value for every past token, it compresses the history into a fixed-size recurrent state and updates that state as each token arrives. Kimi Delta Attention is Moonshot's version, carried forward from their earlier Kimi Linear work [4]. KDA maintains a fixed-size recurrent matrix: each token first decays selected channels of the previous state, then applies a key-conditioned delta update that erases and rewrites part of it. The state size does not grow with the decoded sequence length. That O(1) property is with respect to sequence length, for the recurrent state, and not a claim about the whole model, activations, or training.

Fig. 3: Softmax attention stores a key and value per token, so its cache grows with sequence length. KDA keeps one fixed-size recurrent state and rewrites part of it each token. The state is constant in sequence length; a finite state cannot preserve arbitrary token-level detail.

A fixed-size state must discard information, because it cannot preserve arbitrary detail about every past token. Relying on linear attention alone therefore incurs a quality cost, which Kimi K3 mitigates by interleaving the two mechanisms: 69 KDA layers and 24 Gated MLA layers, arranged as three KDA layers followed by one Gated MLA, repeated, with a final Gated MLA so the backbone ends on global attention [1]. The KDA layers maintain a fixed recurrent state. The Gated MLA layers restore direct global interaction through Multi-Head Latent Attention, which already compresses the cached representation.

Fig. 4: A full-attention model's KV cache grows on every layer. K3 runs KDA on 69 layers with a fixed recurrent state and Gated MLA on 24 layers whose compressed cache still grows with context. The hybrid reduces the rate of growth; it does not make total attention memory constant.

The hybrid architecture does not make total attention memory independent of sequence length. The KDA state is constant in sequence length, but the 24 Gated MLA layers still hold a cache that grows with context. What the hybrid reduces is the rate of that growth: roughly one quarter of the layers contribute a sequence-length-dependent cache, and those use MLA's compressed form.

Figure 5 isolates that growth term with a toy calculation. Take a generic long-context transformer, not Kimi K3's configuration, and compute its KV cache. A full-attention model at a million tokens can need hundreds of gigabytes. That is memory that cannot be allocated to weights or to serving additional concurrent users. The figure is only the sequence-dependent part of the picture, and it deliberately leaves out recurrent-state storage.

Fig. 5: Illustrative sequence-dependent cache only, for a generic transformer, not K3's dimensions. It excludes KDA recurrent state. K3's global layers use MLA, which compresses this term further. Read it as the growth component, not K3's total attention memory.

The prototype numbers commonly cited come from the Kimi Linear study, on a 48-billion-total, 3-billion-active model under its own evaluation conditions: up to 75 percent less KV-cache memory and up to 6 times faster decoding at a million-token context [4]. Those are architectural evidence for the approach. They are not measured Kimi K3 production numbers, and the two should not be conflated.

The hybrid also adds a serving cost. A KDA recurrent state is physically much larger than a single token's KV entry, so serving K3 well means managing two kinds of memory at once: paged KV caches for the Gated MLA layers and large recurrent-state blocks for the KDA layers. The vLLM team had to decouple physical block size from the prefix-match unit just to get partial prefix-cache hits to work [2]. The efficiency gain holds, but it relocates complexity into the serving stack rather than removing it.

Precision and deployment

The last architectural technique is the number format, which reduces the dominant part of the weight footprint. Kimi K3 uses quantization-aware post-training across supervised fine-tuning and reinforcement learning, rather than quantizing after the fact. The routed MoE expert weights, which are the largest part of parameter memory, are stored in MXFP4, and the inputs to those experts are computed in MXFP8 [1].

This is not a whole-model 4-bit scheme. Attention projections, the latent-MoE projections, the shared experts, and the routers stay at higher precision, because they are more sensitive to quantization. The mixed scheme shrinks the biggest weight component while protecting the parts that would degrade.

The MX in those names is microscaling. Instead of one scale per value or one per tensor, a microscaling format groups 32 values into a block that shares a single 8-bit scale (Fig. 6) [5]. The shared scale is what lets a 4-bit weight keep usable dynamic range: the four bits hold each value, and the shared scale holds the block's magnitude. MXFP4 uses an E2M1 element (one sign, two exponent, one mantissa bit); MXFP8 uses E4M3, which keeps more range for the activations that need it.

Fig. 6: K3 applies MXFP4 to routed-expert weights and MXFP8 to their inputs. Attention projections, latent-MoE projections, shared experts, and routers remain at higher precision. This is a targeted mixed-precision scheme, not a whole-model 4-bit conversion.

In Why Quantization Won, I argued that the compression methods worth using are the ones that cut how much data you move at inference time. Expert-only MXFP4 does exactly that, applied to the largest weight component. Because the quantization happens during training rather than after it, the model learns to work with the lower precision. That is what quantization-aware training is for.

The model is still large in absolute terms. Production evidence from vLLM's launch post reveals these deployment performance numbers: 118 tokens per second without speculative decoding, and 370 tokens per second with DSpark speculative decoding, on a specified 16-GPU GB300-NVL72 setup, a 3.14 times gain from speculative decoding plus extensive optimization [2]. Deployment scale varies sharply by accelerator memory and topology. vLLM's guidance identifies eight NVIDIA B300 or eight AMD MI355X GPUs as the easiest configurations, with 16 B200 GPUs also supported; larger production topologies use more.

A training-efficiency result, not the serving argument

One of Moonshot's changes is often misattributed to the serving argument. Attention Residuals (AttnRes) are a training-time result, and they change how information flows through depth. Standard residual accumulation adds preceding transformations with fixed unit weights. Attention Residuals instead use a learned pseudo-query at each sublayer to compute softmax weights over root mean square (RMS) normalized earlier representations, and K3 uses the blockwise form, which attends across block-level summaries to bound memory and communication cost (Fig. 7). The two results below measure different things. In the Attention Residuals study, Block AttnRes matched the loss of a baseline trained with 1.25 times more compute, and its optimized implementation reported under 2 percent inference-latency overhead [3]. Those are separate numbers. Moonshot also reports that the full set of architectural changes gives about 2.5 times better scaling efficiency than Kimi K2. These are improvements on the training side.

Fig. 7: A standard stack passes information from block to adjacent block. Attention Residuals add a learned attention over RMS-normalized earlier block representations, so a later block can weight what it reads from earlier ones instead of relying only on its neighbor.

Reading the benchmark numbers

On Moonshot's reported evaluation, K3's published numbers are strong. It posts the top result among the compared models on Program Bench, SWE Marathon, and BrowseComp, and the highest reported OmniDocBench score for document understanding, at 91.1. It also records a 93.5 on GPQA-Diamond and an 88.3 on Terminal Bench 2.1. On software engineering it is more mixed, ranking second on FrontierSWE behind Claude Fable 5 and behind both Claude Fable 5 and GPT-5.6 Sol on DeepSWE [1].

Selected results at maximum reasoning effort, as reported by Moonshot [1]. K3 leads Program Bench, SWE Marathon, BrowseComp, and OmniDocBench; it ranks second on FrontierSWE behind Claude Fable 5 and trails both Claude Fable 5 and GPT-5.6 Sol on DeepSWE. The report places K3 behind both overall.

I would not read too much into any single figure. These are self-reported and run at maximum reasoning effort, and individual scores can be sensitive to contamination, harness design, reasoning budget, and serving configuration. BrowseComp's 91.2, for instance, used context compaction that triggers at 300K tokens, which is a serving choice bundled into the score [1]. The SWE Marathon result comes from a pre-release calibration branch rather than the final model. The results are uneven. K3 leads on several tasks and comes close on others, while Moonshot's own report still places it behind the strongest proprietary systems across the full suite. Read that way, the leaderboard rank matters less than the size of the gap, which is now small enough that cost and deployment can decide the choice.

Where the real difference is now

Kimi K3 does not make a 2.78-trillion-parameter model small. It makes that scale more tractable, and it works on several costs at once: latent sparse experts cut routed computation and communication, KDA replaces per-token KV growth with a fixed recurrent state on most layers, and expert-only MXFP4 shrinks the largest block of weights. These act on different bottlenecks, and how much each one returns depends on the serving implementation.

Fig. 8: Conceptual, not to scale. As open-weight quality converges with proprietary systems on selected tasks, model selection turns on deployment economics: memory footprint, interconnect traffic, throughput, latency, and control.

Each of these gains carries a cost. The model still needs large aggregate memory, expert-parallel communication, fused kernels, and a cache manager that coordinates recurrent state with paged MLA storage. Three questions decide how much the architecture is worth in practice. Can the hybrid cache and expert-parallel stack deliver attractive cost at ordinary enterprise concurrency, not only on elite clusters? How portable are the MXFP4 and KDA gains across hardware and inference engines? And do the long-context quality advantages survive independent, workload-specific evaluation?

So K3 shifts the cost of scale rather than removing it. The cost shifts to expert placement, kernel quality, interconnect traffic, and hybrid cache management, which is the full-stack picture I traced in The Real AI Cost Problem. What is different this time is that both the weights and a working serving stack are open, so the tradeoff is one you can measure on your own hardware rather than accept on trust.

Techniques and tradeoffs

Each technique acts on a different bottleneck. Together they make 2.78 trillion parameters more tractable to serve, but none of them makes the model cheap.

---

References

[1] Kimi Team, "Kimi K3: Open Frontier Intelligence," arXiv:2607.24653, July 27, 2026. Primary source for parameter counts, the 69 KDA + 24 Gated MLA backbone, Stable LatentMoE and the 3,584-dimensional routed latent path, two shared experts, sigmoid routing with Quantile Balancing, expert-only MXFP4/MXFP8 quantization, and benchmark results.

[2] vLLM, "Kimi K3 Is Here: Efficient Day-0 Support on vLLM," July 27, 2026. Production serving evidence, including 118 tok/s without speculative decoding and 370 tok/s with DSpark on a 16-GPU GB300-NVL72 setup, recurrent-state versus paged-KV cache management, and hardware/topology guidance.

[3] Kimi Team, "Attention Residuals," arXiv:2603.15031, March 2026. Learned attention over RMS-normalized earlier block representations; Block AttnRes matched a baseline trained with 1.25x more compute, with under 2% inference-latency overhead.

[4] Zhang et al., "Kimi Linear: An Expressive, Efficient Attention Architecture," arXiv:2510.26692, October 2025. Introduces Kimi Delta Attention. The up-to-75% KV-cache reduction and up-to-6x decode throughput are prototype results on a 48B-total/3B-active model, not measured K3 production figures. Code: MoonshotAI/Kimi-Linear.

[5] Open Compute Project, "OCP Microscaling Formats (MX) Specification v1.0," 2023. Definition of MX block formats, including the shared 8-bit block scale and the FP4 (E2M1) and FP8 (E4M3) element encodings referenced in Fig. 6.

[6] IBM, "OpenAI hacks Hugging Face in security test gone wrong," IBM Mixture of Experts, Episode 117, July 25, 2026. Panel discussion; the episode includes a Kimi K3 segment.