Inkling-Small Brings Frontier MoE AI to a Single GPU

Thinking Machines Lab has just released Inkling-Small, an open-weights Mixture-of-Experts (MoE) model that challenges the assumption that frontier-scale AI requires frontier-scale hardware. With 276 billion total parameters and only 12 billion active per token, the model achieves performance that rivals—and in several benchmarks surpasses—its larger sibling, the 975-billion-parameter Inkling, while running on a single NVIDIA B300 GPU when quantized. The release signals a pivotal shift in how machine learning teams can approach deployment, customization, and cost.

What Makes Inkling-Small Different

The core innovation lies in the sparse Mixture-of-Experts architecture. Unlike dense transformers that activate every parameter for every token, Inkling-Small routes each token to just 6 of 256 available experts, plus 2 shared experts that fire on every token. This means that while the model has 276 billion parameters stored in memory, it only computes through 12 billion per forward pass. The result is a model with the knowledge capacity of a large system but the inference cost of a much smaller one.

The architecture is a 42-layer decoder-only transformer with a hybrid attention mechanism combining local and global layers. It is encoder-free and natively multimodal—processing text, images, and audio without requiring separate specialized encoders. Images are divided into 40×40-pixel patches and transformed using a four-layer hMLP, while audio is represented as dMel spectrograms. Both modalities pass through a lightweight embedding layer and are processed jointly with text tokens. The context window extends to 1 million tokens, and thinking effort is adjustable, allowing developers to trade latency for reasoning depth.

Benchmark Results: Smaller but Stronger

The most striking finding is that Inkling-Small outperforms its larger teacher model on several critical benchmarks. On Humanity’s Last Exam (text only), Inkling-Small scores 31.6%, ahead of Inkling’s 29.7%. On SWE-bench Verified—a coding agent benchmark—it reaches 80.2% versus the larger model’s 77.6%. Terminal-Bench 2.1 hits 64.7%, and Toolathlon Verified comes in at 54.4% compared to Inkling’s 45.5%. GPQA Diamond stands at 89.5%, AIME 2026 at 95.5%, and ARC-AGI-2 rises to 40.1% from Inkling’s 36.5%.

However, the model does make trade-offs. Factual recall regressed notably: SimpleQA Verified falls to 20.6% from Inkling’s 43.9%, and the AA Omniscience index drops to -9.0 from 2.1. This reflects a deliberate design choice—optimizing for reasoning and agentic capabilities over raw knowledge retention. For teams building coding agents, terminal automation tools, or document understanding pipelines, this trade-off may be well worth it.

Multimodal and Safety Performance

Multimodal scores remain competitive with the full-size Inkling at a fraction of the cost. MMMU Pro reaches 74.0%, CharXiv RQ hits 77.4% (rising to 81.3% when the model uses Python to programmatically crop, zoom, and inspect charts), and VoiceBench achieves 90.1%. Audio MC scores 54.9% and MMAU reaches 77.0%.

On the safety front, Thinking Machines Lab conducted thorough epistemic calibration training using reinforcement learning against proper scoring rules on a large corpus of real-world forecasting questions. ForecastBench without search yields a Brier Index of 61.3, ahead of Inkling’s 60.1. StrongREJECT scores 98.4%, FORTRESS adversarial reaches 71.6%, and FORTRESS benign hits 96.9%. The lab concluded that the model presents no material safety uplift beyond the existing open-weight ecosystem and recommends layering downstream moderation tools such as Llama Guard for consumer-facing deployments.

Deployment: From Frontier Lab to Single Server

This is where Inkling-Small becomes genuinely transformative. The BF16 checkpoint requires approximately 600 GB of aggregated VRAM—achievable with 4x NVIDIA B300 or 8x NVIDIA H200 GPUs. But the NVFP4 quantized checkpoint drops that floor to just 180 GB, enabling W4A4 inference on a single NVIDIA B300 GPU (requiring SM100+) or W4A16 on two H200 GPUs. Supported runtimes include SGLang, vLLM, TokenSpeed, Unsloth, and Hugging Face.

That single-GPU path moves a 276-billion-parameter model firmly out of frontier-lab territory. Consider the implications:

• Startups can self-host a frontier-class model on a single rented B300 cloud instance
• Mid-size enterprises with existing H200 capacity can serve the model without purchasing new hardware
• Regulated sectors—financial services, healthcare, insurance, telecom, and public sector—gain a private-weights option under the Apache 2.0 license
• Applicable workloads include coding agents, terminal automation, document and chart understanding, call-center analytics, voice interfaces, and meeting summarization

The Training Story Behind the Model

Inkling-Small began training after its larger counterpart, which gave the research team an opportunity to revise both the pre-training data mix and the machine learning recipe. The team post-trained an earlier preview checkpoint using on-policy distillation with the full-size Inkling serving as the teacher. From that checkpoint, they continued scaling agentic coding reinforcement learning for two additional weeks.

This approach—distilling from a larger model and then continuing reinforcement learning on agentic tasks—appears to be a key factor in how the smaller model surpasses its teacher on reasoning benchmarks. It suggests that the field is learning how to train smaller models more efficiently, not just by compressing larger ones but by refining the training pipeline itself.

Why This Matters for the Machine Learning Ecosystem

The release of Inkling-Small under Apache 2.0 marks an important moment in the democratization of frontier AI. For the past two years, the most capable open-weights models have required multi-GPU clusters that put them out of reach for most organizations. A model that matches or exceeds frontier performance on reasoning and coding benchmarks—while running on a single GPU—changes the economics of AI deployment.

It also intensifies the competitive landscape. With Alibaba’s Qwen team releasing the 2.4-trillion-parameter Qwen3.8-Max and NVIDIA releasing Molt, a PyTorch-native agentic reinforcement learning framework designed to make RL research more accessible, the pace of innovation in open AI is accelerating rapidly. Each release pushes the boundary of what is possible with open weights, and each narrows the gap between proprietary and open-source frontier models.

For machine learning practitioners, the takeaway is clear: the bottleneck is shifting from access to compute toward the quality of training recipes, data curation, and post-training strategies. Inkling-Small proves that a smaller model, trained thoughtfully, can punch well above its weight class. As the ecosystem continues to evolve, the ability to fine-tune, deploy, and serve these models efficiently on modest hardware will become the defining competitive advantage.

Key Takeaways

• Inkling-Small is a 276B total, 12B active MoE model released under Apache 2.0
• It outperforms the 975B Inkling on HLE, SWE-bench Verified, Terminal-Bench 2.1, and ARC-AGI-2
• The NVFP4 checkpoint runs on a single NVIDIA B300 at 180 GB aggregated VRAM
• Factual recall regressed (SimpleQA Verified 20.6% versus Inkling’s 43.9%), a deliberate trade-off for reasoning strength
• Native text, image, and audio input with a 1M token context window makes it versatile across modalities

---

Edited by Palawan @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous