K2 Horizon Redefines Open Source Machine Learning With Six Model Fleet

The Dawn of Fully Open Model Fleets

The machine learning landscape has long grappled with a fundamental tension: the most powerful models are locked behind closed doors, while open-weight alternatives offer transparency but rarely match frontier performance. The Institute of Foundation Models (IFM) has disrupted this status quo with K2 Horizon, a connected fleet of six language models ranging from a 0.9 billion parameter dense model designed for smartwatches to a 375 billion parameter mixture-of-experts flagship built for enterprise reasoning. Released under the Apache 2.0 license, K2 Horizon represents the largest fully open-source model launch in AI history, shipping not just weights but training data, intermediate checkpoints, code, and complete methodologies.

Architecture Spanning Six Scales

The K2 Horizon fleet comprises four dense models and two sparse mixture-of-experts (MoE) architectures. The dense models include a 0.9B parameter model for highly constrained edge devices, a 3.7B model for smartphones, a 7B model for on-device applications, and a 32B model for local workstations. The sparse models consist of a 36B-A4B MoE model and the flagship 375B-A23B MoE model, where the notation indicates total parameters and active parameters per token respectively.

All models share a core Transformer backbone with grouped-query attention, rotary position embeddings, and SwiGLU feed-forward layers. The 0.9B model uses a reduced vocabulary of 64,256 tokens to conserve memory on device, while all other models share a 250,624-token vocabulary and a 512K-token context window. Four of the six models are trained on a shared 22T-token pre-training corpus, enabling controlled studies of scale and sparsity across the fleet.

Mixture-of-Value Attention: A Reasoning Breakthrough

The 36B-A4B model introduces Mixture-of-Value Attention (MoVA), a novel architectural innovation that applies MoE principles to attention value projections. Rather than routing only feed-forward network experts as in traditional MoE designs, MoVA routes value projections in the attention mechanism. Each token selects 4 of 64 value experts without incurring additional KV-cache cost.

The result is striking: the 36B MoVA model, with approximately 4 billion active parameters per token, nearly matches the performance of the dense 32B model trained on the same data. This means comparable reasoning capability with roughly one-seventh the active parameters. Combined with softplus attention gating, a more flexible and theoretically motivated alternative to standard gating mechanisms, the architecture delivers exceptional capability per active parameter.

Accelerating Inference With Uno Diffusion Adapters

Training large models is only half the challenge; serving them efficiently is equally critical. K2 Horizon introduces Uno, lossless diffusion adapters that accelerate decoding by up to 2.2x without requiring a separate draft model. Unlike speculative decoding approaches that need a smaller companion model, Uno generates blocks of tokens in parallel through a diffusion distillation technique, achieving roughly 3x speedup without degrading response quality.

This innovation has direct implications for deployment economics. Faster inference means lower serving costs, reduced latency for end users, and the ability to run capable models on less expensive hardware. For organizations building agentic applications where models must respond in real time, this speedup can be the difference between a viable product and one that stalls under its own computational weight.

Agentic Post-Training and the RL360 Stack

What sets K2 Horizon apart from previous open-weight releases is the completeness of its agentic post-training pipeline. The fleet is trained for multi-step task execution, tool calling, web search, terminal operation, and file manipulation across PDF, spreadsheet, and presentation formats. The release includes the RL360 reinforcement learning stack, which exposes the full reinforcement learning training process alongside supervised fine-tuning and final token preference optimization stages.

By releasing intermediate checkpoints from every training stage, IFM enables researchers to study how reasoning, tool use, planning, and agentic capabilities emerge during training. This transparency allows others to reproduce the methods that create these capabilities and adapt them to new tools, environments, and domains. The xLLM training framework, used for both dense and MoE training at production scale, is also released as open source.

Performance Across Benchmarks

The 0.9B, 3.7B, and 7B models set new state of the art in their respective size classes across widely used evaluations. The 32B and 375B-A23B models rank among the top models in their classes. On agentic and coding benchmarks, K2 Horizon leads or remains competitive with strong open models at every scale.

The fleet supports deployment on NVIDIA, AMD, and Cerebras hardware with day-zero support from vLLM, SGLang, and Ollama. This cross-platform compatibility ensures that developers can deploy K2 Horizon models regardless of their existing infrastructure, whether running inference locally on a workstation or serving at scale through cloud providers.

Synthetic Data at Pre-Training Scale

One of the most technically significant aspects of K2 Horizon is its use of synthetic data. The training pipelines produced approximately 10 trillion synthetic pre-training tokens, demonstrating that synthetic data can serve as a major ingredient in pre-training at scale rather than a supplementary technique. Four models share the same 22T-token pre-training corpus and recipe, which includes both synthetic and natural data.

This finding challenges the conventional wisdom that high-quality natural data is the primary bottleneck for scaling language models. If synthetic data generation pipelines can reliably produce training-grade tokens at scale, the path to training capable models becomes more accessible to organizations that lack massive natural data collections. The full data construction recipes and pipeline documentation are released alongside the models.

Implications for the Machine Learning Ecosystem

The release of K2 Horizon has several far-reaching implications for practitioners and organizations working in machine learning:

  • Reproducibility — With complete training histories, intermediate checkpoints, and code, researchers can verify every claim and build upon proven methods rather than black-box results.
  • Deployment flexibility — Six models spanning edge to enterprise mean organizations can prototype on small models and scale to the flagship without changing APIs or tooling.
  • Cost efficiency — The MoVA architecture and Uno acceleration adapters dramatically reduce the compute required for both training and inference at competitive quality levels.
  • Agentic readiness — Models trained with tool use, planning, and multi-step execution capabilities are immediately deployable in agent frameworks without additional fine-tuning.
  • Open science — By releasing data pipelines and training logs, IFM establishes a new standard for transparency that goes beyond the open weights debate dominating industry headlines.

Dynamic Model Routing

IFM also introduces a dynamic model routing technique that directs tasks to the most cost-effective model within the fleet. This approach allows developers to build applications that automatically select the appropriate model size based on task complexity, optimizing for both performance and cost. A simple query might route to the 0.9B model, while a complex coding task would route to the 375B flagship.

This routing capability transforms the fleet from a collection of individual models into a unified system where developers can specify quality and latency requirements rather than choosing a specific model. The implications for production deployments are significant, as workloads can be served at the lowest cost that meets the required quality threshold.

The Path Forward

K2 Horizon succeeds K2-V2 in the LLM360 family and extends the fully open approach from a single base model to the complete development lifecycle of six models. The release includes pre-training data, training logs, evaluation results, fine-grained configurations, and the production-scale xLLM framework. All artifacts are available through the IFM repository.

For the machine learning community, K2 Horizon represents a pivotal moment. It demonstrates that fully open model development, from data to deployment, is feasible at frontier scale. As organizations evaluate their AI strategies, the ability to inspect, reproduce, and adapt every component of a model fleet provides a level of control and understanding that closed alternatives cannot match. The question is no longer whether open source can compete with closed models, but how quickly the ecosystem will leverage this transparency to build the next generation of intelligent systems.


Edited by Palawan @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous


Discover more from QUE.com

Subscribe to get the latest posts sent to your email.

Leave a Reply

Connect with

Discover more from QUE.com

Subscribe now to keep reading and get access to the full archive.

Continue reading