Daily papers

Tuesday, 17 November 2026 · 7 min read

K2-Horizon fleet offers open-source LLMs with sparse attention and efficient training

K2-Horizon presents six open language models from 0.9B to 375B parameters, introducing Mixture-of-Value Attention and softplus gating to improve reasoning and efficiency, with full training data, code, and checkpoints released for reproducibility.

Based on the paper K2-Horizon: A Frontier Fleet of Open Source Language Models by K2 Team, Zhengzhong Liu, Haonan Li, Xuezhe Ma and 91 others

Challenges in building open, scalable, and capable language models

Large language models (LLMs) have advanced general-purpose reasoning and agentic tasks across domains like math, science, and software engineering. However, many strong open-weight models lack transparency in their training data, intermediate checkpoints, and development decisions, limiting reproducibility and controlled study.

Scaling LLMs to very large sizes with sparse architectures like mixture-of-experts (MoE) promises efficiency but introduces challenges in training stability, parameter utilization, and inference complexity. Extending context length and improving reasoning also require careful architectural and training design.

K2-Horizon aims to provide a fully open fleet of LLMs spanning from 0.9B to 375B parameters, with innovations in sparse attention and gating mechanisms, alongside complete release of data, code, and checkpoints to enable reproducible research and deployment.

Sign in to read the whole paper

You have read the opening. The rest is free with a Bezen account, and no card is needed.

In the rest of this paper

  • How the method works, step by step, with a flow chart
  • Three key formulas from the paper, explained symbol by symbol
  • Two figures from the paper, and how to read them
  • How it was tested, what was found, and where it falls short
K2-Horizon fleet offers open-source LLMs with sparse attention and efficient training · Bezen