April 30th.MilletLarge ModelThe team has passed the "Xiaomi MiMo"Public announced that today, XiaomiOpen SourceXiaomi MiMo, the first large model "born for reasoning", links pre-training to post-training to comprehensively improve reasoning ability. According to the introduction, MiMo is the initial attempt from the newly established "Xiaomi Big Model Core Team".
On the Mathematical Reasoning (AIME 24-25) and Code Competition (LiveCodeBench v5) public evaluation sets, MiMo was able to achieve the same results using only the Parameter scale for 7BThe model is a closed-source inference model that outperforms OpenAI's closed-source model. o1-mini and Ali Qwen larger scale open source inference models QwQ-32B-Preview.
Officials said,The improvement in MiMo inference capability is driven by a combination of innovations at multiple levels of data and algorithms in the pre-training and post-training phases, including:
- pre-training: the core is for the model to have seen more inference patterns
- Data: focus on mining rich inference corpus and synthesize about 200B tokens of inference data.
- Training: Three phases of training were conducted, progressively increasing the difficulty of the training, totaling 25T tokens.
- after-training: At the core are efficient and stable reinforcement learning algorithms and frameworks
- ALGORITHM: Test Difficulty Driven Reward is proposed to alleviate the reward sparsity problem in difficult algorithmic problems, and Easy Data Re-Sampling strategy is introduced to stabilize RL training.
- Framework: Designed a Seamless Rollout system that accelerates RL training by 2.29x and verification by 1.96x.
1AI Attached open source address:
-
Hugging Face:https://huggingface.co/XiaomiMiMo
-
Technical report:https://github.com/XiaomiMiMo/MiMo/blob/main/MiMo-7B-Technical-Report.pdf