{"id":19940,"date":"2024-09-14T09:42:20","date_gmt":"2024-09-14T01:42:20","guid":{"rendered":"https:\/\/www.1ai.net\/?p=19940"},"modified":"2024-09-14T09:42:20","modified_gmt":"2024-09-14T01:42:20","slug":"%e5%85%83%e8%b1%a1%e5%8f%91%e5%b8%83%e4%b8%ad%e5%9b%bd%e6%9c%80%e5%a4%a7-moe-%e5%bc%80%e6%ba%90%e5%a4%a7%e6%a8%a1%e5%9e%8b%ef%bc%9a%e6%80%bb%e5%8f%82%e6%95%b0-255b%ef%bc%8c%e6%bf%80%e6%b4%bb%e5%8f%82","status":"publish","type":"post","link":"https:\/\/www.1ai.net\/en\/19940.html","title":{"rendered":"Yuanxiang Releases China's Largest MoE Open Source Large Model: 255B Total Parameters, 36B Activation Parameters"},"content":{"rendered":"

Yuanxiang<\/a> XVERSE ReleasedChina's largest MoE open source model<\/strong>\u00a0<\/strong>XVERSE-MoE-A36B<\/strong>.<\/p>\n

The model has 255B total parameters and 36B activation parameters, and the official claim is that the effect can \"roughly reach\" more than 100B large model \"cross-level\" performance leap, while the training time is reduced by 30%, and the inference performance is improved by 100%, which makes the cost per token drop dramatically. At the same time, the training time is reduced by 30%, the inference performance is improved by 100%, and the cost per token is greatly reduced.<\/p>\n

MoE (Mixture of Experts) hybrid expert modeling architecture, combining multiple segmented domain expert models into a single super-model inScale up models while keeping model performance maximized, and even reduce the computational cost of training and inference<\/strong>MoE has been used in a number of big models. Big models like Google Gemini-1.5, OpenAI's GPT-4, and Musk's xAI's Grok all use MoE.<\/p>\n

In several reviews, Meta-Elephant MoE outperforms several similar models, including Skywork-MoE, a 100 billion MoE model in China, Mixtral-8x22B, a traditional MoE dominator, and Grok-1-A86B, a 314 billion parameter MoE open-source model, and so on.<\/p>\n

\"63f3ddc3j00sjs4oo007ed000u000f1m\"<\/p>\n

Attached related links:<\/p>\n