hybrid expert model

June 30, Huawei today officially announced the open source of the Pangu 7 billion-parameter dense model, the Pangu Pro MoE 72 billion-parameter hybrid expert model, and the Rise-based model inference technology. Huawei said, "This move is another key initiative for Huawei to practice the Rise ecological strategy, promote the research and innovative development of large model technology, and accelerate the application and value creation of AI in thousands of industries." The Pangu Pro MoE 72B model weights and base inference code have been officially launched on the open source platform. The ultra-large-scale MoE model inference code based on the Rise...

A few days ago, Huawei's Pangu team officially announced Pangu Pro MoE, a grouped hybrid expert model native to Rise. According to the Pangu team: The Mixed Model of Expertise (MoE) is emerging in Large Language Models (LLMs) as an architecture that can support larger scale parameters at lower computational cost, resulting in greater expressive power. This advantage stems from the design feature of its sparse activation mechanism, i.e., each input token only needs to activate a portion of the parameters to complete the computation. However, in real-world deployments, there is a serious imbalance in the activation frequency of different experts, with a portion of experts being overcalled...

February 24, 2011 - Kimi, Dark Side of the Moon, yesterday released a new technical paper "Muon Scalable for LLM Training" and announced the launch of "Moonlight": a 3 billion / 16 billion parameter hybrid expert model (MoE) trained on Muon. Mixed Expert Model (MoE) trained on Muon. Using 5.7 trillion tokens, it achieves better performance at lower floating point operations (FLOPs), thus improving the Pareto efficiency bound. Dark Side of the Moon says the team found that the Muon optimizer can be carefully tuned by adding weight decay,...