Free for commercial use

Yuanxiang released the XVERSE-MoE-A4.2B large model, which uses a hybrid expert model architecture with an activation parameter of 4.2B and an effect comparable to a 13B model. The model is fully open source and free for commercial use. It can be used by a large number of small and medium-sized enterprises, researchers and developers to promote low-cost deployment. The model has two advantages: extreme compression and extraordinary performance. It uses sparse activation technology, and its effect exceeds that of many top models in the industry and is close to that of super large models. Yuanxiang MoE technology is self-developed and innovative, and has developed efficient fusion operators, fine-grained expert design, load balancing loss terms, etc. Finally, the architecture design corresponding to Experiment 4 was adopted...