News on August 20,Zhipu AI CHIEF SCIENTIST TANG JIE POST STATED THAT THE GLM-53 WOULD BE SCALED UP THROUGH EXPANDED TRAININGModelCAPABILITY, RATHER THAN SIMPLY INCREASING THE NUMBER OF PARAMETERS. THE MODEL USES THE SAME BASE MODEL, STRUCTURE, TOTAL PARAMETER AND ACTIVATED PARAMETER LEVELS AS GM-5.2, BUT AFTER ONE MONTH OF LONG-CYCLE ENVIRONMENTAL TRAINING AND INTENSIVE LEARNING, THE MODEL PERFORMANCE HAS IMPROVED SIGNIFICANTLY。
Tang Jie recalled in his post that early studies had concluded that models should expand the parameters at a faster pace, but that follow-up studies had shown that, in the case of limited computing resources, a more balanced ratio between training data and model parameters was needed. As models are used on a large scale, the proportion of reasoning costs in the total cost of the model ' s life cycle is increasing, and smaller-trained models with more data are a viable direction。
He also mentioned that the thin mix expert model needed to distinguish between total and activated parameters: While the former broadly determine the extent of knowledge that the model can accommodate, the latter and the depth of effective reasoning affect the model ' s ability to complete long-chain reasoning. In the case of software loopholes, this is not just a search of known loopholes, but a requirement for the model to continue to complete multistep reasoning。
IN TANG JIE ' S VIEW, WHEN THE TOTAL NUMBER OF MODEL PARAMETERS REACHES A THRESHOLD OF SUFFICIENT TO CARRY A WIDE RANGE OF KNOWLEDGE, FURTHER ENHANCEMENT OF CAPACITY CAN SHIFT TO EFFECTIVE REASONING DEPTH AND POST-TRAINING. THIS GM-5.3 IS A CONTROLLED EXPERIMENT TO THIS JUDGEMENT, BUT HE ALSO INDICATES THAT THERE IS ROOM FOR EXPANSION OF THE BASE MODEL SCALE, PRE-TRAINED DATA VOLUME AND SINGLE-WAY CALCULATIONS, AND THAT FUTURE WORK MAY CONTINUE IN THE DIRECTION OF ONGOING TRAINING AND PRE-TRAINING。