July 5 News.appleThe company quietly released on Hugging Face a DiffuCode-7B-cpGRPO calledOpen Source AI ModelsThe model has innovative features in terms of generating code.Ability to generate code out of order with performance comparable to top open source coding models.
Note: Traditional Large Language Modeling (LLM) generates code in a left-to-right, top-to-bottom order, as most humans read text.
This is mainly because these LLMs work using Autoregression, meaning that when a user asks them a question, they process the entire question, predict the first token of the answer, then reprocess the entire question with that token, predict the second token, and so on.
The LLM also has a setting called Temperature that controls the randomness of the output. After predicting the next token, the model assigns probabilities to all possible options. Lower temperatures mean that the most likely token is more likely to be chosen, while higher temperatures give the model more freedom to choose less likely tokens.
The other option is the Diffusion model, which is commonly used for image modeling. In short, the model starts with a blurry, noisy image and iteratively removes the noise, while taking into account the user's needs and gradually directing it to an image closer to the user's request.
The model released by Apple is called DiffuCode-7B-cpGRPO, and it's based on a paper published last month called DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation.
The paper describes a code generation model using a diffusion-first strategy, but with a special feature: when the sampling temperature is increased from the default 0.2 to 1.2, the DiffuCoder becomes more flexible in the order in which it generates the tokens, thus breaking away from the strict left-to-right constraint.
More interestingly, Apple built this model on Ali's open-source Qwen2.5-7B model, transformed that model into a diffusion-based decoder as described in the DiffuCoder paper, and then tweaked it to better follow instructions. Once that was done, they trained another version of it with over 20,000 carefully selected coding examples.
In the mainstream programming run, DiffuCode-7B-cpGRPO maintains a test score improvement of 4.4% compared to the mainstream diffusion-based programming model in the case of generating code without strictly relying on left-to-right generation.