Epoch AI, a research organization, announced FrontierMath, a new AI model math benchmark test set for assessing model mathematical reasoning. It is different from the existing test set in that it contains more complex math problems that involve multiple areas of modern mathematics and are extremely difficult and time-consuming for human experts to answer. Its questions are designed by senior AI experts and require the AI to understand the concepts and have the ability to reason in complex situations to prevent comparing answers. After using the test set on the market AI model initial test, found that models including Claude 3.5 and GPT-4, etc. generally poor performance, the success rate of solving the problem is lower than 2%. The research team pointed out that the main difficulty of AI to solve advanced mathematical problems is to rely on similar questions in the training data to generate the answer instead of truly understanding the logical structure of the reasoning problem, and this problem can not be solved by increasing the size of the model, the need to start from the reasoning of the answer to the question, but also by increasing the size of the model. This problem cannot be solved by increasing the model size, and needs to be deeply transformed from the level of reasoning architecture.
Official website address:
https://epoch.ai/frontiermath