30 AI search strategies produced no universal winner: compute should go to the early leaders
30 AI search strategies produced no universal winner: compute should go to the early leaders
On July 20, Akshat Gupta and colleagues posted an arXiv preprint on using AI to search for mathematical solutions. They compared 30 strategies across 12 model and task combinations, completing more than 3,1 million runs. Selection worked best: several strategies were allowed to begin, then the strategies that started scoring earlier received the remaining compute.
Automated search works as follows: a language model proposes a program or a modification to one, an evaluator tests the result, and the system selects the next candidate. The search framework defines the rules for this selection: which previously successful program to build on, how many attempts to allocate to a new direction, and how many to a direction that has already succeeded.
In the analysis of five results in Banach space theory, a model also proposed possible steps, but mathematicians checked the proofs. Here, evaluation is automatic: the program runs and receives a score. The authors of the preprint therefore compared not only the programs, but also the rules that the search used to select the next program.
They separated OpenEvolve and the search component of TTT-Discover into individual design choices, assembled 30 combinations, and gave each combination the same number of runs. The benchmark included circle packing, the Heilbronn triangle problem, and the second autocorrelation inequality. Four models were used, ranging from 3 to 120 billion parameters.
No fixed strategy outperformed simple sequential search across all 12 model and task combinations. Full OpenEvolve configurations often performed worse than simpler strategies. Rules that worked well for one model on one task did not provide a general solution.
The authors tested a different approach: they did not choose a strategy in advance. At the midpoint of a run, the best intermediate score was associated with the final result in 11 of the 12 combinations. They ran several strategies for part of the available budget, compared their intermediate scores, and allocated the remaining compute to the leaders. In the best sequence, 12 initial runs were reduced after three evaluations, first to 5, then to 2, and finally to 1. With a budget equivalent to five full runs, the average score reached 85,75%; distributing the same budget equally among the strategies produced 84,54%.
Here, AI searches not only for a solution, but also for an effective way to search for that solution. When the evaluator can quickly show which search process is making progress, the operator can first examine the early performance of several strategies and then direct the remaining compute toward those whose results are already improving.