How a mathematician used ChatGPT to solve a 42-year-old problem

The strengths and weaknesses of using a chatbot in trying to come up with novel research, as experienced by Ernest Ryu, assistant professor at UCLA

Ernest Ryu had previously failed to solve a long-standing problem in math with the help of ChatGPT, but now, two years later, he wanted to give it another shot.

A couple of months earlier, OpenAI had released the GPT-5 model, including a Pro version with “extended reasoning for even more comprehensive and accurate answers”, and so after having put his son to bed one evening, the assistant professor at UCLA picked up the challenge again.

The problem within the branch of math called optimization was related to a cornerstone method in the field that has been used for a variety of things: creating images from CT and MR scans, making blurred images sharper more quickly, as well as in training AI systems.

The crux was a gap in the theoretical foundation behind the method called Nesterov’s Accelerated Gradient (NAG), named after Russian mathematician Yurii Nesterov who came up with it in 1983: The method yielded good results, but it wasn’t known whether it actually settled at a solution or was just bouncing gently around the right answer.

Ryu wanted to find out once and for all.

ChatGPT as co-author

Initially he had a hunch that the method would not be stable, so he asked ChatGPT for evidence in that direction, but to make such a proof it’s most effective to also attempt proving the opposite and see which path seems more surprising, Ryu explains.

“As I did that, I realized, ‘Oh, maybe it is stable’. So I tried to push in that direction and arrived at the,” he tells excitech.

The discovery is described in a new preprint: “Point Convergence of Nesterov’s Accelerated Gradient Method: An AI-Assisted Proof” by Ryu and Uijeong Jang, a PhD student at UCLA.

Like a maze search

Ernest Ryu compares finding a mathematical proof to doing a maze search: you start somewhere, don’t know where the destination is, and begin searching.

Before systems like ChatGPT, the researcher would do the search one step at a time. It’s important to have an intuition as to whether one is making progress, he explains, because just randomly wandering around is too inefficient.

Interacting with ChatGPT presented two important benefits. First, whenever he wanted to explore a particular direction, it would be very helpful in speeding up exploration and especially in showing if a particular approach could not work.

“If I can rule this approach out in five minutes with the help of ChatGPT, then I’ve saved two hours of my time,” Ryu says.

The second benefit came from the models’ suggestions for new approaches that he had not thought of.

“Very often it would make me think, ‘Huh, I haven’t thought about that approach’. Let me explore that.”

“My heart rate increased”

The process took place over three days. After having the sense that he made progress for the first two, it was on the third that he arrived at the solution.

Ryu estimates that around 80% of ChatGPT’s arguments turned out to be wrong, and at first glance, what turned out to be the solution looked like one of the false positives, so he took his time to double-check it and showed it Jang.

”I was reading it, thinking ’Let me find the error here’, and I just couldn’t find an error. At the end of that reading, my heart rate increased and it was it.”

When the correct answer was provided, all the previous attempts were different and mostly unrelated. It was just in the last step that Ryu prompted the model in a way to roughly explore an area, and that was enough of a hint for the model.

“I was very surprised,” he says.

From the key discovery, there was still some work though to finish the proof. He could have done that himself in about five hours, but asked ChatGPT to do it, which completed it quickly.

“Using the maze analogy: if a particular alley is correct, then once the model takes that alley, it’s very effective in filling in the relatively straightforward details.”

Ernest Ryu shared the discovery on X, where it has amassed 1.2 million views at the time of writing.

Crossed the threshold

Ryu elaborates that ChatGPT is particularly excellent at combining knowledge from different areas.

If a result is easy for a mathematician in a different field, ChatGPT can often replicate that very quickly. That’s really helpful because you’d first have to know that it exists and then read up on that paper to understand the precise semantics of it, which takes time.

“It’s this higher level search engine of mathematics that is extremely effective.”

Ryu acknowledges that the hallucination rate is a downside of using the model, and if the accuracy is too low, then you’d rather work without it. For him there’s a threshold and beyond that the signal-to-noise ratio is such that talking to the model is helpful.

“Certainly in my area, ChatGPT has crossed that threshold and does accelerate my research.”

To underscore the importance of the contribution by ChatGPT, Ryu originally wanted to credit it as a co-author.

“If ChatGPT were a human, I would let it be the lead author,” Ryu said.

But he also doesn’t list Microsoft Word as a co-author, even though he uses that program to write in, he ponders.

“So I thought, okay, I’ll do this in the classical way. I’m going to disclose and make it a central point that this was discovered using ChatGPT.”

Watershed moment

In a follow-up post on X, Ryu says he believes that we are at a watershed moment in the history of mathematics. He draws a parallel to chess, quoting ex-world champion Magnus Carlsen on when some players in the old board game started to use AI, and it was clear that they got an advantage over those who didn’t.

“It just made us understand the game a lot better,” Carlsen said in an episode of the Joe Rogan podcast earlier this year.

Ryu points out that he is not the only one using AI to enhance math research. Adil Salim from Microsoft Research and Timothy Gowers, winner of the prestigious Fields Medal (described as ‘The Nobel Prize of mathematics”) are two other examples.

Recently, the startup Harmonic also celebrated that their AI model Aristotle has made progress on a series of long-standing problems.

The limitations

Still, Ernest Ryu doesn’t think that AI is yet capable of making fundamentally new math discoveries because the most novel and groundbreaking results come from defining new mathematics, he says, comparing it to when Isaac Newton in the 17th century invented calculus to solve problems within physics.

So AI systems cannot replace human mathematicians (for now), he concludes.

“But at least in the kind of research that I do, it does accelerate the work quite significantly, maybe 3x, 5x, or 10x.”