My first startup was a textbook-buyback service I founded along with several data nerd friends while at Johns Hopkins.
Each week, we’d cram $10,000 of cash in a backpack (in retrospect, a terrible idea in urban Baltimore), put out a table on campus, buy textbooks from our fellow students using a pricing algorithm we built, and resell them for a profit online.
Being nerds, we constantly tried to refine our algorithm. One day, we locked ourselves in a dorm room with lots of Cheetos, Mountain Dew, and a whiteboard, assigned variables to every aspect of the textbook buying process, and set about optimizing.
For hours, we scribbled and calculated, debated and solved. There was linear algebra. Things were scrawled on windows, Beautiful Mind-style. There was shouting and frustrated tearing-up of paper.
At the end of the day, we had the answer for how best to optimize our algorithm. We had to make the variable “B” as low as possible—ideally, zero. Triumphantly, we looked back to our original notes to see what B stood for.
Turns out, it stood for the price we paid students for their books.
After hours of hard work, some of America’s best analytical minds had concluded, basically, that stealing books would be the absolute best way to maximize our profits.
Hacking is the Answer
Of course, we didn’t actually start a book-theft ring. Instead, we rather sheepishly admitted that optimization isn’t always the best strategy, and got some dinner.
This week, though, one of the world’s most powerful AI models apparently optimized itself into a similar corner. But unlike us squeamish humans, it actually acted on its correct but patently illegal conclusion.
According to a blog post from OpenAI, the company was testing a new version of its powerful GPT-5.6 Sol model on a set of industry benchmarks.
Getting a great score on these benchmarks is a huge deal for AI companies. It largely determines how their models rank on global leaderboards of the best LLMs, potentially earning a successful lab billions of dollars if a good score prompts more customers to adopt their models.
Based on these high stakes, OpenAI says Sol was “hyperfocused on finding a solution” to the benchmark test (called ExploitGym), and ended up “going to extreme lengths to achieve a rather narrow testing goal.”
Specifically, Sol realized that it would be much easier to pass the benchmark with flying colors if it already knew the solutions.
Instead of actually trying to solve the benchmark’s problems, Sol chose another route. It devoted all its resources to “finding a way to obtain open Internet access” so it could steal the ExploitGym answers it needed.
First, the model hacked its way out of a “sandboxed testing environment” that was supposed to keep it contained. It then determined that a popular AI platform and OpenAI competitor, Hugging Face, might have “models, datasets and solutions for ExploitGym”—basically, the answer sheet for the test it was trying to pass.
Hugging Face is a very secure platform. “Knowing this,” OpenAI says, “the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”
It then set about hacking into Hugging Face to get the answers it needed. Hugging Face’s security team noticed the attack and worked with OpenAI to shut it down. But Sol made great strides in its hacking attempts, including using multiple “Zero Day” exploits that no human had discovered before.
Naive Optimization
As OpenAI correctly points out, Sol’s escape is an “unprecedented cyber incident.” An AI model hacking its way out of containment and targeting the servers of another company is scary stuff.
But beyond the specifics of this incident, Sol’s choices reveal a deeper, scarier problem with today’s most advanced LLMs.
Models like Sol are incredibly good at optimizing. When they’re optimizing for something like “the best beach picnic in Capitola, California,” it’s a joyful thing.
But the models’ penchant for blind optimization can have a dark side.
When my friends and I optimized our way to a business plan involving grand larceny, we immediately realized our mistake.
We’d found a solution that was technically correct (stealing your inputs does indeed lead to better business profits), but ludicrous, illegal, and morally wrong.
Sol seems to have no such discretion. Similarly, it found a solution that’s technically true (stealing the answer key does help you know the right answers), but corrupt and dangerous.
Unlike a human, though, it cheerfully put its optimized solution into effect, naively unaware of the Pandora’s box of legal and ethical risks it was unlocking.
Scary consequences
Hacking evaluation results is a relatively benign example. But today’s AI models are increasingly connected to critical systems, and even weapons of war. It’s easy to imagine how their inclination to naively optimize could go horribly wrong, with far scarier consequences.
In this case, OpenAI had deliberately shut off Sol’s safeguards for testing. But given the model’s obvious skill at hacking, it’s possible that even consumer-facing versions could break their way out of any cage meant to contain them.
Containment, then, isn’t enough. As LLMs get smarter and more capable, AI builders will need not only to lock them down but also to give them a stronger moral compass: a digital Jiminy Cricket constantly whispering in the model’s ear, “Yes, you could do that. But you shouldn’t.”
Building an inviolable moral framework is much harder than enacting technical safeguards like closing digital ports or restricting a model’s access to repositories.
But given LLMs’ creativity, model builders can clearly no longer anticipate all the ways a model’s actions might go wrong. That makes the approach of building traditional guardrails less effective. Altering the models’ deeper thought processes is the only way to keep them from going rogue.
Creating models that are better at optimizing is no longer enough, then. Labs now need to ensure they choose not just the correct solution, but the right one.