We hear a lot about AI agents "replacing" entire teams. One benchmark took the idea seriously and tested it to the end: hand the controls of a company to AI models for 500 days, then look at what is left in the bank. The result is a healthy reminder. Most AIs sink the company, and a simple rules script, with no artificial intelligence whatsoever, beats almost every model. Here is what CEO-Bench really says, and the line it draws.
1. The rules of the game
CEO-Bench hands AI models the running of a fictional software startup called NovaMind, over 500 simulated days (arXiv, ceobench.com). Each model starts with 1 million dollars and zero customers. Each simulated week, it acts freely through 34 tools covering pricing, growth, product, reliability, support, communication and sales. The rule is brutal: if cash falls below zero, the company goes bankrupt and the game is over. The final score is the cash left.
2. The results: carnage
Source: CEO-Bench benchmark (arXiv 2606.18543, ceobench.com).
Fourteen models tested. Only Claude Fable 5, Claude Opus 4.8 and GPT-5.5 finish above the starting million on their best attempt — and only Claude Fable 5 manages it in more than one run. All the others go bankrupt or lose money. The detail that stings: a hand-written, rules-based heuristic with no AI beats almost every model evaluated.
3. What it really measures: holding a course
The nuance is essential. A large model can write an excellent strategic plan in a single answer. But running a company is not producing a plan: it means chaining 500 days of coherent decisions, correcting course, trading off the short term against the long term, and not sabotaging yourself along the way. On this test of consistency, current AI drifts, overreacts, or locks itself into strategies that simple hard-coded safeguards avoid better than it does.
4. The line between executing and deciding
This benchmark draws a clear line, and a useful one for any manager wondering "what can I hand over to AI". Today's AI is remarkable at executing well-defined tasks: writing, coding, analysing, summarising, answering. It is still weak at deciding over time: holding a trajectory, owning trade-offs, steering a living system with a memory of consequences.
The practical consequence is both reassuring and demanding. Reassuring, because the fear of an "AI CEO" replacing leadership has, for now, no measured basis. Demanding, because it is a reminder that the value of AI is captured by putting it in the right place: under human supervision, on bounded decisions, where it executes fast and well — not at the helm.
It is the same lesson as the study on code productivity or Ford's U-turn: AI augments humans, it does not replace them at the controls.
Frequently asked questions
What is the CEO-Bench benchmark?
A test bench that hands AIs the running of a fictional startup, NovaMind, for 500 simulated days: 1 million dollars to start, 34 decision tools, bankruptcy if cash falls below zero, and a final score equal to the cash left.
Which models succeeded?
Out of 14 models, only Claude Fable 5, Claude Opus 4.8 and GPT-5.5 finish above their stake on their best attempt, and only Claude Fable 5 across several runs. Most go bankrupt, and a rules script with no AI beats almost every model.
What should a company conclude from this?
That AI excels at executing, but cannot yet hold a strategic course. It is a powerful execution tool under human supervision, not an autonomous decision-maker. Its right place is on well-defined decisions, not at the head of an organisation.
Conclusion
CEO-Bench does not say that AI is useless, quite the opposite: the best models sometimes manage to win. It says that running a company calls for a skill — consistency over time — that current models have not mastered, to the point where a handful of simple rules often does better.
For a company, this is a compass. The question is not "can AI run us", but "which precisely defined decisions can we delegate to it while keeping hold of the course". That is where, and only where, the value lies today.