I have spent my whole career trying to understand intelligence. I studied neuroscience because I was interested in AI and figured learning how the brain works first was a good way to begin. Intelligence first before “artificial” intelligence. That training left me with a habit. When a new capability or technology arrives, my first question is practical. What can this really do, and for what type of real-world problems?
As an AI guy, I rarely cheer on AI headlines. I look past the noise and try to understand not just the breakthroughs but also limits of the new capabilities.
Back in 2020, when the safety conversation was running hot on hype, I had some plain advice. Treat AI as a tool, a powerful one, but still a tool. Stick to the boring use cases. Work at the task level where the technology was reliable enough to help. My optimistic scenario was one of boring but useful tools.
When the agentic wave started building a couple of years later, I brought the same lens, and I came away unconvinced. An agentic system is only as strong as the tasks underneath it, and I did not yet see the task-level reliability required that would let these systems be effective.
However, the hype was growing. By the time I wrote from RSA early last year, the gap between the talk and the substance was still hard to ignore. There wasn’t even a shared understanding of what the word “agentic” meant, every person had a different definition and perspective. And I had a worry about something structural. Take ten agents, each about ninety-five percent accurate. Chain them so the output of one becomes the input of the next. By the end of that chain you are down to about sixty percent. Reliability does not survive being stacked. I was starting to think about systems of agents by then, though my concern was still whether they could be trusted to work to make a meaningful difference.
Then, the rapid improvements in coding agents changed my mind about the clock.
Within months of that article, the incredible pace of progress made one thing clear to me. Reliability was a matter of time for several real-world tasks. Better models, with better engineering harnesses built around them, were going to close the gap I had been worried about. The ceiling I was worried about was going to lift.
That is when the real problem came into focus, and it was not the one I had been watching. Once you can trust the individual agents, you don’t stop deploying after the first one. You deploy many. Fleets of them, across every function, each one capable, each one doing its job. And a new question appears that has nothing to do with reliability. If every agent in the fleet does what it was built to do, does the fleet still add up to what the organization intended?
That question was the seed of the book. It sent me straight back to where I started.
A body with capable limbs cannot move well without proprioception, the constant inner sense of where all its parts are and what they are doing. The cerebellum does more than react to that feedback. It predicts. When the brain issues a movement, it forms an expectation of the sensation that movement should produce, then checks the expectation against what actually shows up. The gap between the two is the signal that corrects what comes next. Skilled movement is a loop of predicting the result and correcting for the difference.
An organization running fleets of agents needs the same sense of itself. It has to know, continuously, where its systems are and how far they have drifted from what was intended. Without that inner awareness, capable parts do not combine into coordinated action. This is not a sensing mechanism you add to a body later to make it safer. You need it for the body to move at all.
There is a well studied case of a man named Ian Waterman who lost this sense in most of his body. He learned to move again but only by watching himself, steering every step and reach with his eyes. It works. It is also exhausting and fragile. Turn off the lights and he cannot coordinate at all. An organization that governs its agents through manual audits and periodic reviews is in his position. It compensates with constant effort for a sense it never built in, and that compensation costs more than what it replaces. The whole system collapses the moment conditions change.
There is a stranger failure worth naming too, written about in a fascinating book by neuroscientist V S Ramachandran. When a limb is amputated, the brain does not fall silent. It keeps generating signals for the limb that is gone, and the person feels it vividly. Governance can fail the same way. Strip the real judgment out of an oversight function through restructuring or neglect, and the function does not go quiet. Reviews still get completed and metrics still get produced. The organization keeps feeling the sensation of oversight while the judgment behind it has been hollowed out. Here my 2020 advice comes back, about transparency. There is no oversight without transparency, and there is no transparency in a system that only produces the appearance of being watched.
Intelligence is getting cheap. Soon it will sit in every workflow and every tool. The scarce resource in that world is coherence, the living link between what each system does on its own and what the enterprise is trying to do. Coherence is the connective tissue that keeps distributed intelligence pointed at one purpose.
I spent years studying how a brain keeps its many parts working as one. The agentic era turns out to ask the enterprise the same question. My two worlds met, and that meeting is the book, Coherence, arriving this Fall. The goal has not changed since 2020. Boring but useful, still. Only now the boring and useful thing to build is coherence itself. To follow the thinking as it develops, join the list at coherise.com. The one-page decision tool from the book is the first thing I send.
Comments
One response to “I studied the Brain to Build AI. The Agentic Era Sent Me Back to It.”
[…] wrote recently about how studying the brain led me to the question at the center of my book. The short version: […]