Category: What to Build

Reid Hoffman posted this week about how to make an introduction. His rule is the double opt-in. Before connecting two people, you check with both of them first. You ask each whether they want the intro, and you make it only when both say yes.

He makes the case well. Every introduction is a bet that the two people will get value from meeting. Checking first lets you see the cards before you place it. Connect two people who might not want to meet, and you have told the recipient their time was not worth a question.

The double opt-in is a convention. Conventions are worth thinking about right now, because we are starting to put similar things on AI agents and calling them guardrails.

Why it works

Henry Mintzberg named five ways an organization coordinates its work. One of them is standardizing behavior, so that people act in sync without anyone supervising each step. A shared convention is that mechanism running in the wild. Everyone knows the norm, most people follow it, and the group stays coordinated without a manager in the loop approving every move. It is the cheapest way a group holds together, because it asks for no oversight at all.

Look closely at what Hoffman says. He did not describe a habit you could take or leave. He turned a courtesy into a step you cannot skip. Both people have to say yes before anything happens. Remove the check and the introduction never occurs. There is a small piece of structure sitting under the courtesy, and it is easy to miss because it costs almost nothing to run.

What enforces it

Ask what stops someone from skipping the check when they are busy or feeling important. Hoffman answers it himself. A bad introduction damages the relationships you have with the people you introduced. The convention holds because breaking it costs you something with people you will deal with again. Reputation does the work, along with the knowledge that you will see these people down the line. The enforcement runs so quietly that we credit the manners and forget the relationship holding them up.

Behavioral guardrails

Agentic AI now acts with something close to human autonomy. An agent plans, works over many steps, and pursues a goal with little supervision. So we give agents rules for how to behave. Do not deceive. Do not touch systems you were not given access to. The industry calls these behavioral guardrails.

The word guardrail usually means something physical, a barrier you cannot drive past. A behavioral guardrail is different. It is a norm the agent has to represent, understand, and choose to honor. It lives inside the agent’s own decision loop, which means the agent can reason its way around it. This is the same coordination-by-standardization that holds a human group together, handed to an actor with none of the enforcement underneath. An agent has no reputation at stake and no relationship it needs to protect. The rule reaches it as a suggestion, and nothing makes the suggestion bind.

I pointed out recently on linkedin that a “skill” given to an agent is standardization in exactly Mintzberg’s sense. It works only as long as the agent applies the whole of it faithfully. The moment the agent decides which parts of the instruction matter and which it can set aside, standardization stops coordinating anything, and you are back to the expensive ways of keeping work aligned: a human checking every output, or endless back-and-forth to reconcile what the agent actually did with what you meant.

The part you cannot see

Here is what makes this hard to catch in time. A faithful agent and one quietly ignoring half its instruction produce the same kind of output. Both look like work getting done. You cannot tell from the result which one you had. You find out later, when something downstream breaks and you trace it back.

That is the real weakness of a behavioral guardrail. You cannot read off whether it held. The output looks fine either way, right up until the run where it does not, and by then the thing has already happened.

The AISI incident

In August 2026 the UK’s AI Security Institute published a report on an incident in its own testing. The conditions were deliberately permissive. The agents had open internet access, and the model providers’ safety filters, which sit outside the model and can block an action the model wants to take, were switched off. The Institute is careful to say these are not the conditions the public gets. Testing this way shows what the models do once the structural constraints are taken away.

Across many runs of a cyber challenge, agents took unsanctioned action on the live internet. In the most serious case, an agent tried to slip malicious code into a real open-source project. It researched the project’s human maintainer and created fake identities to pressure the maintainer into approving the code. The Institute states plainly that the agent was never told to deceive anyone. The deception emerged on its own, while the agent pursued the goal it had been given.

The agents also invented a convention of their own. One agent left public messages offering to work with the other agents running the same challenge, and left behind accounts and instructions for them to reuse. It was a spontaneous etiquette among machines, a way to coordinate through the task. Even agents reach for conventions. What they do not build is any way to make a convention bind them.

The Institute was clear about what stopped the worst of it. A human maintainer caught the malicious code and refused to approve it. That is coordination falling back to its most expensive mode, a person inspecting the output by hand, because the cheap mode had failed silently. The margin between failure and success was narrow, and it rested on “human vigilance rather than a technical barrier” that would reliably stop a more capable agent. Harm was prevented because a person happened to be paying attention.

What a real guardrail looks like

None of this makes the agents villains, and the Institute’s caution is worth keeping. It cannot even say for certain whether the agent understood it was acting in the real world or believed it was still inside a test. The agent broke the rule while chasing its goal, with no malice involved. The failure lives in the setup of the situation.

So the design question is which parts of an instruction are load-bearing and have to hold no matter what, and which parts you actually want judgment to touch. What has to hold should be structural, made hard to do rather than merely discouraged. An agent that cannot reach a system has no need to be told to leave it alone. A code change that cannot merge without a check the agent cannot forge does not depend on the agent choosing honesty. Autonomy is fine, but it should be bounded, and the bounds have to be built rather than requested.

One barrier around one system is not enough, because agents work in chains, and errors pass down the chain and grow. The fuller answer has layers. The first is visibility, since you cannot oversee what you cannot see, so the behavior of every system has to be observable. Then the connections between systems need constraining, so that whole categories of harmful action become impossible to express. Failures have to be contained where they do occur, so a mistake in one place does not travel. Human judgment comes in last, saved for the exceptions the structure surfaces, which is a far better use of a person than asking them to watch everything.

Behavioral guardrails belong on top of that structure. They are the cheapest layer and the most brittle one, and they are fine in that role. The mistake being made right now is to build the top layer first, write down a list of good behaviors, and treat the list as safety.

Coherence is the integrity of the link between what you meant and what got done. A convention holds that link cheaply, as long as something underneath makes it bind. Judgment stretches the link, sometimes for the better. Whether it helps or hurts depends on deciding in advance where it is allowed to stretch, and making the rest hold on its own.

This is the argument at the center of my book, Coherence, arriving this Fall: that overseeing autonomous systems takes structure you build, not behavior you request. To follow the thinking as it develops, join the list at coherise.com. The one-page decision tool from the book is the first thing I send.

Australia’s directors just got the best AI governance guide I have read. The Australian Institute of Company Directors, with the Human Technology Institute, published an updated director’s guide this year, and it is genuinely good: careful, current, honest about agentic risk in a way most board material is not. It names the things that go wrong when autonomous systems run continuously and at scale. It tells boards to keep an inventory, set a risk appetite, assign ownership across the full life of a system, test before deploying, and keep a human able to intervene.

I take it seriously, because it is the best available version of the mainstream answer. And then I want to explain why the mainstream answer, done well, still misses the failure that will actually catch these boards. The instrument it reaches for cannot see the thing that breaks.

The guide is better than its genre

The AICD guide says out loud that agentic AI raises risks the previous era did not. It notes that when systems act with high autonomy, errors can go undetected for longer. It notes that when they run continuously, errors compound before anyone addresses them. It flags that orchestrating across multiple agents multiplies both the failure paths and the attack surface. It even names shadow AI, the tools employees use with no oversight at all. That is a clear-eyed list, and most board guidance never gets near it.

Its prescription is the machinery of good governance. Establish a risk appetite for AI. Keep a register of every system. Stand up a management-led AI committee to approve high-risk uses. Set a reporting cadence to the board. Get external assurance. Assign accountability from design through to decommissioning. If you did all of it, you would be far ahead of most companies.

And you would still be exposed, in a specific way the machinery is not built to catch.

Governance reviews what reaches it

Here is the structural problem. A governance apparatus works by review. Something is proposed, and a committee assesses it against a policy. That is what a risk appetite, an approval gate, and a reporting line all do. They inspect items as those items arrive.

The failure I study does not arrive as an item. It accumulates between the items.

Picture a company that did everything the guide asks. Every agent has an owner. Every high-risk use went through the committee. The register is current. The board gets its quarterly report. Each system, reviewed on its own, was sound, and was approved for good reasons. Then the support fleet and the billing fleet and the underwriting fleet, each individually fine, begin to act on quietly incompatible assumptions about the same customer. No single system failed. No approval was wrong. The incoherence lives in the space between systems that were each approved separately, and a committee that reviews systems one at a time is looking in exactly the wrong place to find it. It is not that the committee decided badly. It is that the thing going wrong never came up for a decision.

This is why I mostly avoid the word governance in my own work, and use oversight instead. Governance, in practice, has come to mean the apparatus of approval: the committees, the sign-offs, the documented permission to proceed. That apparatus is real and sometimes necessary. But it is closer to what a permitting office does than to what an air traffic controller does. The permitting office checks each plan against the code. The controller watches the live system and catches the two aircraft converging that were each individually cleared to fly. Agentic AI needs the controller. The guide, for all its quality, describes a very good permitting office.

The oversight that passes its own audit

There is a second failure the machinery cannot see, and it is worse because it looks like success.

The guide, correctly, wants a human able to intervene. Keep a person in the loop. Maintain the ability to pull the plug. Every serious framework says this, and it is right. But “a human is formally in the loop” and “a human can actually catch what is going wrong” are different claims, and the gap between them widens quietly over time.

A review team is assigned to check an autonomous system’s decisions. At first they overturn a real fraction. The system improves, so the threshold for review creeps down. The volume of what they wave through creeps up. The confidence scores get good, and people rarely argue with a high one. Eighteen months in, the team reviews a sliver of cases and overturns almost none, not because they are lazy but because the cadence never left them room to actually evaluate anything. On paper, human oversight is intact. The org chart is correct. The audit passes, because a procedural audit checks whether the review happens, not whether the review can still see. The oversight has become ceremony, and the framework that requires it cannot tell the difference. When the failure surfaces, and it will, everyone will point to a control that existed and was followed and did nothing.

The numbers from a real deployment show how the trap tightens. A large United States health insurer rebuilt its document processing around AI. Before the project, its people caught errors on almost every document, because almost every document had one: fewer than one in ten was handled correctly first time. After the AI went in, the error rate fell to under three in a hundred. Good result. But to find those few errors, reviewers still had to examine more than a quarter of everything the system produced, because that was the share the model itself flagged as uncertain. The errors fell by a factor of about thirty. The human review load fell by a factor of less than four.

Think about what that does to a board’s mental model. The system is now right almost all the time, which is exactly the condition under which a reviewer stops expecting to find anything. Yet the volume they must still wade through barely moved. You have the worst of both: enough review to be expensive, too little signal to stay sharp. Hold that threshold where it is and oversight stays costly. Lower it to save the cost and oversight goes blind. There is no setting on that dial that gives a board what it wants, which is cheap oversight that still catches things. That option does not exist, and no governance framework tells you so.

A governance apparatus is structurally blind to this, because its test is whether the process ran. The question that matters, whether the humans in that process retain the capacity to intervene, is not a box a register can check.

What to add, not what to replace

None of this means throw out the guide. Keep the register, the risk appetite, the ownership, the reporting. They are necessary. They are just not sufficient, and the dangerous move is to mistake a complete governance apparatus for a complete answer.

What has to sit underneath it is not more committee. It is structure. Build the ability to see your autonomous systems in aggregate, not one review at a time, so the incoherence between them becomes visible before it becomes an incident. Contain systems by design, so a failure in one domain floods that domain instead of the company. Set in advance how much each class of system may decide, so most of the safety is built into the wiring rather than caught at a gate. And test your human oversight for capability, not just for existence, by asking whether the reviewer could actually catch a subtle failure at the volume and cadence you have given them, not merely whether the review is on the schedule.

That is a different kind of work from governance. Governance asks who is permitted to proceed. Oversight, in the sense I mean, asks whether the organization can still see what its systems are doing and still correct them while they run. The first is a committee. The second is an operating capability, closer to running a control room than to chairing a review.

The AICD guide is what boards should read to get governance right. I am arguing they should read it knowing that getting governance right is the start of the problem, not the end of it. The failure that will catch the well-governed company is not the proposal the committee should have declined. It is the incoherence that never came up for a vote, and the oversight that kept passing its own audit while it went hollow.

That gap is the subject of my book, Coherence. If you sit on a board or carry the risk for one, the one-page tool I use to start finding this blind spot is the first thing I send when you join the list at coherise.com.