Notes from inside companies that run on AI.

Your AI problem is an identity problem

Every agent rollout we've run has stalled in the same place. Not model quality. Not context windows. Permissions.

There's a version of the AI conversation that's entirely about capability. Which model reasons better. How long the context window is. Whether the next release will finally handle your edge cases.

We deploy agents inside real operating companies, and we can tell you that almost none of our blocked projects have been blocked by capability. They've been blocked by an admin who couldn't grant a scope, an OAuth consent screen nobody owned, a credential sitting in someone's password manager, or a simple unanswered question: who is this agent, exactly, and what is it allowed to do?

The models got good faster than our permission systems got ready for them.

Every enterprise tool assumes a human

Identity and access management was designed around a person: one human, one account, one login, one set of roles, revoked when they leave. It handles delegation badly and non-human actors barely at all.

Now you want to introduce something that isn't a person, works across a dozen systems at once, acts hundreds of times a day, and needs a permission surface that's narrower than an admin but wider than any single employee. There's usually no clean way to express that. So teams do one of two things, and both are wrong:

• Give the agent a super-admin account. It works immediately. You now have an unaccountable actor with keys to everything and no meaningful audit trail.
• Give the agent a real person's credentials. Also works. Now your audit log is a lie, and when that person leaves, an unknown amount of automation dies with them.

We've seen both, more than once. The second is more common and more dangerous, because it looks fine until it doesn't.

Treat the agent as a first-class identity

The fix isn't clever prompting. It's boring infrastructure, and it should be in place before the first agent does anything useful:

• Its own identity. The agent gets an account, not a borrowed one. It appears in your directory. It can be suspended without taking a human's access with it.
• Scopes tied to a job, not a person. Don't ask "what can Nina do?" Ask "what does processing an invoice require?" That's a much smaller list, and it's stable when people change roles.
• Credentials held centrally, never pasted. A vault, with rotation. If a key lives in a prompt, a config file or a chat thread, you've already lost the thread of who can do what.
• An audit trail that names the agent. Every action attributable, reviewable, reversible. If you can't answer "who did this and under whose authority", you don't have automation — you have exposure.
• A revocation path you've actually tested. Not theoretically. Actually turn it off once and see what breaks.

The uncomfortable part

This work is unglamorous. It produces no demo. It's the least interesting slide in any deck and the first thing cut when a pilot needs to show progress by Friday.

It's also, in our experience, the single largest determinant of whether an AI programme reaches production or dies as a proof of concept. The pilots that stall don't stall because the agent was stupid. They stall because month three arrives and nobody can safely widen the agent's access without someone senior signing off on a risk they can't articulate.

Capability is being solved for you, quickly, by people with far more compute than you have. Identity is the part nobody else can do on your behalf. If you want a serious answer to "can we let this thing act", start there — long before you start comparing models.

Self-improving companies need memory, not just better models

A loop that forgets isn't a loop. It's a repetition. The missing primitive in most AI deployments is somewhere to put what was learned.

Tom Blomfield — who built Monzo and GoCardless, then spent years at Y Combinator — recently joined Anthropic, and has been making the case for what he calls recursive, self-improving AI loops: companies that get better while the people who run them are asleep.

We think that's right, and we'd add the part that gets skipped. A self-improving system needs three things, and everyone focuses on the first two.

Act. Observe the outcome. Remember what you learned. The third one is where almost everyone stops

Plenty of companies have agents that act. A good number now measure what happened afterwards. Very few have anywhere to put the lesson so the next run starts smarter than the last one. Without that, you don't have a loop. You have a very fast circle.

What "memory" actually means here

Not chat history. Not a longer context window. Those are memory of a conversation, and conversations end.

What a company needs is memory of how it works — a durable, structured record of its own operating decisions. When an agent gets something wrong and a human corrects it, that correction has to land somewhere permanent, and it has to change what the agent does next time. Otherwise every person who works with it pays the same tax, forever, teaching the same lesson to something that cannot learn.

The organisational version of this is worse than the technical one. Companies already fail at it without AI. The reason a refund gets approved is in someone's head. The reason you don't discount that account is in a thread from 2023. The exception that exists for one customer was agreed verbally and never written down. That knowledge doesn't survive a resignation, let alone get handed to a machine.

Make correction a normal thing people do

The most useful habit we've introduced isn't technical. It's that when an agent gets something wrong, the person who noticed tells it — in the open, in the channel where the work happens — and that correction is captured rather than muttered.

Two things happen. The obvious one: the system improves. The less obvious one, which matters more: people stop treating the agent as a fixed appliance that is either good or bad, and start treating it as something they're responsible for. Adoption stops being about trust in the vendor and starts being about ownership.

It also surfaces a genuinely uncomfortable fact. The moment you try to write down why something is done a particular way, you find out how much of your company nobody can explain. That's not an AI problem you've discovered. It's an organisational one you've been carrying, and it was always going to be expensive.

The compounding argument

Here's why this matters commercially rather than philosophically.

A model is available to your competitors on the same terms as to you, at the same price, this afternoon. Whatever advantage you get from picking a better one lasts until the next release. It is rented, and everyone rents it.

The record of how your specific company operates — every correction, every exception, every hard-won reason a thing is done this way and not that way — is not available to anyone else at any price. It's the one asset in an AI deployment that compounds instead of depreciating.

So by all means switch models when a better one lands; we do, and you should stay ready to. But be clear about which layer is the commodity and which is the company. The model is the engine. The memory is the business.

The marginal cost of correctness is going to zero. Knowing what "correct" means isn't.

When producing the right answer becomes cheap, the bottleneck moves to specifying it. Almost no company has written theirs down.

Ben Glickenhaus put a line in front of a lot of engineers recently: the marginal cost of correctness is going to zero, and you need to internalise what that means for how you practise software engineering.

He's right, and the implication runs well past engineering.

For most of the history of software, correctness was rationed. Tests, types, documentation, code review, careful error handling — all obviously good, all competing for time against shipping. Being thorough was a trade-off you argued about in planning. When the cost of that thoroughness collapses, the argument disappears. There's no longer an excuse for the untested path or the undocumented decision.

The constraint just moved

But cheap correctness only helps if something defines what correct is.

In code, we're lucky: correctness is comparatively well specified. A test encodes it. A type system encodes it. A schema encodes it. Hand an agent a failing test and it can iterate against a definition of done that somebody wrote down.

Now try the same thing one level up, in the business.

• Is this refund correct?
• Is this pricing exception correct?
• Is this the correct way to onboard this particular kind of customer?
• Is this response to this complaint correct?

There's no test suite. There's a person who knows. Sometimes there's a document, usually out of date, often contradicted in practice by the person who knows. The specification exists — companies are astonishingly consistent about these decisions — but it lives in heads and habits, not anywhere a system can read.

So you get the failure mode that defines this era of enterprise AI: agents that are flawlessly correct against the wrong specification. The output is well-formed, confident, fast, and wrong in a way that takes a human who knows the business about four seconds to spot and about four weeks to systematically prevent.

Specification is now the scarce skill

If producing the answer is cheap and defining the answer is expensive, then the valuable work in a company shifts decisively toward definition.

That's a real change in what good operators do. Less doing the task; more articulating, precisely enough for a machine, how the task is judged. What the exceptions are. Which rule wins when two conflict. Where the boundary sits between "decide it" and "escalate it".

Most organisations have never had to be that explicit, because humans are extraordinarily good at absorbing ambiguity. A new joiner watches, asks, gets it wrong twice, and calibrates. That tolerance is exactly what let the specification stay unwritten for twenty years — and it's exactly what agents don't have.

What we'd do first

Take one process that actually matters and costs real money — refunds, approvals, onboarding, incident response. Write down how it truly works, including the exceptions people apply without thinking. Then check that against what actually happened over the last quarter.

The gap you find is the real state of your operation. Everyone we've done this with has found one, us included. Close it and you've built something more durable than any automation: a definition of correct that a system can be held to.

Cheap correctness is arriving whether you prepare or not. The companies that benefit are the ones that can say what they mean.

The self-driving company: six levels of operational autonomy

The car industry agreed on what autonomy means before it built it. Companies are automating without that vocabulary — so here's ours.

PostHog recently shipped something they describe as putting your product into self-driving mode. The shape of it is worth studying: signals come in from errors and session replays, agents cluster them into reports, reports become pull requests, and nothing reaches production until a human clicks merge. Every change ships with the instrumentation to measure whether it worked — and if the metric doesn't move, that becomes a new signal.

That last part is the bit most people skim, and it's the whole thing. The loop closes.

It's also not unique to product engineering. The same shape is already running in places most companies don't think of as autonomous:

• Dependency management. Tools like Dependabot and Renovate watch for updates, open the pull request, run your tests, and wait for a human. Nobody calls this AI any more. It's just how repositories work.
• Ad platforms. Google and Meta's automated bidding has been setting budgets and creative allocation against an outcome signal for years. Most marketers now describe their own performance targets rather than their bids.
• Fraud and risk. Payment fraud systems score, decide, and retrain on the outcome of their own decisions, at a volume no review team could touch.
• Infrastructure. Autoscaling and automated rollback have been closing the observe-decide-act loop in production for a decade.

In each case the pattern is identical: a signal, a decision, an action, and — critically — a measurement that feeds back. What's new is that the same loop is now affordable for the messy, language-shaped work that used to need a person: support, operations, finance admin, reporting, coordination.

The levels

Automotive engineering got something right early: before building autonomy, the industry agreed a shared vocabulary for what each degree of it meant. Companies are automating without that, which is why "we're doing AI" tells you nothing. Here's the scale we use internally.

Two observations from doing this in live businesses.

Most companies are at Level 1 and think they're at Level 3. Rolling out a chat assistant to everyone feels transformative — usage is up, people are pleased — but nothing changed structurally. Remove the tool and the company runs exactly as before, slightly slower. That's a productivity feature, not autonomy. A useful test: if a person has to remember to start it, you're at Level 2 at best.

The hard jump is 3 to 4, not 0 to 1. Getting an agent to act is now easy. Getting it to know whether it did the right thing means you need a definition of the right thing, instrumentation on the outcome, and somewhere durable to keep what was learned. That's not a model problem. That's an operating-system problem, and it's why most deployments plateau at "impressive demo, unclear impact".

Humans in charge, not humans in the loop

One deliberate word choice. "Human in the loop" positions the person as a step in a pipeline — a checkpoint, and eventually a bottleneck to be removed. That framing quietly makes your best operators into approval clicks.

We build for humans in charge. The person sets intent, defines the boundary, and reviews exceptions. The system does the volume and reports what it did. Nobody's job is to rubber-stamp; the job is to decide what good looks like and to notice when reality disagrees.

That distinction isn't sentimentality. Level 4 only works if someone with judgement is watching the measurements and willing to say the loop is optimising for the wrong thing. Take that person out and you don't get a self-driving company — you get one confidently accelerating in a direction nobody chose.

Systems thinking is the whole job now

Netflix's CPTO says it's the top skill she hires for. We'd go further: most companies can't think in systems because they can't see their own.

Elizabeth Stone, Netflix's Chief Product and Technology Officer, was asked what skill matters most now. Her answer wasn't a technology. It was systems thinking — the ability to hold the whole interconnected thing in your head rather than being excellent at one isolated piece. She also frames excellence as an operating system: something structural a company runs on, not a personal virtue a few people happen to have.

We'd push that one step further, from the perspective of people who install this machinery inside businesses for a living.

Systems thinking isn't primarily a skill shortage. It's a visibility shortage. You can hire the most capable systems thinker available and drop them into a company where the system is unobservable, and they will do what everyone else does: form a partial mental model from whatever they can see, and optimise their corner of it.

Your org chart is not your system

The org chart is a map of reporting lines. It says almost nothing about how work actually moves.

The real system is the path a refund takes from request to money leaving the account. It's who gets consulted before a pricing exception is granted, and on what basis. It's how an incident escalates at 2am, including the step where someone messages a colleague directly because the official route is too slow.

None of that is in the org chart. Very little of it is in any document. It's distributed across people's heads, and every individual holds a genuine but partial view. The company as a whole knows how it works. No single person does — and no system does either.

That was survivable when execution was human. People are superb at routing around missing documentation: they ask, they infer, they escalate, they learn by getting it wrong once. Institutional knowledge could stay tacit because the executors were the same people who held it.

AI removes the tolerance

Software that can act — rather than just record — has to be told how the company works. It doesn't absorb ambiguity by watching. It can't read the room, ask the person two desks over, or sense that this customer is a special case for reasons nobody wrote down.

This is why so many capable AI deployments underdeliver. The model isn't the limitation. The limitation is that it's operating on a company that has never made itself legible — to software, or frankly to itself.

So the demand for systems thinking is real, but the sequence usually gets inverted. You don't get systems thinking by hiring for it. You get it by making the system visible, at which point a lot of ordinarily capable people start thinking in systems — because they finally can.

Excellence as an operating system

Stone's framing is the useful one. Treat excellence as infrastructure rather than as a set of talented individuals, and it survives people leaving. Treat it as individual virtue and you're re-teaching it to every new joiner, forever, with drift each time.

The practical form of that: a living map of how the company operates — how decisions get made, what the exceptions are, which rule wins when two conflict — maintained as a working artefact rather than a document written once and abandoned. Once that exists, two things become possible that weren't before. People can see the whole. And software can act on it.

Those are the same problem. That's the part we think most companies still haven't clocked: the work of becoming legible to AI is identical to the work of becoming comprehensible to your own team. You were going to have to do it anyway. AI just made it urgent, and put a number on the cost of not having done it.