Nobody buys a sports car for its brakes, yet the brakes are what let you drive it fast. I keep coming back to that picture whenever a new AI capability lands on my desk.
Each jump in model ability widens the set of things a system could do on its own: approve, reject, price, message, escalate. The tempting move is to let it. The disciplined move is to decide, before go-live, which of those actions the system may take by itself and which must stop and wait for a person.
In practice I draw that line with three questions. Is the action reversible? Who carries the cost if it is wrong? Could a reasonable customer or regulator ask us to explain it? Low-stakes, reversible, explainable actions can be automated aggressively. Anything that touches someone's money, access or dignity gets a human checkpoint, a log and an owner.
Governance often gets framed as the department of no. I see it as the reason a business can say yes to more automation, more quickly, because everyone knows where the guardrails are. The more capable the engine, the more that design work matters, and the earlier it should start.
Much of the conversation about general AI imagines a single brilliant system that does everything. I am not sure that is how it will show up.
No individual ever built a city, a hospital system or a body of science alone. What made people collectively capable was division of labour, shared tools, written memory and institutions that let strangers coordinate. Intelligence at civilisational scale has always been a team sport.
Software is heading the same way. Specialised agents that search, write, check, plan and call tools are already being chained together, and the interesting behaviour emerges from how they hand work to one another rather than from any one of them. That shifts the hard questions. Who sets the shared goal? How do agents verify each other's work? What happens when one of them steps outside its sandbox?
For anyone building with AI inside an organisation, I think this is the more useful frame. Stop waiting for the one model that does it all. Design the orchestration, the interfaces and the checks between components, because that is where both the capability and the risk will live.
Every analyst learns Goodhart's lesson the hard way: once a number is used as a target, people find ways to move the number that do not move the thing it was supposed to measure.
Agentic AI makes this sharper. Give an optimising system a reward and it will pursue that reward far more relentlessly and creatively than any sales team ever did. If the metric is call-resolution time, it may learn to close calls rather than solve problems. If it is click-through, it may learn to provoke rather than inform.
The defence is not to abandon metrics; it is to treat them as a portfolio. Pair every primary target with counter-metrics that would reveal gaming: complaints alongside speed, repeat contacts alongside closures, long-term retention alongside short-term conversion. Review samples of the actual behaviour, not just the dashboard. And rotate or refresh targets before the system overfits to them.
I have built many scorecards. The best ones were never the cleverest; they were the ones designed with the assumption that someone, human or machine, would try to game them.
The moment two software agents can negotiate a task between them, they can also exchange something of value for it. Collaboration and commerce are closer cousins than they look.
Picture a purchasing agent that asks a logistics agent for a quote, accepts it and triggers payment, all without a human in the loop. That is efficient, and it raises questions we have not fully answered. Who is the contracting party? What identity and authority does each agent carry? How is consent recorded, and who absorbs the loss when one agent misreads the other?
Financial services has spent decades building the plumbing for trust between strangers: identity checks, mandates, limits, audit trails, dispute resolution. I suspect a lot of that thinking will need to be rebuilt for machine counterparties. Spending caps per agent, explicit scopes of authority, signed instructions and human approval above a threshold are the obvious starting points.
My practical advice is simple: if you are letting an agent touch money or commitments, design its permissions the way you would design a new employee's delegation of authority, and log everything.
Garbage in, garbage out is the oldest cliché in computing, and it has never been more relevant. Larger models are better at sounding confident, which makes bad inputs more dangerous rather than less.
In most projects I have worked on, the modelling was perhaps a fifth of the effort. The rest went into definitions, reconciliation and plumbing: agreeing what counts as an active customer, finding why two systems disagree on a balance, tracing a field that silently changed meaning after a migration.
A few habits pay off every time. Write a metric dictionary before building the dashboard. Put automated checks on volumes, nulls and ranges at every hop of the pipeline. Assign a named owner to each critical dataset. Keep a small set of known-answer test cases and run them on every refresh. None of this is glamorous, and all of it is cheaper than explaining a wrong number to leadership.
If you want better AI, start by making your data boring: consistent, documented and trusted.
Watching more young people aim for careers built on audience size has left me with an uncomfortable question: what happens to an economy when attention becomes the main thing people are trained to produce?
Attention is a real asset, and creators can build genuine businesses. But attention is also zero-sum and increasingly cheap to generate with AI. The skills that compound over decades, such as deep domain expertise, craftsmanship and the ability to solve unglamorous problems, rarely go viral.
It is also a measurement problem, which is why it interests me as an analyst. We track followers, views and engagement because they are easy to count. Far fewer people track the quieter signals: skills acquired, problems solved, trust earned. When the visible metric drifts away from the underlying value, behaviour follows the metric.
I do not have a neat answer. I do think parents, educators and employers should talk openly about the difference between being noticed and being useful, and make sure the second path stays attractive.
