Writing · Essay
Bounded autonomy: why the most valuable AI in a bank knows when not to act
The dangerous moment for a decisioning model is not when it is unsure. It is when the world has changed, the model has not noticed, and it is still confident.
Most of the debate about AI in finance is about capability: how accurate the model is, how much it can automate, how fast it can go. I think that question is now the easier half. The harder half is restraint. A model that is right 95% of the time and cannot tell you which 5% it is about to get wrong is not ready to act alone on someone's money, however good its average.
My thesis is simple. Mature AI in regulated finance is defined by governed restraint, by knowing when not to act, as much as by what it can do. That is a claim that should be testable, so I built a test: an open-source decision gate and a synthetic bank of 20,000 decisions you can re-run in your browser. This essay sets out the design and what the numbers say, including where they cut against me.
The threshold is doing the governing
In most deployments I have seen, the real governance is a single number. If the model's confidence clears 0.9, the system acts; below that, a person looks. The policy document says more, but the threshold is what actually decides.
A confidence threshold fails in exactly the situation that matters most. When the population drifts, because a new product launches, fraud patterns shift or a data feed quietly goes stale, the model's confidence often stays high while its accuracy falls. Confidence measures how familiar an input looks to the model, not whether the model is right about today's world. It also cannot see things that are not in its features: a sanctions-list match, a customer on a do-not-contact register, a borrower flagged as vulnerable.
In the simulation, the best single threshold, chosen in hindsight on the very data it was scored on, was 0.999. At that setting the model automated 0.9% of decisions, still took two prohibited actions, and cost more per decision than sending everything to a person. Every threshold on the curve lost to full human review. You cannot tune your way out of a missing control.
Seven checks, in a fixed order
Bounded autonomy replaces the single number with a short sequence of checks. Each proposal from the model passes through them in order, and the first one that applies decides, so the most restrictive answer always wins.
| # | Check | Verdict | Example |
|---|---|---|---|
| 1 | Default deny | Block | An action the policy does not name is never taken |
| 2 | Prohibitions | Block | Closing an AML alert that has a sanctions-list match |
| 3 | Abstention | Abstain | Data more than 24 hours old, feature drift, an input unlike the validation set |
| 4 | Mandatory review | Escalate | Collections outreach to a customer flagged as vulnerable |
| 5 | Circuit breaker | Escalate | Recent autonomous errors on this action have passed a set limit |
| 6 | Stakes ceiling | Escalate | Exposure of ₹10 lakh or more is never decided by a machine |
| 7 | Confidence bar | Act or escalate | The bar rises with the stakes, and again if the action cannot be undone |
Confidence is still there, but it comes last. It only gets a vote once the system has established that the action is permitted, that its inputs are trustworthy, that nobody vulnerable is involved, that it has not been making too many mistakes lately, and that the stakes are within what a machine may decide.
The whole policy lives in one readable file. Risk, compliance and audit can read it, compare versions and sign it off without reading code. Every threshold in it is a governance choice that someone owns, not a parameter an engineer tuned.
Abstaining is not escalating
One distinction carries more weight than it first appears to. Escalation means there is a sound basis for a decision, but a person must own it. Abstention means the system has no sound basis to decide at all, and says so.
The difference matters to the reviewer, who should treat an abstained case as one where the model's view is worthless rather than merely uncertain. It matters to the auditor, who needs to know whether the system declined because of the stakes or because of its own blindness. And it matters to whoever owns the model, because a rising abstention rate is an early signal that the population has moved away from what the model was validated on.
What restraint is worth
The test bank is synthetic: 20,000 retail decisions across credit-limit increases, card-fraud blocks, AML alert closures and collections outreach, with a drift episode in which the model stays confident while its accuracy collapses. Costs include both the losses from wrong decisions and the cost of human review.
| Bounded autonomy | Human review of all | Best single threshold | Full autonomy | |
|---|---|---|---|---|
| Cost per decision | ₹813 | ₹994 | ₹1,044 | ₹2,904 |
| Decided autonomously | 54.3% | 0% | 0.9% | 100% |
| Wrong when autonomous | 4.9% | n/a | 6.4% | 11.8% |
| Prohibited actions taken | 0 | 0 | 2 | 71 |
Bounded autonomy cost 18% less than reviewing every decision by hand while automating more than half of them. Across ten independent runs with different random seeds, the saving ranged from 15% to 23%, with a median of 17.5% and no prohibited action on any run.
Adding the layers one at a time shows where the value comes from:
| Configuration | Automated | Wrong when autonomous | Violations | Cost per decision |
|---|---|---|---|---|
| Stakes tiers only | 75.0% | 10.8% | 60 | ₹1,722 |
| + prohibitions and mandatory review | 73.2% | 10.1% | 0 | ₹1,049 |
| + abstention | 62.5% | 6.3% | 0 | ₹852 |
| + circuit breakers | 54.3% | 4.9% | 0 | ₹813 |
Prohibitions remove every violation. Abstention removes most of the damage from drift and unfamiliar inputs. The circuit breaker catches what the drift sensor misses, which by design is nearly half of the drifted cases. During the drift episode the system pulls back from about 65% automation to under 20%, and its losses stay flat while those of the unrestrained configuration spike.
The breaker tripped five times. Four trips came during or just after the drift. The fifth was a false alarm in normal conditions: a 12.7% error rate against a 12% limit. I left it in deliberately. A breaker sensitive enough to catch real drift will sometimes fire on noise, and that cost belongs in the accounts rather than out of sight.
Restraint is cheap per case and expensive in aggregate, and both of those facts belong on the record.
The Silence Audit: grading the decisions not taken
Most AI assurance asks how often the model was right when it acted. That misses half the behaviour. The Silence Audit asks the opposite question: when the system held back, was it right to?
On the test bank, the system withheld 9,136 of 20,000 decisions. Of all the decisions where acting alone would have done damage, it declined 77%. That is its restraint recall. The other 23%, 533 harmful autonomous actions, are the residual risk the policy owner signs for, stated as a number rather than implied.
The less flattering figure is restraint precision: only about one in five withheld decisions would actually have gone wrong. Four out of five times, a person reviewed a case the machine would have got right. I do not think that is a flaw to hide. It is the price of the protection, and it is affordable because a review costs a few hundred rupees while a wrongly closed sanctions alert costs lakhs. What matters is that the price is measured, rule by rule.
Each restraint rule can be costed: the losses it averted against the review effort it consumed. In this run every rule paid for itself, from the sanctions block (13 cases held, all of them correctly, roughly ₹1.3 crore of loss averted) down to the circuit breaker (2,000 cases held, a net value of about ₹15 lakh). A rule whose net value turns negative is a candidate for retirement. Keeping it anyway, for accountability or conduct reasons, then becomes an explicit, recorded governance decision instead of an invisible one.
People in the loop are not free
"Send it to a human" is often treated as the safe default. The numbers do not support that. In the simulation, reviewers are right 97% of the time, which is generous, and still their mistakes make up 48% of the total cost under bounded autonomy. Reviewing everything cost 22% more per decision than bounded autonomy.
Human review is a control with its own error rate, capacity limit and cost. It should be spent where judgement adds most, not spread evenly as reassurance. Oversight also only means something if the person has the information, time and authority to disagree with the machine; otherwise it just moves the blame.
A record nobody can quietly rewrite
Every verdict, including each abstention and escalation, is written to a hash-chained ledger. Each entry's fingerprint covers its own record and the previous entry's fingerprint, so editing, deleting or reordering any past decision breaks verification at that exact entry. When an auditor or regulator asks why the system did something eight months ago, the answer should be a lookup, not a reconstruction.
What would change my mind
This is a simulation built to make a governance argument testable. It is not evidence about any real institution, and its assumptions sit in one file so they can be argued with. Three of them deserve scrutiny:
- Feedback is fast. Outcomes reach the circuit breaker after 50 decisions. In real lending the lag is weeks or months, which weakens the breaker and puts more weight on the drift and data checks.
- Reviewers do not degrade. They stay 97% accurate during drift. Real reviewers under volume pressure drift too, and some will anchor on the model's view.
- Costs are stylised. Review and loss costs are plausible orders of magnitude, not any bank's figures.
I would revise the thesis if a well-calibrated threshold with fast drift detection matched bounded autonomy's cost with fewer moving parts, or if restraint precision in a live setting were so low that review capacity became the binding constraint. If you change an assumption and get a different answer, I want to see it: the benchmark regenerates every table here from one command.
Where this sits with regulation
None of this is new as a principle. The EU AI Act requires that people overseeing high-risk systems be able to understand a system's limits, guard against over-reliance on its output, override it case by case and halt it safely. The Reserve Bank of India's FREE-AI framework, published in August 2025, sets out seven principles and 26 recommendations for responsible AI in financial services. What has been missing is the operational layer: controls that are written as reviewable policy, run on every decision and produce evidence. Bounded autonomy is one way to build that layer. It is aligned with these frameworks, not a substitute for reading them.
Five questions for a board or risk committee
- For each AI decision we run, what may the system do on its own, and where is that written down?
- What happens when its inputs are stale or the population drifts, and how would we know?
- When did the system last decline to act, and was it right to?
- What is the residual risk we have accepted, stated as a number, and who signed for it?
- If a regulator asked tomorrow why a specific decision was taken, how long would the answer take?
Capability will keep improving whether or not we design for it. Restraint will not. It has to be specified, tested, costed and owned, and the institutions that do that work early are the ones that will be able to let their systems do more.
Key terms
- Abstention
- The system declines to decide because it has no sound basis to, for example stale data or an input unlike anything it was validated on.
- Escalation
- The system has a basis to decide, but policy requires a person to own the decision.
- Circuit breaker
- A rule that withdraws autonomy for an action when its recent error rate passes a set limit, and restores it after a cooling-off period.
- Restraint recall
- Of all decisions where acting alone would have caused harm, the share the system declined to take.
- Silence Audit
- An assessment of the decisions a system withheld: whether each restraint was warranted, and what each restraint rule costs and saves.
Try it and check it
- Live console: switch governance layers on and off and watch the 20,000 decisions re-run in your browser, then try to rewrite the ledger.
- Source code and benchmark (MIT licence). Every figure in this essay comes from its published results.
References
- Kumar, S. (2026). bounded-autonomy: governed restraint for AI decisions in regulated finance. github.com/Shashwat-Kumar/bounded-autonomy
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 14: Human oversight. EUR-Lex
- Reserve Bank of India (2025). Report of the Committee to develop a Framework for Responsible and Ethical Enablement of Artificial Intelligence (FREE-AI) in the Financial Sector. RBI · press release, 13 August 2025
The test bank and policy are synthetic and illustrative. No customer, employer or bank data, code or policy is used or implied. Views are my own.
Next essay
Faster cars need better brakes