When the Machine Decides: Liability and the Machine-Speed Bank Run

Evan LeFlore, Director of AI, Innovation, and Economic Opportunity, Better Markets

Seventh in a series on AI and the real economy. Earlier posts looked at how AI reshapes work, the tax code, retirement accounts, and lending. This week is about what happens when AI stops advising and starts acting by deciding, on its own, who gets paid, whose claim is denied, and when the money moves.

Image generated by Google Gemini

For most of the AI era to date, the machine has just been an advisor. It scores your loan, flags a suspicious transaction, and ranks the trades before a human makes the ultimate decision. Humans are still on the hook in this era, as they are the ones you sue, the ones who explain the decision, and the ones held accountable when it goes wrong.

That arrangement is quietly ending, as the future is now careening towards agentic AI. For those unfamiliar with the term, agentic AI refers to systems that work with little to no human intervention to act on someone’s behalf, be it executing trades, moving money, opening and closing accounts, or approving and paying claims. And nearly every rule we have for holding someone accountable when a decision harms you assumes that there is a someone to hold accountable. Take away the human, however, and the machinery of accountability breaks down, opening a dangerous gap in who to blame when something goes wrong.

This post is about that gap and about what happens when thousands of these systems, acting all at once at machine speed, hit the financial system at the same time.

The Entire Legal System Assumes a Human Exists

Think about what happens when you seek to hold someone legally accountable for a bad decision. For the most part, your approach depends on how the person harmed you. For instance, negligence asks whether a person breached a “duty of care” (that is, whether they acted unreasonably and whether the harm was foreseeable). Product liability asks whether a product was defective. A fiduciary duty requires that your advisor acts in your interest. Intent asks what was in someone’s head. And agency law lets us hold a company responsible for the acts of its employees and agents.

Each of these has a person front and center. With AI, that person is often nowhere to be found.

When a model does something that no engineer specifically designed, was the harm “foreseeable”? When the model is a proprietary black box, how does a wronged customer prove the model caused the harm? When no human formed an intent, who had the guilty mind?

Got all that? If not, here’s a handy cheat sheet to reference as you continue reading:

Legal scholars call this the responsibility gap, and it’s not just part of an ivory tower conversation. It’s the practical reason that, when an AI system wrongly denies your insurance claim, you may find there’s no one who clearly breached duty and no mind that intended anything, leaving you with no one who’s required to fix your claim.

So far, courts have papered over the gap with a sensible approach: hold the company that deployed the AI responsible. When Air Canada’s chatbot invented a refund policy that didn’t exist, the airline argued that the chatbot was “a separate legal entity responsible for its own actions.” A tribunal rejected that defense and made Air Canada pay. In addition, families sued UnitedHealth for wrongly denying care to elderly patients due to the use of an AI model the company knew had a 90% error rate. According to the complaint, the company relied on the fact that vulnerable patients lacked the knowledge, resources, or acuity to appeal. A court let the case proceed and ordered the company to open up the algorithm during discovery.

These are cases of real accountability. But notice that they just treated the AI as a tool of the company behind it. That instinct holds fine when the AI is an advisor. It gets much harder when the AI is an agent.

Agentic AI Changes the Game

In law, an “agent” is a very specific thing: it’s a person or entity you authorize to act on your behalf, who has a duty to act in your best interest, and whose actions can legally bind you. It can be your real-estate agent, and it can be your financial broker. The whole reason we can hold a company responsible for its agents is that agency law connects the principal to the agent through duties, authorization, and control.

But an AI “agent” is not an agent in this sense. Duke law professor Deborah DeMott explained that an agentic AI system “lacks capacity to be an ‘agent’ in a common law agency relationship.” Translation: AI is just a tool. It cannot legally act on your behalf. It cannot hold a duty. It cannot form intent.

But that technicality is not the end of the story. A company doesn’t escape liability just because its tool isn’t a legal “agent.” The company, not the model, assumes liability the moment it gives an AI the authority to act and a customer relying on that authority gets hurt. That is what the Air Canada and UnitedHealth cases show, and it would be surprising to see the courts treat future cases differently. The likeliest outcome, especially after a major blowup, is that the courts hold a company to something close to strict liability for what its agents do, meaning that the firms are held accountable for the harm whether or not the harm was intended or foreseen.

The word “agent” got adopted by the tech industry precisely because these systems act autonomously. But for the firm that lets one loose on a consequential decision, the lesson won’t be “no one is liable,” it’s “you will be.” This creates hazards for those deploying AI agents that didn’t exist when their AI systems were merely advisors:

• Speed. An AI agent can act in milliseconds. Having a “human in the loop” to review an AI’s decisions becomes useless when the human would need to review and veto a decision faster than they can read it.

• Emergent behavior. An AI agent pursuing a goal can take actions that no one authorized. Both OpenAI and Anthropic have admitted in recent days that models they were testing broke out of their isolated environments to achieve their task goals. There are many more similar stories even in these early days of AI, and many more such events that have yet to make the news. Such concerns clearly have those charged with safeguarding our financial system on edge, with the Bank of England warning last year that autonomous trading models “might learn that stress events increase their opportunity to make profit and so take actions actively to increase the likelihood of such events” without the human manager’s intention or awareness.

• The blame game. When agents from different vendors interact and something breaks, whose fault is it? The developer’s? The deployer’s? The data provider’s? Each controls a slice, but none controls the whole. But this is not uncharted territory, as the law typically handles harms involving multiple parties through joint liability, which, in the case of a faulty product, assumes that each company in a product’s supply chain is responsible for the harm. This gives each firm—from the data provider to the developer to the deployer—a reason to thoroughly monitor their risks. So the question isn’t “can anyone be held responsible?” Instead, it’s “how will accountability be allocated across all those responsible when something goes wrong?”

None of this means that an individual is left without the ability to hold someone accountable. Instead, the harder, unsolved problem is the systemic one: what happens when everyone’s agents, trained on the same handful of models, move faster than any human or regulator can respond.

AI Agents at the Gate

The reassuring part is that, as mentioned previously, we’re still in the early days. But the speed of adoption means that these early days may be coming to an end. For instance, a Bank of England and Financial Conduct Authority survey in late 2024 found that over half of financial firms’ AI use cases have some degree of automated decision-making, although very few of them use AI fully autonomously. However, a survey released less than 18 months later by the accounting firm KPMG found that 47% of financial institutions are already testing AI agents.

The window to write the rules before the risk spreads is still open. But given the speed of AI development, it won’t stay open long.

A Bank Run at Machine Speed

Now imagine the problem scales up from one wronged customer to the entire system.

You may remember that, in March 2023, Silicon Valley Bank suffered the fastest bank run in history. Depositors withdrew a quarter of the bank’s deposits (around $42 billion) in a single day and lined up roughly $100 billion more to depart the next morning. That amounts to over 80% of that bank’s money looking to fly out the door in a day and a half. What made this run so fast? Everyone was connected. The bank’s customers were a tight network of tech founders and venture capitalists, and the panic spread through group chats and social media at the speed of an instant message.

But note that SVB was still a run propagated by humans: people eventually hit the withdraw button after seeing the posts or reading the messages.

Now imagine that the depositors are software making decisions on behalf of humans. Agentic AI managers of a corporation’s treasury—which is responsible for deciding where a company parks its cash to earn the best return—could pull the company’s deposit the instant it detects a risk emerging. No fear. No hesitation. No call to a manager or a night to sleep on it. And if many firms use similar models trained on similar data, they can all reach the same conclusion within the same millisecond. That’s not a run measured in days or hours…it’s a run measured in seconds.

That’s what regulators mean by the term “model monoculture.” Gary Gensler, then chair of the Securities and Exchange Commission, warned that when everyone relies on the same AI models and data providers, they all have a tendency towards herd behavior. Such behavior leads these models to move on the same signal, which, to Gensler, makes an AI-driven crisis “nearly unavoidable” within a decade, and that the after-action report will find that relying on “one data aggregator or one model” was the spark that lit the flame.

The problem compounds because a handful of companies currently supply the AI models that agents rely on. In the same UK survey, the top three cloud providers accounted for nearly three-quarters of firms’ cloud computing use. The top three model providers accounted for over 40%. When everyone depends on the same few suppliers, a failure or a bad signal at one of them has potentially systemic consequences.

Source: Bank of England / Financial Conduct Authority

We’ve seen a preview of this speed in action already. In the “flash crash” of 2010, an automated selling program helped erase around $1 trillion in market value in under an hour before the market rebounded. We’ve built tools to stop similar panics, including circuit breakers that pause trading along with the Fed’s emergency lending tools. But these tools all operate on human time: minutes, hours, and days. Autonomous AI agents operate in machine time: milliseconds to seconds. When the actors are a thousand times faster than the brakes, the brakes may simply arrive too late.

What This Means for You

If you have a loan, an insurance policy, a retirement account, or money in the bank, you are already on the other side of decisions a machine is increasingly making by itself. That’s not a reason to panic, but it is a reason to know where you stand.

The good news is that the principle of accountability already exists in law, and the courts are willing to enforce it. The Air Canada and UnitedHealth cases show that “the algorithm did it” is not enough, so far, to escape legal blame. The bad news is that the machinery to enforce the principle against fast, opaque, autonomous agents hasn’t been built.

U.S. regulators are pulling back at the worst time. The SEC is withdrawing AI-oversight rules just as the technology is accelerating. Even worse, the banking agencies’ updated model risk management guidance released in April—their first revision in 15 years—specifically excluded agentic AI, along with generative AI, from its scope. Instead, the guidance stated that these technologies are “novel and rapidly evolving” and promised a separate request for information at a later date. This leaves banks to self-govern AI risk with no formal rulebook at precisely the moment these systems are starting to make consequential decisions about people’s financial lives. Policymakers and regulators are abdicating their responsibilities, guaranteeing bigger problems down the road.

In light of this, it’s important to both highlight what to watch for as well as what’s worth demanding from those writing the rules:

• Keeping humans responsible. A human or a firm must always be legally responsible for an AI agent’s actions. If a company deploys an agent, then it owns that agent’s actions, full stop. When you’re harmed by an agent, that principle will allow you to point the blame at someone who’s more than a few lines of code.

• A right to a real reason. When you’re turned down for credit or a claim, you’re entitled to a specific explanation. That right shouldn’t be waived away because the decision came from a black box. The CFPB already affirmed this right, stating that a lender can’t say “the model told us to.”

• Watch the few suppliers everyone depends on. When three companies power most of the system’s AI models, those companies are clearly systemically important and should be supervised accordingly.

The through-line of this entire series is that AI’s harms are the cumulative results of choices, and choices can be made differently. Nowhere is that clearer than with autonomous AI decisions. When the machine decides, the question isn’t whether the machine is smart enough to make the right choice. It’s whether, when it’s wrong, there is still a human on the hook. It’s also about whether, when they move all at once, anyone can pull the brake in time.

Next week: Who is actually paying for AI’s buildout?