AI is beginning to act independently of humans, but if it makes a flawed decision, who is responsible? asks SCOTT MONCUR
Artificial Intelligence is on everyone’s agenda and, as it develops beyond passive tools, we need to be aware of something more dynamic and potentially disruptive: Agentic AI. This appears to act independently, make decisions, and execute multi-step tasks without constant human validation. For businesses, this evolution offers significant efficiency gains but introduces a new risk category.
At its core, the starting point remains simple: AI is a tool. Intelligently designed and deployed with appropriate safeguards, it can operate effectively.
AI systems improve productivity, reduce costs (excluding its own), and enable better decision-making by colleagues. The analogy to a “tool” begins to stretch as AI decision making improves.
The moment a system begins to decide for, not simply support, a business urgent attention needs to be given to responsibility for, control of and accountability for the outcomes. The defining feature of agentic AI is autonomy, which equals significant risk in terms of decision-making authority.
Humans will set objectives, but the system determines how to achieve them. This creates uncertainty: if AI makes a flawed decision, who is responsible? The developer, the deployer, or the firm that delivered the output? What are the commercial ramifications for the user?
The “black box” problem is to accept solutions without fully understanding how they were generated. Black boxes have always existed in business.
If Argentic AI is deployed businesses must define clear boundaries for the system, its permissions and set checkpoints for decisions taken by Agentic AI to be ratified and cleared, just as if it were a human processed solution for customers. Put simply businesses must treat Agentic AI as they would any colleague with delegated authority.
Agentic AI without guardrails is reckless and making “guardrails” work involves organisational change. Guardrails explain to everyone, particularly customers, what the AI does in delivering objectives and are important to manage risk
Testing through sandbox environments is the best guardrail. AI systems tested against intended outcomes in controlled settings against synthetic and real data will test and evidence how the system behaves, identify edge cases, and ensure that outputs meet expectations. Sandboxing is not just a technical exercise but a key component of winning and retaining clients through risk management.
An audit function is also critical. Agentic AI systems can take multiple steps in sequence, and their decision-making paths may not always be obvious. Maintaining detailed logs of inputs, outputs, decisions, and actions is essential, both for checking decisions, governance and for defending potential claims.
And, of course, there is scope for further concern if Agentic AI is integrated with customer communications platforms, financial systems, or customer databases, where their actions have immediate impact. An unauthorised communication, an incorrect transaction, or a misapplied decision can create liability risk from contractual disputes to regulatory breaches.
What does this mean for your business if you use Agentic AI? Here are few thoughts for starters:
• Treat it with caution: it is a powerful tool, and you must recognise its potential for harm as well as good.
• Ensure there is transparency with no “black boxes.”
• Rigorously sandbox test to validate its use and application for the whole product lifecycle.
• Set clear processes, procedures, and controls to govern its autonomous decision-making.
• Balance efficiency gains against effective governance, management information, and audit.
Do all of these and you will be in a good a place. We believe that the businesses which succeed with Agentic AI will be those that combine technical capability with legal and operational discipline to manage AI autonomy.
One final, but very important point is that it will be essential for any business using Argentic AI to review its contractual terms and advice on that should be sought as an immediate priority.
Scott Moncur is legal director at Higgs
Related
#opportunity #autonomy #risk #Daily #Business #Magazine