Part 2 of 3 in our series AI Agents in Design.

If an AI agent makes a decision, who is responsible for it? Patrick Henz is a business ethicist, futurist and author of Tomorrow’s Business Ethics Revisited, and Head of Environmental, Society, Governance, Risk & Compliance at Primetals Technologies, a Mitsubishi Heavy Industries company. His answer is clear: companies and people stay accountable, so they need to design for that from day one.

“Advanced system-thinking on all levels is necessary to ensure that the AI is the beta, while the human stays the alpha.”

How do you view the role of AI agents in shaping ethical decision-making? Can they be designed to act as ethical guardians or moderators in digital ecosystems?

An AI agent’s database can be built with regulations and guidelines, and documents on ethical decision-making can be included too. Because ethics vary from region to region, the agent must be adapted to its use case and region. This can be done by managing the database, together with coaching the agent. Trained by an expert, AI systems can detect red flags, for example in payments or communication.

How should governance, risk and compliance frameworks be built into the design and deployment of AI agents, especially those that interact with users autonomously?

AI systems must consider the human. Companies and employees are responsible for the decisions an AI agent makes, but they cannot control it 100%. A risk assessment is mandatory, to understand what risks may arise if its behaviour deviated from ethics and law.

Processes and controls must keep the database properly maintained, and stress tests should show how the agent would react in extreme cases. The internal owner of the AI system must explain its purpose, and a manual backup process must exist in case the agent has to be turned off, even temporarily.

AI agents are becoming not just tools but “companions”. What philosophical challenges does this raise for trust, agency and accountability in design?

It will be a new development for human societies. Humans may build trusting relationships with AI agents, much as they do today with their pets. People may read attitudes, ethics and knowledge into an agent from its avatar, physical appearance or behaviour. Such perceptions may be misleading, and can lead to dependence, isolation and tragic outcomes.

How can AI agents contribute positively to long-term sustainability goals? What design considerations make AI socially and environmentally responsible by default?

AI agents must meet the requirements of trustworthy AI systems, which include consideration of humans, society and the environment. Advanced system-thinking on all levels is necessary to ensure that the AI is the beta, while the human stays the alpha.

How can organisations anticipate and reduce the new risks AI agents bring, such as bias, opacity, manipulation or unintended autonomy?

Organisations should put processes and rules in place now, while AI systems are still rolled out on a small scale. They must understand how vulnerable humans are when it comes to controlling AI. Processes and training must be human-centric, to protect and empower employees and prepare them to spend more of their time working with AI. Depending on the scale and type of AI systems they use, organisations should consider appointing a dedicated AI Officer and/or an AI Ethics and Compliance Officer.

Previously: Kumar Gaurav on designing for intent. Next: Carlo Edoardo Ferraris on technology that fits human nature.