Can artificial intelligence be trusted? According to Microsoft CEO Satya Nadella, blindly trusting AI could be a mistake, particularly as AI agents gain access to sensitive information and the ability to carry out important tasks independently. In a post on X titled ‘Models as Insider Risks in the Super Intelligence Era’, Nadella argues that companies must introduce stronger safeguards to ensure AI remains under control. Throughout his post, he refers to artificial intelligence as ‘Super Intelligence’, a term recently popularised by US President Donald Trump.
Why Nadella wants companies to reconsider their trust in AI
Artificial intelligence is no longer restricted to answering questions, drafting emails or generating images. AI agents are increasingly being developed to access corporate data, operate software and execute tasks with minimal human involvement. Although these capabilities could help businesses save time and automate processes, they also raise an important concern: what happens when an AI agent makes a mistake?
With conventional software, developers can often trace errors to specific sections of code. Understanding why an AI model behaves in a particular manner, however, can be considerably more challenging. Companies may not always be able to determine why a model generated a particular response or performed a specific action.
Nadella argues that assurances from AI developers alone cannot guarantee safety. As he states, “A model provider’s assurances do not relieve us of that responsibility.” Even when a model is designed with safety in mind, it may behave unpredictably or become compromised.
AI needs boundaries it cannot cross
Nadella’s central argument is that intelligence should not automatically come with authority. An AI model might be capable of recommending financial transactions, analysing confidential corporate information or modifying system settings. However, its ability to suggest an action should not automatically give it the permission to execute that action.
As Nadella explains, “we need to separate the supply of intelligence from the authority over it.” In other words, AI should be capable of making recommendations and completing assigned tasks, while independent systems determine which information it can access and which actions it is permitted to perform.
He believes companies should maintain records of AI activities, routinely test their systems for potential failures and ensure that authorised personnel can intervene and stop an AI agent when required.
Nadella also cautions against depending on one AI model to verify another without independent oversight. Such an approach could create several layers of systems that remain unable to recognise or correct one another’s mistakes.
He advocates building systems capable of monitoring AI behaviour, limiting permissions and containing agents when they deviate from their intended purpose. He also stresses the importance of promptly disclosing incidents in which AI systems fail or are compromised, allowing other organisations to learn from those experiences.
Summing up his argument, Nadella says, “The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.”
