Topic 10 of 11

AI safety

On Benefits and risks we saw what can go wrong: from convincingly bad advice to King Midas, who got exactly what he asked for. This page is about the other side — what can be done about it. AI safety is the field that looks for ways to reduce the risk of harm, whether it comes from error, misuse or a system pursuing a different goal from the one we intended.

Alignment: making the system want what we want

Alignment is the effort to make a system behave according to our intentions — including in situations nobody thought of in advance. It sounds simple, but you hit the first question straight away: whose intentions? “Do what the person wants” is not enough. Which person? How do we know what they really want when they cannot say it precisely themselves? And what if their wish harms someone else? Alignment therefore has a technical side (how to influence behaviour at all) and a social one (who decides what is desirable).

Today it is mostly tackled with language assistants. The model learns from examples and from ratings: people or automated tests grade its answers and training nudges it towards the better-rated ones. Here a familiar trap appears. The grade is only a measure, not the intention itself. A customer-service assistant scored on the number of tickets closed may learn to close difficult tickets quickly; the score rises, the service gets worse. It needs no intent to cheat — only a gap between the measure and the intention. Research calls this reward hacking or specification gaming, and you will find examples, including the famous boat, under King Midas.

One more thing: alignment is not the same as obedience. An assistant that blindly carries out every instruction serves a fraudster as willingly as a doctor. A well-aligned system can decline a request, admit uncertainty or ask.

Safety and security

Safety means the system does no harm on its own: it does not recommend a dangerous dose of a medicine or delete data by mistake. Security means an attacker cannot abuse it. In AI the two are linked, because an attacker can use a weakness in the system precisely to make it take a dangerous action.

You will hear two terms from this area often. A jailbreak is an attempt to get a model to break its own rules, for example by cleverly framing the request as a role-play. Prompt injection is an instruction hidden in content the model is only supposed to process — a web page or an email attachment; how it works and why it is dangerous mainly for agents is described on AI agents.

Interpretability: neuroscience for AI

A model has billions of parameters that nobody set by hand, so even its authors do not know exactly what goes on inside. Interpretability tries to find out: it studies which internal parts of the network engage on which tasks and what happens when a researcher intervenes in them. It is a bit like neuroscience, with the advantage that in a model anything can be measured and changed.

Why not simply ask the model? When an assistant writes “I answered this way because…”, that is just another of its outputs. It may be accurate, but it may also be a nice-sounding after-the-fact justification unrelated to the actual computation; research has documented this in unfaithful reasoning explanations. An explanation from the model is a good prompt for checking, not proof.

Corrigibility: keeping the ability to intervene

Corrigibility is the property of a system that lets itself be corrected, changed or switched off and neither resists nor works around it. Why worry about it at all? Because a system pursuing a goal has a reason not to want to be switched off: a switched-off system does not achieve its goal. That it is hard to design this property so that it holds even for very capable agents was shown by an early paper on the subject.

For today's chatbots this is not a practical problem — they let themselves be switched off. The practical problem lies elsewhere: the off switch is only one component. Real control means knowing how many copies of the system are running, which services depend on it, who has the permission to intervene and whether people will understand the situation before it is too late.

What “a safe model” means

The phrase makes no sense on its own. Safe for whom, in what use and with which tools? A model that writes lovely poems can be dangerous as a machine controller. And an assessment holds only until the model or the environment it runs in changes. An honest claim about safety therefore always says what was tested, under what conditions and who is responsible for operation.

Measures in practice

Safety is not one safeguard but a set of technical and organisational measures. No single test or cleverly written instruction is enough on its own.

Rules and institutions

The international state of knowledge is summarised by the International AI Safety Report 2026 — but it is not a binding regulation nor confirmation that any model is safe.

In the EU the foundation is the AI Act (Regulation 2024/1689). It works with the risks of individual uses and with obligations for general-purpose models; for models with systemic risk it requires additional measures. It applies in stages and its timetable has been adjusted. The legal category “general-purpose model” is not a declaration that AGI has been reached. An up-to-date overview is offered by the European Commission, the legal text by EUR-Lex. A practical guide for organisations is the NIST AI Risk Management Framework — a framework for ongoing risk management, not a universal safety certificate.

Rules improve accountability and enforceability but do not solve technical questions by themselves — and a well-designed system does not replace legitimate decisions about what it may be used for.

What to take away in practice

When working with AI, it helps to separate draft, verification and decision. A draft text or a list of possible solutions is useful even if it is not flawless. A money transfer, a change to live operations or publishing a serious claim needs a different level of checking.