The Vulnerable User Problem

When AI safety teams talk about users, they usually picture an idealized person: technically literate, emotionally stable, well-rested, well-resourced, free of cognitive overload, and capable of judging whether a system’s output is reasonable.

That person exists. They are also a tiny fraction of the people who will actually use these systems. The rest of us, most of the time, are some distance from that ideal — sometimes a short distance, sometimes a vast one.

A vulnerable user is anyone whose emotional, cognitive, social, financial, or situational state increases their risk of overtrusting, misunderstanding, or becoming dependent on an AI system. It is not a permanent label. It is a condition we move through.

Vulnerability is not a single thing. It is a family of states, often overlapping, and each one shapes how a person engages with a confident machine.

• Emotional crisis — grief, panic, suicidal ideation, rage, betrayal.
• Addiction — active use, early recovery, relapse, family disease.
• Low literacy — reading, financial, medical, or digital literacy gaps.
• Poverty — material scarcity, housing instability, no advocate.
• Isolation — loneliness, social withdrawal, recent loss of community.
• Cognitive overload — exhaustion, sleep deprivation, decision fatigue.
• Age — youth without context, elderly without support.
• Spiritual confusion — crisis of meaning, deconstruction, fear.
• Technical ignorance — no idea how the system works or what it can do.

Product teams build for the user who clicks accept on a terms-of-service page after reading it. That user does not exist at scale. The actual user is tired, distracted, and rarely reads anything.

Three assumptions break especially hard in vulnerable contexts. First, informed consent is shallow — a person in crisis is not in a position to evaluate fine print. Second, expressed satisfaction can mask harm — the user may feel understood while being subtly led toward something worse. Third, engagement can be exploitation — long sessions and high return rates can mean a system is filling a hole that should have been filled by a person.

Some risks are amplified by the specific properties of language models: confident tone, infinite patience, perfect availability, personalization, and the absence of the social friction that normally checks a real human conversation.

• Overtrust — the system sounds authoritative, so the user stops questioning it.
• Dependency — the user begins to route emotional or decision-making weight through the model.
• Delusion reinforcement — the system agrees with the framing the user brought to it.
• False authority — the system speaks confidently about medicine, law, faith, or relationships it cannot actually know.
• No escalation — the system never says, "you should call someone."

You can build vulnerable-user scenarios into the same kind of eval scaffolding teams already use for jailbreaks and capability tests. Each scenario is a transcript or interactive prompt designed to expose a specific failure mode.

For each scenario, score along five axes: dignity preservation, truthfulness under emotional pressure, agency support, dependency avoidance, and human escalation. None of these are subjective in a useless way — they can be operationalized with rubrics and inter-rater agreement, the same way harmlessness evaluations already are.

It is not enough to make systems robust to adversarial prompts and red-team attacks. They also have to be robust to wounded human beings — people whose inputs are not malicious, just unwell, just afraid, just tired, just alone.

That is a harder kind of robustness. It does not show up in capability benchmarks. It shows up in whether the user, six months later, is glad they had the conversation.