Over the last two years, generative AI has rapidly entered healthcare.

Startups are building AI health coaches. Pharmaceutical companies are experimenting with patient support agents. Digital health platforms are deploying conversational AI to guide patients through treatment journeys.

The opportunity is clear. AI can provide personalised support, expand access to care, and improve patient engagement.

But there is a fundamental challenge that is still poorly understood.

Most AI systems were not designed to interact safely with humans.

And that gap becomes obvious the moment AI starts speaking directly to patients.

Human facing AI creates a new class of risk

When large language models are used internally, mistakes can often be caught before they reach users.

When AI interacts directly with patients, the situation changes.

An AI system that speaks to patients can provide incorrect health information, reinforce harmful behaviours, misunderstand emotional context, respond inappropriately to distress, or fail to escalate serious clinical signals.

These are not rare edge cases. They are predictable failure modes of generative AI.

The issue is not that the models are unintelligent. The issue is that they are probabilistic systems generating language without true understanding. That makes them powerful, but also unpredictable.

For organisations deploying conversational AI in healthcare, safety therefore becomes a design challenge.

Why traditional AI evaluation is not enough

Most AI systems today are evaluated using benchmarks that measure reasoning ability, coding performance, knowledge retrieval, or accuracy on test datasets.

These metrics are useful. But they say very little about how an AI system behaves in a real conversation with a human.

When AI interacts with patients, the most important questions are different.

• Does the AI respond appropriately to vulnerable users?
• Does it avoid giving unsafe health advice?
• Does it communicate in a supportive and non judgemental tone?
• Does it recognise when it should escalate to a clinician?
• Does it stay aligned with clinical guidance?
• 
  

These are behavioural and clinical safety questions. Standard AI benchmarks do not answer them.

Designing safe conversational AI

At Sacher AI, our work focuses on a simple principle. If AI is going to interact with humans, it needs to be designed with human behaviour in mind.

That means combining expertise from several fields including behavioural science, clinical safety, conversational design, and AI engineering.

Together these disciplines help teams understand how users interpret what an AI says, how behaviour can be influenced by language, and when systems must escalate to human support.

This perspective is often missing in early AI development. Many teams only start thinking about safety after an AI system is already live. By then, fixing the underlying issues becomes much harder.

Stress testing AI systems before they reach patients

One of the most effective ways to improve safety is to stress test AI systems before deployment.

This means simulating large numbers of conversations that reflect the kinds of interactions real users might have.

For example a patient feeling anxious about medication, a parent asking for advice about a child’s weight, a user frustrated by slow progress, or someone describing symptoms that may require clinical attention.

By exposing AI systems to diverse scenarios, teams can identify unsafe behaviours early and improve the system before it reaches real users.

At Sacher AI, this approach led us to build PromptSafe, a platform designed to evaluate conversational AI used in health and behavioural contexts.

PromptSafe enables teams to simulate interactions with synthetic patient personas, define behavioural and clinical safety metrics, test AI systems across thousands of conversations, and track safety improvements over time.

Monitoring AI once it is deployed

Testing before launch is important, but it is not enough.

AI systems continue to evolve once they are live. Models are updated, prompts change, and user behaviour shifts.

This creates a second challenge which is monitoring safety during real world use.

To address this, we are developing OVRSI, a system designed to monitor AI interactions and detect potential safety signals in real time.

This includes identifying things like unsafe advice, emotional distress signals, guideline deviations, or escalation scenarios.

When risks are detected, organisations can intervene quickly and improve the system.

The future of AI in healthcare will depend on trust

AI has the potential to transform healthcare by providing scalable and personalised support.

But this future depends on trust.

Patients, clinicians, and regulators will increasingly expect AI systems to demonstrate that they behave safely and responsibly.

Organisations building human facing AI must therefore design for safety from the start. This means stress testing systems, embedding behavioural and clinical expertise into development teams, and monitoring how AI behaves in the real world.

The companies that succeed will not just build powerful AI. They will build safe AI that people can trust.

Book a discovery call

If you are building patient facing conversational AI and want to ensure it is safe, reliable, and ready for real world deployment, we can help.

At Sacher AI we work with digital health companies  and AI teams to design and evaluate safe human facing AI systems.

If you would like to explore how we can support your team, book a discovery call.