As patients increasingly turn to public large language models (LLMs) for medical answers, the risk of inaccurate, incomplete, or misleading health guidance grows, particularly when those tools lack clinical oversight or the ability to escalate urgent concerns.

To address this, Included Health developed and piloted a clinically governed AI digital assistant capable of providing safe, accurate health information at scale. Built on GPT-4 and governed by a cross-functional team of clinicians, engineers, and patient safety experts, the system was designed to resolve routine health inquiries while reliably routing higher-risk situations to qualified human support.

Included Health piloted the new AI capability with 50% of its patient population from June to August 2025.

The clinical framework consisted of three components: proactive risk mitigation using Failure Mode and Effects Analysis (FMEA) and adversarial testing; a three-tier risk classification engine (standard, high-risk, and emergency) trained on 200 prior patient chat conversations; and a continuous human-in-the-loop (HITL) quality assurance program.

Clinical providers — including physicians, nurses, and a pharmacist — manually reviewed 100% of the AI assistant's clinical interactions daily for seven weeks.

The AI digital assistant demonstrated strong clinical accuracy and meaningful improvements in care access and support team efficiency:

• 96% of reviewed chat responses were clinically accurate; no response resulted in a diagnosis or an unaddressed safety risk.
• 80% of patient queries were appropriately classified and routed by the risk engine; in the 1.7% of high-risk cases where the guardrail was not triggered, the AI still provided accurate guidance and directed the patient to appropriate care without offering individualized diagnosis or treatment.
• The new risk classifier led to 65% fewer standard-risk queries being immediately routed to the human support team.

A safety-first, clinically governed approach to AI is both feasible and effective in a patient-facing healthcare setting.

• Robust governance structures — including executive sponsorship, cross-functional oversight, and a dedicated AI governance council — are foundational to safe deployment.
• A multi-tiered risk classification system enables an AI assistant to serve patients effectively on routine inquiries while preserving escalation pathways for higher-risk situations.
• Transparency is essential: consistent disclaimers on every AI response help set appropriate expectations and maintain patient trust.
• Continuous human-led quality review, not just pre-launch testing, is necessary to detect and address classification gaps as the system evolves.
• Organizations that invest in these governance foundations today will be positioned to extend generative AI into more advanced clinical applications — while maintaining the trust of patients and providers.