AI · 7 min read
Generative AI in Healthcare: Use Cases, BAAs, FDA Rules and Human Review
Generative AI in healthcare in 2026: scribing, patient message drafts, prior authorisation and coding, which model vendors sign BAAs, and FDA CDS rules.
Zain Khalid MalikCTO & Co-founder, Innovation Insight·Published 
Short answer
Generative AI in healthcare works best where it drafts and a clinician or coder approves: ambient visit notes, replies to patient messages, prior authorisation letters and coding suggestions. Any feature that touches PHI needs a BAA with the model vendor; in October 2026 OpenAI, Anthropic, AWS (Bedrock) and Google Cloud offered them for specific configurations. Innovation Insight builds such features for $8k to $60k, and keeps them inside FDA's non-device clinical decision support criteria.
Where generative AI earns its place in healthcare
The useful pattern in healthcare is narrow: the model writes a first draft of something a qualified person would otherwise write from scratch, and that person reads, edits and signs it. The draft saves time; the signature keeps responsibility where regulators, payers and malpractice insurers expect it. Products that skip the human step, or that present the model's output as a diagnosis, run into HIPAA, FDA and liability problems quickly.
| Use case | What the model drafts | Who approves | Main risk to design for |
|---|---|---|---|
| Ambient clinical documentation (scribing) | A structured visit note (history, exam, assessment, plan) from the recorded conversation | The treating clinician, before the note is signed in the EHR | Hallucinated findings or medications that were never discussed; consent to record |
| Patient message replies | A reply to a portal message, using the chart and the practice's templates | Clinician or nurse, who edits and sends | Clinical advice going out unreviewed; tone; urgent symptoms buried in routine messages |
| Prior authorisation letters and appeals | A letter of medical necessity citing the chart and the payer's criteria | Clinician or authorisation specialist | Citing evidence that is not in the chart; wrong payer policy version |
| Coding assistance | Suggested ICD-10-CM and CPT codes with the note text that supports each one | Certified coder | Upcoding exposure under the False Claims Act; unsupported codes |
| Summaries for handover or referral | A short summary of a long record for the receiving clinician | Sending clinician | Omissions; summarising an outdated problem list |
| Patient education and after-visit summaries | Plain-language instructions at an agreed reading level | Clinician or nurse | Contradicting the actual care plan |
Commercial products already exist for the first two rows (Abridge, Microsoft Dragon Copilot and Nabla in ambient scribing; EHR vendors' own message-drafting tools). A custom build makes sense when you are a digital health company building your own clinical product, when your workflow is not in a mainstream EHR, or when you need the output in your own data model rather than a vendor's.
BAAs: which model vendors sign them (checked October 2026)
Under HIPAA, any vendor that receives protected health information on your behalf is a business associate and needs a business associate agreement. That includes the model API. Coverage is never "the whole vendor"; it is specific products in specific configurations. This is what each vendor's own documentation said when we checked in October 2026.
| Vendor | BAA available? | What is covered | Conditions we noted |
|---|---|---|---|
| OpenAI (API) | Yes | API services on OpenAI's HIPAA-eligible products list | Request through baa@openai.com or self-serve in the API platform once the organisation has an established usage history; reviewed case by case; once enabled HIPAA support cannot be switched off |
| Anthropic (Claude API) | Yes | The HIPAA-ready Claude API configuration with standard retention; HIPAA-ready Enterprise plans for chat, Claude Code and Cowork | The regular API, even with zero data retention, is not BAA-eligible; Claude bought through a third-party cloud falls under that cloud's BAA, not Anthropic's |
| AWS (Amazon Bedrock) | Yes, under the AWS BAA | Amazon Bedrock is on AWS's HIPAA-eligible services list, which includes Claude and other models served through it | Sign the AWS BAA first; configure logging and storage so PHI stays in eligible services |
| Google Cloud | Yes, under the Google Cloud BAA | Generative AI on Gemini Enterprise Agent Platform and Gemini Enterprise are on Google's covered services list | Use regional endpoints and resource locations for PHI, as Google's list advises |
FDA: when a generative feature becomes a medical device
The FDA's Clinical Decision Support Software guidance, finalised in January 2026 (and revised on 29 January 2026), replaced the 2022 version. Software is excluded from the device definition only if it meets all four criteria in section 520(o)(1)(E) of the FD&C Act: it does not analyse medical images, signals or patterns; it displays or analyses medical information; it supports a healthcare professional's recommendation rather than replacing it; and the professional can independently review the basis for the recommendation, so they are not relying primarily on the software.
| Feature | Likely position | Why |
|---|---|---|
| Draft visit note from a transcript, clinician edits and signs | Generally not CDS at all; documentation | It records what happened; it does not recommend care |
| Draft prior authorisation letter from chart facts | Administrative | Supports a payer request, not a treatment decision |
| Suggest differential diagnoses with the chart evidence shown | Possible non-device CDS if all four criteria are met | The clinician must be able to see and check the basis; a black-box answer weakens that |
| Analyse an ECG trace or image and flag a condition | Device | Analysing signals or images fails the first criterion |
| Patient-facing symptom triage that tells the patient what to do | Device risk is high | The criteria are written for healthcare professionals, not patients |
Commentators read the 2026 revision as more permissive on single recommendations and stricter on transparency: the less a clinician can see how a suggestion was reached, the more likely FDA treats the software as a device. For generative features that means showing the source passages next to every claim. This is general guidance, not legal advice; a regulatory consultant should review anything that recommends treatment.
Design rules for human review
- Every generated document has a status (draft, edited, approved) and an approver. Nothing reaches the EHR, the payer or the patient in draft status.
- Show the evidence. Each statement in a draft links to the transcript segment, chart entry or policy clause it came from; if there is no source, the model must not write it.
- Record the model, prompt version and temperature with every output. We did this on the AI body camera operations platform, where each Vertex AI incident review records provider, model, prompt and temperature for reproducibility and is stored beside the officer's own record without overwriting it. The same pattern suits clinical drafts.
- Measure edits. The share of drafts approved unchanged, lightly edited or rewritten is your quality metric, and it should be reviewed by a clinician monthly.
- Keep red-flag routing outside the model. Rules for urgent symptoms in patient messages (chest pain, suicidal ideation) should be deterministic and fire before any draft is produced.
- Put minimum necessary data in the prompt. Send the encounter, not the whole chart, and strip identifiers where the task does not need them.
Prior authorisation is worth a closer look
CMS's Interoperability and Prior Authorization final rule (CMS-0057-F) requires impacted payers to implement a Prior Authorization API primarily by 1 January 2027. As payers expose structured requirements through FHIR, provider-side software can match chart facts to payer criteria and draft the request automatically. The generative step is the letter and the justification; the matching should be structured logic over FHIR resources. Our EHR and FHIR integration work is where most of this budget goes.
What it costs
| Scope | Our price | Notes |
|---|---|---|
| One generative feature in an existing healthcare product (message drafts, letters, summaries) | $8k to $60k | Includes BAA-covered model setup, review screens, evaluation and audit logging |
| EHR integration needed to read or write the chart | $15k to $40k | Epic, Cerner (Oracle Health) or athenahealth through FHIR APIs |
| New regulated platform with AI features | $80k to $200k+ | Patient app, clinician portal, integrations and compliance work |
See generative AI development cost for model fees and how the build budget splits, and our healthcare app development cost guide for the wider product.
Our healthcare experience, stated plainly
We have built HIPAA-aligned healthcare software and we have built production generative AI systems, but we have not yet shipped a generative AI feature inside a clinical product. Our telehealth project for Zeuss used no generative AI; it covered patient intake, a doctor and agent CRM, pharmacy fulfilment and HIPAA-aligned safeguards such as AES-256 encryption, Azure Key Vault secrets, a complete HTTP audit trail, role-based access and PII masking in API responses. Our generative AI work includes structured extraction with Claude on AWS Bedrock for GovDoc AI and the auditable Vertex AI review pipeline mentioned above. A clinical generative feature combines both sets of practice, and we scope it that way.
If you are planning one, start with our HIPAA DevOps and testing checklist and the generative AI development page, or compare with generative AI in fintech, the other heavily regulated vertical in this series.
Sources
- OpenAI Help Center: business associate agreement for the API: https://help.openai.com/en/articles/8660679 (checked October 2026)
- Anthropic support: BAA coverage by configuration: https://support.claude.com/en/articles/15455031 (checked October 2026)
- AWS HIPAA eligible services reference: https://aws.amazon.com/compliance/hipaa-eligible-services-reference/ (checked October 2026)
- Google Cloud HIPAA compliance and covered products: https://docs.cloud.google.com/docs/security/compliance/hipaa (checked October 2026)
- FDA: Clinical Decision Support Software guidance (January 2026): https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software
- CMS Interoperability and Prior Authorization final rule (CMS-0057-F): https://www.cms.gov/priorities/burden-reduction/overview/interoperability/policies-regulations/cms-interoperability-prior-authorization-final-rule-cms-0057-f
Need a number for your project?
Send a short brief and get a written estimate.
A senior engineer replies within one business day. No sales call required.
Zain Khalid Malik
CTO & Co-founder, Innovation Insight
Zain owns architecture, engineering standards and the platform team at Innovation Insight. He sets the bar for code quality, security and the tooling every squad ships with.
LinkedInRelated questions.
Can I use ChatGPT or Claude with patient data?
Only through a configuration covered by a signed BAA. In October 2026 OpenAI offered a BAA for HIPAA-eligible API services, Anthropic for its HIPAA-ready API, and AWS and Google Cloud for Bedrock and Gemini Enterprise Agent Platform. Consumer chat apps are not covered.
Is a generative AI scribe a medical device?
Usually not, when it drafts documentation that a clinician reviews and signs. Software that analyses images or signals, or that recommends treatment without letting the clinician check the basis, can be a device under FDA's 2026 CDS guidance.
How do you stop the model inventing clinical details?
Require every statement to cite a source passage, block statements without one, keep the clinician's approval mandatory, and measure how often drafts are edited. An evaluation set of real, de-identified encounters catches regressions before release.
How much does a generative AI feature for a healthcare app cost?
With Innovation Insight, $8k to $60k for one feature in an existing product, plus $15k to $40k if it must read from or write to an EHR. Model fees are usually small next to the build.
Have you built generative AI for a healthcare client?
Not yet in a clinical product. We have built HIPAA-aligned telehealth software (Zeuss) and production generative AI systems (GovDoc AI on Bedrock), and we say so plainly when scoping.