Generative AI in Healthcare & HIPAA: PHI Risks, BAAs

Key Takeaways

• HIPAA applies based on the parties, their roles, the information, and the use, not on the technology. Start by mapping PHI flows: input, processing, storage, logs, and output.
• A vendor that creates, receives, maintains, or transmits PHI on your behalf is generally a business associate. Not every vendor in an AI workflow has that status, and the contract defines what each may do with PHI.
• Minimum necessary is policy-driven and has exceptions. It is neither a universal ban on broad context nor irrelevant to treatment-related uses.
• Redaction, tokenization, pseudonymization, and synthetic data are not automatically HIPAA de-identification. The Privacy Rule recognizes two methods: Safe Harbor and Expert Determination.
• In retrieval-based assistants, permissions must be enforced before content reaches the model. Restricting the model alone is not enough.
• Human review of clinical output is an important safety control. Treat it as a clinical governance decision, not a quoted HIPAA mandate.
• A checklist documents due diligence. It does not establish compliance.

HIPAA doesn’t ban generative AI, and it doesn’t treat it as a special category. Its obligations depend on who is involved, what role they play, what information is handled, and the circumstances of the use or disclosure. The practical work is mapping each AI workflow to those facts. This guide covers where PHI is exposed in three common workflows (clinical documentation, patient chatbots, and internal AI search), how the Privacy and Security Rules apply, when a vendor is a business associate, and what to verify before approving a tool. It is general information, not legal advice. Involve privacy counsel for your specific facts.

What Generative AI Looks Like in Healthcare

Generative AI refers to models, most commonly large language models (LLMs), that produce new text, images, audio, or structured data rather than classifying existing data. In healthcare it shows up as:

• Clinical documentation tools that turn patient-provider conversations into draft notes (ambient AI scribes)
• Clinical decision support copilots that summarize charts or draft options for clinician review
• Patient-facing chatbots for scheduling, intake, billing, and portal questions
• Administrative and revenue-cycle AI that drafts prior authorization letters, appeals, or correspondence
• Internal knowledge assistants that search policies, manuals, and sometimes patient records
• Research and analytics tools that summarize datasets or generate synthetic data

Generative AI often needs broad, unstructured access to patient data to work well, and its outputs are probabilistic. Two identical prompts can yield different phrasing, and errors look like fluent, confident prose. That combination of deep data access and unpredictable output is what makes these tools harder to assess than a conventional EHR module.

When HIPAA Applies to a Generative AI Tool

HIPAA’s Privacy, Security, and Breach Notification Rules apply to covered entities (health plans, clearinghouses, and health care providers that conduct certain electronic transactions) and, in specified ways, to their business associates. “Does HIPAA apply to this tool?” breaks into three questions:

1. Is the organization using it a covered entity or business associate?
2. Is the information PHI?
3. Is the use or disclosure happening on the covered entity’s behalf?

A clinic deploying an ambient scribe on patient encounters answers yes to all three. A patient pasting their own lab results into a consumer chatbot is not a covered entity acting on that data, though other laws and the vendor’s own terms may still matter. The same workflow can also implicate state health-privacy laws, 42 CFR Part 2 for substance use disorder records, and separate clinical-safety or device-regulation questions for decision-support tools. This guide focuses on HIPAA.

Is the AI Vendor a Business Associate?

HHS describes a business associate as a person or entity that performs functions or activities on behalf of a covered entity, or provides certain services to it, involving PHI. A subcontractor that creates, receives, maintains, or transmits PHI for a business associate is also a business associate. HHS’s own examples include a third-party AI chatbot on a provider’s patient portal handling things like symptom assessment, reminders, and appointment scheduling. They also include transcription services and cloud providers that process or store electronic PHI (see OCR’s cloud computing guidance).

In practice:

Role follows function. An AI vendor processing PHI for you is likely a business associate. A vendor that receives only data de-identified under HIPAA’s standard isn’t handling PHI, although the de-identification step itself may be business associate activity.

Chains matter. A model provider behind an app vendor may be a subcontractor business associate if it handles PHI for that vendor. HHS notes a covered entity needn’t have a BAA directly with its business associate’s subcontractor, but the business associate must have one before disclosing PHI to it.

Exceptions exist. HHS lists situations where no BAA is required, such as disclosures to another provider for treatment and entities acting as a pure conduit. The conduit exception is narrow: entities that access PHI regularly to perform a service are not conduits.

The contract sets the boundaries. A BAA must describe permitted uses and disclosures, require safeguards, and require reporting of unauthorized uses and breaches, among other elements. HHS publishes sample provisions, which are not required language. Whether a vendor may use PHI for product improvement, or to de-identify it, depends on what the BAA permits.

A BAA is necessary where required, not sufficient. It says nothing about whether the product is secure or accurate.

HIPAA Privacy Rule and Generative AI

The Privacy Rule governs whether and how PHI may be used or disclosed.

Minimum Necessary

HHS’s guidance says covered entities must take reasonable steps to limit uses, disclosures, and requests of PHI to the minimum necessary for the purpose. The standard is policy-driven. For uses, an entity identifies who needs access to which categories of PHI, and may allow access to the entire record where that is necessary and the policy says so and justifies it. HHS lists exceptions, including disclosures to or requests by a provider for treatment, disclosures to the individual, uses under an authorization, and uses or disclosures required by law.

So an AI tool receiving a full longitudinal record isn’t automatically a violation, and treatment context doesn’t automatically excuse broad access. Ask which of your policies govern this use, whether the AI’s data scope matches them, and whether any broader scope is justified and documented. HHS’s sample BAA provisions include minimum necessary language, so vendor contracts commonly carry it forward.

Permitted Uses and Vendor Training

Uses for treatment, payment, and health care operations are generally permitted, but a vendor’s own purposes (training shared models, product analytics) are not automatically within them. Check what the BAA allows and which Privacy Rule pathway, such as authorization or de-identification, would support any secondary use.

HIPAA Security Rule and Generative AI

The Security Rule sets administrative, physical, and technical safeguards for electronic PHI. It is flexible and risk-based.

• Risk analysis. A new AI workflow that moves ePHI into new systems (a speech-to-text service, a vector database, a model provider) should be reflected in your risk analysis and risk management. Generic assessments that predate the deployment are unlikely to describe it.
• Audit controls. The rule requires mechanisms to record and examine activity in systems containing ePHI. It does not prescribe prompt-level or output-level logging. How much content to log is a design decision. More detail helps incident investigation, but logs that contain prompts and outputs are themselves PHI-bearing and need access controls and retention limits.
• Encryption. It is an addressable specification, meaning you must implement it or document an equivalent, risk-based alternative. Look for encryption aligned with recognized standards and documented key management rather than a single mandated algorithm.
• Cloud and vendors. You can’t delegate accountability. Vendor due diligence is how you decide whether their controls are adequate for your risk.
• Reassessment. HIPAA doesn’t set an AI-specific reassessment schedule. Set a cadence in policy and trigger reviews when the model, vendor, data flow, or use changes materially.
• Breach reporting. A business associate must report breaches of unsecured PHI to the covered entity. The Breach Notification Rule requires notification without unreasonable delay and no later than 60 days after discovery. Contracts often set shorter periods and assign who notifies individuals and OCR.

PHI Exposure in Three Healthcare AI Workflows

The following are illustrative examples, not customer case studies. Architectures vary widely between products, so use these as prompts for your own data-flow mapping.

Why restricting the model isn’t enough. A model can only protect what it is never given. If the retrieval layer or index returns restricted passages to a user whose role shouldn’t see them, instructing the model “don’t reveal confidential information” is not a reliable control. Enforce permissions at retrieval and at output. Also treat indexed documents as untrusted input, since text planted in a document can attempt to steer the model (see prompt injection).

Risks That Cut Across Workflows

• Shadow AI. Staff pasting PHI into consumer tools outside any contract. Publish an approved-tools list and make it easy to find. Most shadow use happens because staff don’t know a compliant option exists. See our shadow AI guide.
• Unintended retention or reuse. Submitted data kept by a vendor or used to improve shared models without contractual permission.
• Prompt injection and cross-user exposure. Malicious or accidental content causing a system to reveal data across users or sessions.
• Hallucination. Fabricated clinical or administrative details entering records or patient communications.
• Undisclosed subprocessing. Data routed through providers you haven’t assessed.
• Overstated de-identification. Data labeled “anonymized” that hasn’t met a HIPAA standard (below).

De-Identification and AI: What HIPAA Recognizes

HHS’s de-identification guidance describes two methods under the Privacy Rule:

• Safe Harbor removes 18 categories of identifiers (names, small geographic units, date elements other than year, contact numbers, record numbers, and others), and the entity must have no actual knowledge that the remaining information could identify someone.
• Expert Determination relies on a qualified expert who determines the risk of identification is very small and documents the methods and results.

Properly de-identified data is no longer PHI. HHS also notes that both methods leave a small, non-zero risk.

Several techniques that teams often equate with de-identification do not automatically qualify:

• Redaction that misses identifiers in free text. HHS says Safe Harbor identifiers must be removed wherever they appear, including narrative notes, and narrative text can carry context that identifies someone.
• Pseudonymization and tokenization. Replacing identifiers with reversible tokens is not the same as de-identifying. HHS says disclosing a code designed to allow re-identification is itself a disclosure of PHI, and codes derived from the individual’s information, or from hashes without a secret key, can be identifying.
• Synthetic data. Whether synthetic data is de-identified depends on how it was generated and what it could reveal about source individuals. It does not qualify just because it is synthetic. Evaluate it, ideally with an expert.

HHS says an expert determination has no required expiration date, but experts may time-limit their conclusions. Revisit it when the data, recipient, or available outside data change. A business associate may de-identify PHI on a covered entity’s behalf only to the extent its BAA authorizes.

HIPAA Security Rule and Generative AI

Where the Privacy Rule governs whether PHI can be used, the Security Rule governs how it must be protected technically, physically, and administratively.

Administrative Safeguards

A current, AI-specific risk analysis is non-negotiable. Generic risk assessments that predate your AI deployment do not satisfy this requirement — OCR has been explicit that risk analyses must reflect the actual systems in use, including third-party AI tools.

Technical Safeguards

Encryption in transit and at rest, audit controls, and access management all apply to AI systems exactly as they apply to any other system handling PHI — but implementation looks different. Audit logging for an LLM needs to capture not just "who accessed the record" but "what was the model shown, and what did it generate." Many off-the-shelf AI products don't log at this granularity by default, which is a gap compliance teams need to close contractually before go-live.

Physical Safeguards

Cloud-hosted AI models shift physical safeguard responsibility largely to the vendor, but covered entities remain accountable for verifying that vendor infrastructure meets HIPAA's physical security expectations — this belongs in vendor due diligence, not assumed by default.

For a deeper technical walkthrough, see our related article on HIPAA Security Rule requirements explained.

Key Risks of Generative AI in Healthcare Settings

• Training data leakage — PHI inadvertently memorized by a model and reproduced in unrelated outputs
• Prompt injection and jailbreaks — malicious inputs designed to extract sensitive context from a chatbot's memory
• Shadow AI usage — clinicians or staff pasting patient data into consumer-grade tools that carry no BAA
• Model hallucination in clinical contexts — fabricated but plausible-sounding clinical details entering a record
• Vendor sub-processing — AI companies routing data through additional third-party model providers without disclosure
• Insufficient de-identification — datasets marketed as "anonymized" that still carry re-identification risk under the Safe Harbor or Expert Determination standards

Common HIPAA Violations Involving AI Tools

In practice, most AI-related HIPAA violations trace back to a small number of recurring patterns:

• No BAA in place before PHI touches a third-party AI tool — often because a pilot moved faster than procurement.
• Over-collection, where an AI feature is given database-wide access instead of scoped, task-specific access.
• Inadequate audit trails, making it impossible to reconstruct what a model saw or generated after an incident.
• Consumer AI tools used informally by clinical or administrative staff for drafting, summarizing, or coding — a fast-growing and under-monitored risk category.
• Weak de-identification claims, where data labeled "anonymized" for AI training still contains quasi-identifiers.
• Unreviewed AI-generated content entering the medical record without a human verification step, creating both a compliance and a patient-safety issue.

How to Evaluate Healthcare AI Vendors

Compliance, security, privacy, legal, and IT teams should separate three kinds of evidence. Marketing language and certifications don’t substitute for any of them.

• Contractual commitments are what the vendor is bound to do.
• Technical capabilities are what the product can do and that you can test.
• Independent assurance is third-party evidence, such as audit reports, scoped to what was actually assessed.

A security certification or attestation speaks to the controls within its stated scope. It doesn’t show that a particular deployment, contract, or use meets HIPAA. For a broader framework, see our guide to evaluating enterprise AI vendors.

Governing Healthcare AI After Deployment

A durable program usually rests on four practices:

1. Cross-functional oversight. Compliance, security, clinical leadership, and legal review PHI-touching tools before deployment. See why AI governance is a security priority.
2. A living inventory. Every generative AI tool in use, its data scope, BAA status, and risk tier. Expect surprises the first time.
3. Integration with existing HIPAA policies. Extend your current risk analysis, access, and incident response procedures rather than creating a parallel track.
4. Monitoring tied to change. Review logs, test for prompt injection, and re-review when the model, vendor, data flow, or use changes.

Where Sensitive-Data Detection and Anonymization Fit

Some exposure is avoidable. A prompt summarizing call-center themes doesn’t need a caller’s name, phone number, or medical record number. Detecting sensitive identifiers and removing or masking them before text reaches an AI model reduces what leaves your controlled environment and what ends up in prompts, logs, and provider systems.

Questa AI’s healthcare approach places an anonymization layer between source data and the LLM. Per its product documentation, it combines named-entity recognition and PII detection and can run on-premises (Blackbox) or in a dedicated cloud environment. For a technical look, see how a privacy-first anonymizer for LLMs works and how AI privacy firewalls prevent sensitive data leakage.

Real-World Use Cases

Ambient clinical documentation. A multi-site clinic group deployed an AI scribe to reduce charting time. The rollout succeeded compliance review only after the vendor agreed to on-device audio processing, a signed BAA, and configurable data retention limits — a useful template for similar deployments.

Payer correspondence drafting. A regional insurer uses generative AI to draft appeal and denial letters from claims data. Because the tool operates on a segmented dataset with role-based access, PHI exposure is limited to the minimum fields required for each letter type.

Research cohort generation. An academic medical center generates synthetic patient cohorts for early-stage research using de-identified source data, avoiding IRB and Privacy Rule complications that direct PHI use would trigger. Our article on data minimization strategies in healthcare AI explores this pattern in more depth.

Patient chatbot triage. A hospital system's intake chatbot was restricted to non-diagnostic, administrative questions only, after early testing showed it would otherwise attempt to interpret symptoms — a scope decision that reduced both clinical and compliance risk simultaneously.

Best Practices for HIPAA-Compliant Generative AI

Scope access before scale. Give AI tools the narrowest data access that still lets them do their job, and expand only with documented justification.

Treat de-identification as an engineering discipline, not a checkbox. Techniques like tokenization, redaction, and synthetic data generation each carry different re-identification risk profiles, and the right choice depends on the use case. Solutions such as a Privacy Firewall that intercepts and redacts PHI in real time before it reaches a model can meaningfully reduce exposure at the point of ingestion, rather than relying solely on downstream controls.

Log model inputs and outputs, not just access events. Traditional access logging tells you who touched a record; AI-specific logging needs to tell you what the model was shown and what it produced.

Require human review for anything entering the clinical record. AI-generated content should be a draft, not a final artifact, until a clinician signs off.

Build a fallback for shadow AI. Publish an approved tools list and make it easy to find — most shadow AI use happens because staff didn't know a compliant alternative existed, not because they ignored policy. See our related guidance on incident response planning for healthcare breaches for how to handle the cases that slip through anyway.

Re-verify de-identification periodically. Data considered de-identified today can become re-identifiable as more auxiliary datasets become publicly — de-identification is not a one-time event.

Regulatory Status

HHS proposed a significant update to the Security Rule in January 2025. Check HHS’s Security Rule NPRM page for current status, because proposed requirements aren’t binding until a final rule is published. Meanwhile the current rules apply, and state-level AI and health-privacy laws may add obligations. Governance work you do now (inventories, vendor diligence, data-flow mapping) is useful regardless of how those rules evolve.

Frequently Asked Questions

Conclusion

HIPAA already provides the framework for generative AI in healthcare. The work is applying it precisely: who is the covered entity or business associate, which data flows carry PHI, what the contracts permit, and which controls sit at each step. Map the workflow before approving the tool, enforce permissions at the data layer rather than relying on prompts, treat de-identification as a defined method rather than a label, and keep clinicians in the loop on clinical output. Organizations that document these decisions will be better placed as vendors, models, and regulations change.