Why AI Security in Healthcare and Finance Can’t Wait

Key Takeaways

• AI expands the attack surface beyond traditional IT. It now includes models, prompts, training data, retrieval indexes, and AI agents.
• Hospitals must secure both the AI they deploy and the systems it connects to: EHRs, imaging platforms, and connected medical devices.
• Banks and insurers face a different risk profile: model manipulation, AI-enabled fraud, third-party AI provider concentration, and model risk governance. Protecting sensitive data before it reaches a model and securing the model itself are two different jobs. Neither replaces the other.
• Shadow AI is one of the most common and least visible exposures in both sectors, and it is usually a workflow problem before it is a discipline problem.
• Compliance frameworks give structure but do not make an AI system secure on their own.

What Is AI Security in Healthcare and Finance?

AI security is the combination of cybersecurity, privacy, access control, model protection, data protection, monitoring, and governance used to protect AI systems, the sensitive data they handle, and the workflows built around them from unauthorized access, manipulation, misuse, and disruption.

The term covers two distinct ideas, and mixing them up is where many organizations go wrong. The first is securing AI systems: the models, prompts, training data, APIs, and agents that now sit inside clinical and financial workflows. The second is using AI to strengthen cybersecurity: machine learning for intrusion detection, anomaly detection, phishing detection, vulnerability prioritization, and faster incident response.

An organization can be strong at one and weak at the other. A hospital can run excellent AI-powered threat detection and still operate an LLM that leaks patient details through poorly filtered outputs. AI-powered detection does not protect the AI itself, and a hardened model does not detect a ransomware intrusion. A mature program needs both. In practice, AI helps analysts surface signal in noisy data and respond faster, but it is poor at judging clinical or legal consequences, so the organizations getting value from it use it to make analysts faster, not to remove them from the loop.

AI Security in Healthcare: What Needs Protecting

Hospital AI is not one system. It is a set of overlapping assets inside clinical and operational workflows, and a failure in any of them is a patient-safety, privacy, or availability problem as well as an IT problem. Five areas deserve specific attention.

Clinical AI assistants and medical documentation

Ambient scribes, documentation assistants, and patient-communication tools process some of the most sensitive text an organization holds: symptoms, diagnoses, medications, and family details, often as free-form speech or notes. The security questions are concrete. Where is the audio or text processed and stored? Is it retained, and for how long? Is it used to train or improve the vendor’s models? Who inside the organization can retrieve a past transcript? Can a clinician’s draft note be reviewed and corrected before it enters the record? A documentation assistant that cannot answer these questions should not be near a production chart.

Hallucination is also a security-adjacent risk here. A fabricated medication or an invented history in a draft note is an integrity problem, which is why clinician review before anything is written to the record should be a hard requirement rather than a default setting.

EHR integrations and patient-data access

The moment an AI tool is connected to an EHR, it inherits the hospital’s access problem. Common failure patterns include a service account with read access to every chart when the task needs one, retrieval-augmented generation (RAG) indexes built from patient records without the source system’s permissions carried over, and embeddings that persist after the underlying record has been corrected or access has been revoked. If a retrieval layer returns a document the requesting user could not open in the EHR itself, the AI has become a side door.

Practical controls: scope AI access to the minimum necessary data for the specific task, enforce the source system’s permissions at retrieval time rather than at indexing time, log every record an AI system reads, and re-index or purge when records change. For deeper coverage of the retrieval layer, see Questa AI’s guide to RAG security best practices for enterprise AI.

AI-assisted diagnosis and medical imaging

Imaging and decision-support models influence treatment, so their integrity matters as much as their accuracy. Two bodies of published research are worth knowing.

In a 2019 paper in Science, Finlayson and colleagues showed that medical machine learning models, including imaging classifiers, can be made to misclassify by adding small, deliberately crafted perturbations that a human viewer would not notice. Separately, researchers at Ben-Gurion University published CT-GAN at USENIX Security 2019, a proof of concept in which malware inserted or removed lung tumors in 3D CT scans; in their study, radiologists misdiagnosed the large majority of manipulated scans. Both are research demonstrations. They show what is technically possible when an attacker can reach the imaging pipeline, not that hospitals are being attacked this way at scale. The practical lesson is to protect the path images travel (PACS, modality connections, storage) and to test deployed models against adversarial inputs, which few organizations currently do.

A second, less dramatic lesson comes from model monitoring. A 2021 external validation of the widely deployed Epic Sepsis Model, published in JAMA Internal Medicine by Wong and colleagues, found performance well below what the developer had reported (an area under the curve of 0.63 at one health system). Vendor claims are not a substitute for local validation, and a model’s behavior should be monitored after deployment, not only before it.

Patient data leaking through public and third-party AI tools

The most common healthcare AI data exposure is not an attack. It is a well-meaning employee pasting something into a consumer chatbot to save time. A real-world illustration from outside healthcare: in 2023, Samsung employees were reported to have pasted proprietary source code and internal meeting notes into ChatGPT, after which the company restricted generative AI use on company devices. The same behavior in a hospital would involve discharge summaries, referral letters, or billing notes.

Whether that is a HIPAA violation depends on the facts. Under HIPAA, disclosing protected health information (PHI) to a vendor that handles it on the organization’s behalf generally requires a business associate agreement (BAA), and consumer-grade AI products typically do not offer one. An impermissible disclosure is presumed to be a breach unless a documented risk assessment shows a low probability that the information was compromised, so notification duties turn on the specific data, recipient, and circumstances. It is not accurate to say that every use of an external AI tool with sensitive data is automatically a reportable breach, and it is equally unsafe to assume it never is. The right response is a documented assessment each time, with legal and privacy teams involved.

AI vendor assessment and patient-data protection

Before any AI vendor receives patient data, a hospital should be able to answer these questions in writing:

• Will the vendor sign a BAA, and does it cover the specific AI service being used?
• Is patient data retained, logged, or used for training or model improvement? Can that be disabled contractually?
• Where is data processed, and which sub-processors are involved?
• How are tenants isolated, and what prevents one customer’s context appearing in another’s output?
• What happens to data on termination, and how is deletion verified?
• What is the vendor’s incident-notification commitment?

Third-party dependency is also an availability issue. The 2024 ransomware attack on Change Healthcare was a conventional cyberattack, not an AI incident, but it showed how a single vendor outage can interrupt claims processing and pharmacy operations across a national system, with the number of affected individuals later reported at roughly 190 million. When AI becomes part of clinical workflows, the same concentration risk applies, so vendor assessment should include continuity planning, not just privacy terms.

Connected medical devices and IoMT

Hospitals have a wider attack surface than almost any other industry because it extends to patient monitors, imaging systems, infusion pumps, and wearables. Many now feed data into AI models, so a device-level weakness can become an AI integrity problem. The essentials are device authentication, network segmentation so a compromised device cannot reach the wider clinical network, monitoring for unusual device behavior, firmware patching (often neglected for years on medical hardware), and secure transmission between device and cloud. In the US, the FDA now expects premarket cybersecurity documentation for “cyber devices” under section 524B of the FD&C Act, which gives hospitals a reasonable basis to ask manufacturers for a software bill of materials and a vulnerability-management plan.

AI agents in clinical and administrative workflows

Agents go beyond generating text. They can query records, call APIs, send messages, and run multi-step workflows. That introduces excessive permissions, unclear identity attribution, tool abuse, and thin audit trails, and an agent that can act can act wrongly at machine speed. The controls that matter are least-privilege access scoped to the narrowest task, human approval for consequential actions, complete logging of every agent action, continuous behavior monitoring, and a verifiable identity for each agent instead of a shared service account.

Generative AI and LLM Risks Shared by Both Sectors

Some risks apply regardless of industry, and each has published research behind it.

Prompt injection and indirect prompt injection. Malicious instructions hidden in a document, email, or web page can steer an AI system into unintended actions or into revealing data. Greshake and colleagues demonstrated this against LLM-integrated applications in 2023, and it remains an open problem for any system that reads untrusted content. It is especially relevant to assistants that summarize inbound faxes, referral letters, claims, or customer emails.

Training-data extraction. Carlini and colleagues showed in 2021 that large language models can memorize and reproduce verbatim sequences from their training data, including personal information. The risk is greatest for models fine-tuned on sensitive records without adequate controls, which is one reason training data deserves the same scrutiny as production data. See Questa AI’s overview of enterprise AI training data privacy and security risks.

Data poisoning. Subtle corruption of training or retrieval data can degrade accuracy or bias outputs without triggering obvious alarms.

Output leakage. A model that has ingested sensitive context can surface it to the wrong user if isolation is not enforced.

Insecure APIs, jailbreaks, and supply-chain risk. Over-permissioned integrations, attempts to bypass safety constraints, and vulnerabilities inherited from a provider or plugin all expand the blast radius.

OWASP’s Top 10 for LLM Applications and MITRE ATLAS are practical references for threat-modeling these risks.

AI Security in Financial Services

Financial institutions are long-standing targets of organized cybercrime, and AI changes both what attackers can do and what defenders must protect. The risks are different in emphasis from healthcare: the central concerns are financial loss, market integrity, and model risk rather than patient safety.

AI fraud detection and model manipulation

Fraud, credit, and risk models are adversarial by nature: someone is actively trying to defeat them. Attackers can probe a model with test transactions to learn its thresholds, craft behavior that stays just under detection boundaries, or attempt to poison the data a model retrains on so that fraudulent patterns become normal. Defenses include limiting what a model reveals about why it declined a transaction, monitoring for probing patterns, controlling and validating retraining data, and keeping independent rule-based controls so that a manipulated model is not the only safeguard.

One regulatory nuance for EU-facing institutions: the EU AI Act lists AI used to evaluate the creditworthiness of natural persons as high-risk, but expressly carves out AI used to detect financial fraud from that category. Model-security obligations therefore differ between credit scoring and fraud detection even though the technical threats overlap.

Customer financial data and confidential records

Financial AI assistants handle account data, transaction histories, loan files, deal documents, and non-public information. Beyond personal data law, banks carry confidentiality duties and, in some contexts, restrictions around material non-public information. An analyst summarizing a draft transaction with a public chatbot has created a confidentiality problem even if no personal data is involved. This is a useful difference from healthcare: the sensitive asset is not only the individual’s data but the institution’s and its clients’ commercial information.

AI-assisted banking and financial operations

Attackers are also using AI against financial staff. The clearest documented case is Arup, the engineering firm, which confirmed in 2024 that an employee in its Hong Kong office was deceived by a video call featuring deepfaked colleagues, including a person appearing to be the CFO, and made transfers totalling about HK$200 million (roughly US$25 million). Arup was not a bank, but the pattern applies directly to payment approvals, treasury operations, and customer-service channels. Controls that work are procedural as much as technical: out-of-band verification for high-value or unusual payment requests, dual authorization, and call-back procedures that do not rely on the channel the request arrived on.

Third-party AI provider risk

Financial regulators treat AI vendors as a subset of third-party risk. In the EU, the Digital Operational Resilience Act (DORA) has applied since January 2025 and sets requirements for ICT third-party risk management, incident reporting, and resilience testing for in-scope financial entities. In the US, the 2023 Interagency Guidance on Third-Party Relationships sets expectations for risk management across the vendor lifecycle. For AI specifically, institutions should be able to show what data a provider receives, where it is processed, how concentration risk is managed, and how they would exit or substitute a provider.

Access control, auditability, and model monitoring

In finance, the audit trail is often as important as the control. Supervisors and internal audit expect to reconstruct who used an AI system, with what data, and what it produced. Model risk expectations also apply: in the US, the Federal Reserve and OCC’s long-standing model risk management guidance (SR 11-7) calls for validation, ongoing monitoring, and clear ownership, and the New York Department of Financial Services issued an industry letter in October 2024 on AI-related cybersecurity risks for regulated entities. Practically, that means role-based access to AI tools, logged prompts and outputs where lawful, documented model inventories, drift monitoring, and defined escalation when a model behaves unexpectedly.

How AI Security Differs Between Healthcare and Finance

The building blocks are shared: protect the data, control access, monitor behavior, govern agents. What changes is the consequence of failure and the rulebook. In healthcare, the worst outcomes are an incorrect clinical decision or disrupted care, and the governing frame centers on HIPAA, GDPR where EU residents are involved, FDA device expectations, and the EU AI Act. In finance, the worst outcomes are fraud, market disruption, and loss of customer trust, and the frame centers on operational-resilience, third-party, and model-risk rules that vary by jurisdiction.

That is why an AI security program cannot simply be copied between sectors. A hospital will invest more in imaging integrity, device security, and clinician-in-the-loop review. A bank will invest more in adversarial robustness of fraud and credit models, payment-authorization procedures, and vendor concentration. Both rely on shared foundations such as AI governance, but the priorities differ.

Shadow AI: The Exposure Nobody Sees

Shadow AI is staff using unsanctioned AI tools to work faster. It usually happens because approved tools are slow, restricted, or the risk was never explained, not because people are careless. The exposure is that sensitive data leaves the organization’s control, vendor retention practices are unknown, there is no audit trail, and traditional network monitoring sees only ordinary browser traffic.

Large financial institutions recognized this early. In early 2023, JPMorgan Chase reportedly restricted employee use of ChatGPT, and other major banks were reported to have taken similar steps. Reports at the time described this as a precaution over third-party software rather than a response to a specific incident. It is a useful model: act on the risk before the incident rather than after it.

Effective programs combine several measures: an inventory of AI tools actually in use, a clear acceptable-use policy that names what must never be entered into public tools, approved alternatives that are fast enough to be the path of least resistance, DLP on outbound traffic, and training based on real examples. A secure AI gateway between staff and any AI tool gives security teams visibility without forcing employees to change how they work. For more on this risk, see Shadow AI: the biggest data risk.

Protecting Data Before It Reaches a Model vs. Securing the Model Itself

These are two different layers of defense, and buyers often conflate them.

Protecting data before it reaches a model means sensitive-data discovery, masking, redaction, or anonymization, access controls, and policy enforcement applied at the point of use, typically through an AI gateway. The goal is to reduce how much sensitive information any model, provider, log, or output can expose. If a patient name or account number never enters a prompt, it cannot be leaked through that prompt.

Securing the model and its infrastructure means defending against threats that exist whatever data goes in: prompt injection, jailbreaks, adversarial inputs, data poisoning, insecure APIs, over-permissioned agents, and supply-chain compromise. It involves testing, hardening, monitoring, and architecture decisions around the model itself.

Data-layer protection does not prevent prompt injection, model manipulation, or adversarial examples, because those attacks do not depend on the presence of identifiers. Equally, a well-hardened model that receives raw PHI is still a privacy exposure. Two further caveats matter for regulated organizations. First, masking and pseudonymization are not automatically anonymization: under GDPR, pseudonymized data is still personal data, and under HIPAA, information must meet the Safe Harbor or Expert Determination standard to count as de-identified. Second, redacting obvious identifiers does not remove context that can re-identify a person in free text. A sound program uses data-layer protection to reduce exposure and model-layer security to reduce attack success, and treats neither as a complete answer.

Compliance: Legal Obligations vs. Recommended Practice

Frameworks give structure, but none makes an AI system secure or compliant by itself. It helps to separate what is legally required from what is recommended.

Healthcare. HIPAA governs PHI and generally requires a BAA with vendors that handle PHI on a covered entity’s behalf, along with Security Rule safeguards and Breach Notification Rule assessments. GDPR applies where EU residents’ data is involved, treats health data as a special category requiring a lawful basis and extra protection, requires appropriate security, and generally requires notifying the supervisory authority of a personal data breach within 72 hours unless it is unlikely to result in risk to individuals. Medical devices carry their own cybersecurity expectations, such as FDA premarket requirements in the US.

The EU AI Act. The Act has been in force since August 2024, and general-purpose AI obligations have applied since August 2025. The Digital Omnibus on AI, adopted in mid-2026, deferred the high-risk obligations: stand-alone Annex III systems to 2 December 2027, and AI embedded in regulated products such as medical devices to 2 August 2028, while transparency obligations under Article 50 were not moved. Because dates and scope have shifted, confirm the current position with counsel before building a compliance plan around any single date.

Finance. Requirements vary by jurisdiction: DORA in the EU, GLBA safeguards and interagency third-party and model-risk guidance in the US, plus supervisory expectations such as the NYDFS letter noted above. GDPR and the EU AI Act apply where the institution’s footprint reaches the EU.

Recommended practice, not law. The NIST AI Risk Management Framework, its Generative AI Profile (NIST AI 600-1), and the NIST Cybersecurity Framework are voluntary guidance in the US. They are useful for structuring a program and demonstrating diligence, but following them is not a certification or a legal safe harbor. No vendor, including Questa AI, can promise blanket compliance independent of how a system is actually deployed.

Reference Architecture: Where Controls Belong

A practical way to place controls is to trace the path data takes: source systems (EHR, core banking, document stores) feed an AI application, which sends requests through an AI gateway to a model or provider, and the output returns to a human or downstream workflow, with audit logging across the whole path. The gateway is the natural place for sensitive-data detection and masking, policy enforcement, and usage visibility. The source systems need authentication and access logging. The application needs input validation and DLP. The model layer needs vendor review, output filtering, and monitoring. The human layer needs mandatory review for clinical or financial decisions with real consequences. The point of the layered view is that AI security is not one product but controls distributed along the whole path from record to decision.

AI Security Checklist for Hospitals and Financial Institutions

1. Inventory every AI system in use, sanctioned or not, including embedded AI features in existing software.
2. Map what sensitive data each system touches and how it flows in and out.
3. Identify every AI vendor and sub-processor, and review retention, training, and deletion terms.
4. Execute BAAs (healthcare) or equivalent contractual controls (finance) before sensitive data is shared.
5. Apply least privilege to every integration and agent, and enforce source-system permissions at retrieval.
6. Mask or redact sensitive data before it reaches a model wherever the task allows.
7. Test specifically for prompt injection and, for imaging or scoring models, adversarial inputs.
8. Validate vendor performance claims locally and monitor models for drift after deployment.
9. Require human review for clinical decisions and high-value or irreversible financial actions.
10. Keep full audit logs of AI-driven actions and the data they accessed.
11. Extend protection to connected devices (healthcare) and payment-authorization workflows (finance).
12. Control Shadow AI with approved alternatives, not only prohibitions.
13. Build an incident response plan that covers AI failures, including a process for assessing whether an event triggers reporting duties.
14. Review the whole posture on a recurring schedule.

What to Look for in an AI Security Solution

Healthcare providers and financial institutions evaluating vendors should look for capability match rather than brand claims. The most useful questions are about where the control sits and what it can prove.

• Sensitive-data discovery and masking at the point of use, applied before data reaches a model, with clear documentation of what is and is not detected.
• An AI gateway that provides visibility and policy control over which users, applications, and providers handle which data.
• Granular access control tied to identity, role, and purpose, including for agents.
• Policy enforcement that can be updated as regulations and internal rules change.
• Audit logging sufficient to reconstruct who used which AI system, with what data, and what happened.
• Prompt and output protection, and honest answers about what it does and does not stop. Be wary of any vendor claiming to eliminate prompt injection.
• Model and vendor visibility, including shadow usage across third-party providers.
• Testing and red-teaming support for injection, leakage, and adversarial scenarios.
• Compliance mapping that supports your frameworks without claiming certification on your behalf.
• Integration with existing security tooling, so deployment does not require replacing what already works.

In healthcare, add questions about BAAs, PHI handling, and clinician-review workflows. In finance, add questions about model-risk documentation, DORA or third-party-risk evidence, and confidentiality for non-public information. A product that covers a few of these well is more useful than one that claims all of them vaguely. For a broader buyer’s view, see Questa AI’s guide to AI security solutions for enterprises.

How Questa AI Supports Privacy-First Enterprise AI

Questa AI works with healthcare providers, financial institutions, and compliance teams on privacy-first AI adoption. The core of that work is architectural: anonymizing sensitive data before it reaches a model, enforcing access controls at the AI layer, and giving security teams visibility into AI usage that would otherwise be invisible, including Shadow AI. You can see how this works on the How It Works page, or review Questa AI’s approach for healthcare and financial services.

Questa AI does not claim to guarantee cybersecurity outcomes, automatically confer HIPAA or GDPR compliance, or protect against every possible attack. What a privacy-first architecture does is reduce the sensitive data any AI system is exposed to and give organizations the visibility and control to catch problems early rather than after a breach.

Frequently Asked Questions

Conclusion

Healthcare and financial organizations are past the point of treating AI as an experiment running alongside “real” security. It now informs clinical decisions and financial transactions, so it has to be governed like any other critical system. The organizations handling this well are not the ones with the most AI. They know what data reaches their models, who and what can access them, how the models behave over time, and what they will do when something goes wrong. That is what a mature AI security program looks like in either sector.