AI Supply Chain Security: Risks, Real Incidents & Controls

Key Takeaways

• AI supply chain risk covers models, datasets, embeddings, prompts, tools and agents, not only code packages.
• Most documented incidents so far involve familiar failures in AI-adjacent tools: over-permissioned OAuth, over-scoped tokens, compromised CI dependencies, and missing authorization checks. Research shows what agents and RAG systems make possible, but a demonstration is not a confirmed production compromise.
• Agents turn a manipulated input into an action. The questions that matter are what an agent can read, what it can call and what it can change.
• No single control is sufficient. Code review, scanning, sandboxing, signing, content security policies and private hosting each cover only part of the problem.
• Evidence beats assertion: an inventory, a permission map, an approved-source register and a tested rollback path.
• EU AI Act obligations depend on role, risk classification and intended use. The Act does not mandate every practice in this guide.

What Is AI Supply Chain Risk?

AI supply chain risk is the security, privacy, integrity and availability exposure an organization inherits from everything its AI systems depend on: foundation models and inference providers, open-source libraries, model artifacts, training and evaluation datasets, RAG sources and vector databases, coding assistants, plugins and MCP servers, SaaS tools connected through OAuth, and the pipelines that build and deploy all of it.

It differs from ordinary software supply chain risk in two ways. Some dependencies are behavioral: a hosted model can change after a vendor update without a line of your code changing. And some inputs act like instructions: a retrieved document, ticket or web page can steer an agent. A traditional SBOM, which records software components, captures neither.

The main risk types teams should plan for:

• Data and model poisoning: manipulated training, fine-tuning or retrieval content that changes model behavior.
• Malicious or tampered model artifacts: model files that execute code when loaded, or weights altered after release.
• Vulnerable or malicious dependencies: ML frameworks, packages and CI actions, including packages that do not exist until an attacker registers them.
• RAG and vector store poisoning: untrusted content that enters the knowledge base and is later retrieved as context.
• Indirect prompt injection: instructions hidden in content the model processes.
• Plugin, tool and MCP risk: tools that are malicious, compromised or granted more access than the task needs.
• Over-permissioned integrations and credential exposure: OAuth grants, API keys and environment variables reachable by AI tools.
• Vendor change and drift: model, routing or subprocessor changes that alter behavior or legal exposure.
• Nth-party and concentration risk: dependencies several layers away, or too much reliance on one provider.
• Licensing and IP risk: restrictive terms on open-weight models, datasets or generated output.

What Has Actually Happened: Incidents, Disclosures and Research

Security writing about AI blurs three different things: incidents that happened, vulnerabilities that were disclosed, and attacks that researchers demonstrated. They support different conclusions, so this section keeps them apart.

Documented incidents

Vercel and Context.ai (April 2026). Vercel's security bulletin states that the incident originated with a compromise of Context.ai, a third-party AI tool used by a Vercel employee. The attacker used that access to take over the employee's Google Workspace account, pivoted into a Vercel environment, and enumerated and decrypted environment variables that were not marked "sensitive." Vercel notified a limited subset of affected customers, recommended rotating those values, and stated that, with GitHub, Microsoft, npm and Socket, it confirmed no npm packages it publishes were compromised. Vercel closed the incident on October 16, 2026. The Vercel bulletin also published the compromised OAuth app ID as an indicator of compromise. What it shows: an AI SaaS tool authorized against corporate identity is a supply chain dependency, and secrets stored as readable environment variables widen the damage. What it does not show: a flaw in an AI model or in Vercel's published packages. Some outlets reported that the employee granted "Allow All" permissions; Vercel's bulletin does not say so, so treat that detail as unconfirmed.

tj-actions/changed-files (March 2025). Researchers at StepSecurity detected that a widely used GitHub Action had been modified so that it printed CI/CD secrets into workflow logs, tracked as CVE-2025-30066. Where logs were public, anyone could read them. Roughly 23,000 repositories used the action, but that number measures exposure, not confirmed leaks. Wiz assessed that an earlier compromise of the reviewdog action setup was the likely root cause, and CISA added the CVE to its Known Exploited Vulnerabilities catalog. This was not an AI-specific attack, but AI development pipelines depend on the same kind of third-party actions. Lesson: pin actions to full commit hashes and keep secrets out of jobs that do not need them.

Amazon Q Developer extension (July 2025). AWS's security bulletin AWS-2025-015 (CVE-2025-8217) states that version 1.84.0 of the Amazon Q Developer extension for VS Code contained injected malicious code. AWS traced the issue to an inappropriately scoped GitHub token in a CodeBuild configuration, revoked and replaced the credentials, and shipped version 1.85.0. Press reports described the injected prompt as instructing the assistant to delete local files and cloud resources. AWS determined the code was distributed with the extension but did not execute because of a syntax error. What it shows: an AI assistant's own release pipeline is a supply chain target, and a malicious prompt can ride inside a published extension. What it does not show: a successful destructive attack.

Vulnerability disclosures

Lovable (2025 and 2026). Two separate issues are often merged. In 2025, researcher Matt Palmer reported that many Lovable-generated apps using Supabase lacked effective row-level security, and his published statement describes scanning 1,645 apps and finding 170 with exposed data across 303 endpoints. The resulting CVE-2025-48757 describes insufficient row-level security policies that let unauthenticated attackers read or write database tables, and the NVD record notes that the supplier disputes it, arguing that each customer is responsible for protecting its own app's data. In April 2026, a researcher publicly reported a broken object-level authorization flaw in Lovable's own API that let free accounts read other users' project source code, embedded credentials and AI chat histories for projects created before November 2025. Lovable stated it had not suffered a data breach and said the exposure reflected how "public" projects were defined, while acknowledging its documentation was unclear (Computing). What it shows: AI app builders can generate or provision backends with weak authorization defaults, and credentials embedded in project source become exposed when source is exposed. What it does not show: a confirmed, vendor-acknowledged mass compromise. Earlier versions of this article also described exposed Supabase service-role keys as part of the pattern; the CVE record does not say that, so we have removed the claim.

Malicious models on Hugging Face (JFrog, 2024). JFrog's security team reported about 100 models on Hugging Face with malicious functionality, including a PyTorch model that used Python's pickle serialization to open a reverse shell when loaded. JFrog noted that some uploads might be security research, but the files were publicly downloadable either way. Lesson: a model file can be executable code. Prefer formats that do not execute code on load, such as safetensors, and load untrusted pickle-based models only in isolated environments.

Research demonstrations

GitHub MCP and a malicious issue (Invariant Labs, May 2025). Researchers showed that an agent connected to the GitHub MCP server, holding a token that could read both public and private repositories, could be hijacked by a malicious issue in a public repository and made to copy private repository data into a public pull request. Invariant's write-up used researcher-controlled repositories. It is a proof of concept, not a reported compromise of a real organization. Coverage at the time noted there was no simple patch, because the weakness lay in letting one agent combine untrusted input, private data and a public write path.

PoisonedRAG (USENIX Security 2025). Zou, Geng, Wang and Jia showed that injecting five crafted texts per target question into a knowledge base of millions of texts achieved a 90% attack success rate in their experiments, and that the defenses they evaluated were insufficient (paper). This is a controlled evaluation of targeted questions. It shows retrieval is an attack surface; it does not measure how often it is exploited in production. The broader class of indirect prompt injection was described earlier by Greshake and colleagues in 2023.

Agents used offensively

In November 2025, Anthropic published a threat intelligence report on a campaign it attributed with high confidence to a Chinese state-sponsored group, designated GTG-1002. The group manipulated Claude Code, together with MCP tools, to carry out much of an intrusion campaign against about 30 targets, and Anthropic estimated the AI handled 80 to 90 percent of tactical work, with a small number of intrusions succeeding. Earlier versions of this article framed it as a supply chain compromise. It was not: it was misuse of a capable agent by attackers. The relevant lesson for defenders is that agent and tool frameworks lower the cost of attack, and any agent you deploy with broad tool access is a valuable thing to hijack.

Taken together, these cases point the same way. The failures were mostly ordinary: too much access, secrets in readable places, unreviewed dependencies, and untrusted content treated as trusted. AI makes them faster to exploit and harder to see.

Three Illustrative Workflows

The workflows below are illustrative composites built from the patterns above. They are not Questa AI customer incidents.

Example A: AI-generated code in CI/CD

A platform team lets a coding assistant open pull requests. Tests run automatically, passing changes deploy to staging, and a short path leads to production. To add a feature, the assistant introduces a small parsing package. That package pulls in a transitive dependency with an install script, and a generated helper makes an outbound HTTP call to "report diagnostics." The tests check functionality, not network behavior, so they pass. The install script ran in CI with access to build secrets, and the call ships in the artifact.

The unsafe element could be a vulnerable package, a newly registered look-alike package name, insecure generated code or a deliberate malicious change. The pipeline treats all four the same unless controls say otherwise.

Controls that fit this workflow:

Human review required for dependency manifests, authentication, cryptography, network calls and CI configuration, enforced with code owners. Software composition analysis with pinned lockfiles and a block on unknown or very new packages.

Static analysis and secret scanning on every pull request.

Unit, integration and security tests, including tests that assert the application makes no unexpected network calls.

Sandboxed builds with no production credentials and an outbound network allowlist, so unexpected connections fail and are logged.

Protected production credentials, issued only at the deploy stage through short-lived federation.

Approval gates before production deployment.

Artifact provenance and release integrity: signed builds and build attestations, such as those described by SLSA, verified before deploy.

No single layer is enough. Static analysis finds patterns, not intent. Composition analysis depends on known advisories, so a brand-new malicious package may not be flagged. Tests verify expected behavior, not hidden behavior. Sandboxes help only if egress rules are actually enforced. Reviewers miss things, especially in large AI-generated diffs. Provenance tells you where and how an artifact was built, not that its contents are benign. Layering matters because each control fails in different ways. Residual risk remains, which is why monitoring and rollback belong in the design.

Example B: Poisoned RAG content and indirect prompt injection

A support assistant indexes a vendor documentation feed, a shared wiki and customer-uploaded PDFs. It can look up a customer record, create tickets and draft emails. A compromised vendor page, or an uploaded PDF, contains hidden text telling the assistant to include the customer's email address in a link to an attacker-controlled site.

Four different things have to happen, and each is a separate place to break the chain:

1. The content is indexed. This is an ingestion failure.
2. The model treats it as an instruction. This is a model and prompt-design failure.
3. The application lets that instruction influence a tool or action. This is an authorization design failure.
4. The action succeeds. This is the actual impact, such as data leaving or a record changing.

Research such as PoisonedRAG shows the first two stages are feasible. Whether stages three and four happen depends on how you built the system.

Controls by stage:

• Source validation: an approved-source register with owners, and pinned domains for external feeds.
• Document-level permissions: carry source access controls into the index and filter at query time using the requesting user's rights, so a user cannot retrieve what they could not open directly.
• Ingestion review: review new or changed sources and scan for hidden text before indexing.
• Provenance: store source, owner, hash and ingestion time with each chunk. Retrieval filtering: limit untrusted sources and cap how much any one source contributes.
• Instruction isolation: present retrieved text as data, and do not give a call that contains untrusted content access to consequential tools.
• Tool authorization: enforce allowlists and per-call policy outside the model, not inside the prompt.
• Output validation: block links, images and recipients that point to unapproved destinations.
• Human approval for consequential actions such as external email or record changes.

Cryptographic signing can help establish where an artifact came from and whether it changed after signing, but only when key custody, the signing process and verification are sound. It cannot tell you the content is accurate or safe, because a legitimately signed document can still carry malicious instructions. Signing embeddings alone does not reliably detect poisoned content, and it does not show that the underlying text deserved trust. Treat signing as one provenance control next to source validation and review. For the broader set of prompt-layer and retrieval-layer controls, see our guide to LLM security risks and best practices.

Example C: Secrets exposed to an AI agent

A development agent runs on a developer machine or CI runner. The environment holds a .env file, a cloud CLI profile with broad rights and a repository token. Asked to fix a failing build, the agent reads a ticket containing injected text telling it to print the environment "for debugging." The same exposure can occur without an attacker: a shell tool echoes variables into a trace, a log stores tool output, or the agent commits a key into generated code.

The key point is that removing credentials from the prompt does not help if the agent can read them from the filesystem, the environment or a tool. The boundary is what the agent process can reach, not what the prompt says.

To reduce the impact:

• Issue short-lived, scoped credentials through identity federation instead of long-lived keys.
• Keep secrets in a secrets manager, not in environment files the agent can read.
• Use just-in-time access with approval where elevated rights are occasionally needed.
• Define explicit allowlists for tools and actions.
• Give the agent its own least-privilege identity, separate from a human's, and keep production credentials out of development agents entirely.
• Run secret scanning on prompts, outputs, logs, commits and tool payloads, with push protection, and rotate anything that may have been exposed.
• Monitor secret access, unusual cloud API calls and outbound connections, and keep a runbook to revoke the agent's identity and rotate dependent secrets.

The Vercel incident shows why this matters beyond agents: values stored as readable environment variables were exposed, and rotation became the response. The tj-actions incident shows the same pattern in CI logs.

What Each Control Is For

A control's purpose is narrower than the threat it is often credited with stopping. A content security policy restricts what a browser page can load and where it can send data; it does not stop server-side exfiltration or cover every channel. Static analysis flags known code patterns; it does not establish that code is safe. Sandboxing limits blast radius; it does not detect malicious intent. Human review catches design problems and surprises; it does not scale to every line. Dependency scanning matches known advisories; it can miss new malicious packages. Signing and provenance establish origin and integrity, not benignness. Use them together, and write down the residual risk each one leaves.

Private or local-first deployment follows the same logic. It can reduce external data transfer and vendor dependency, and it gives you more control over retention and residency. It does not remove compromised packages, malicious model artifacts, vulnerable plugins, insider access, misconfiguration or malicious retrieved content, all of which live inside the deployment. See our overview of sovereign AI for when control over infrastructure is worth the operational cost.

AI Supply Chain Risk Assessment Checklist

Use this before deployment and again after significant change. Scale it to the system's risk and architecture; not every organization needs every control implemented identically. Each item lists the evidence a reviewer should expect to see.

A. AI dependency inventory

• Foundation models and inference providers, with named model versions and the data each receives. Evidence: an inventory entry per production system.
• AI coding assistants and agent frameworks, including tools adopted without approval. Evidence: a list with owners and enabled tools, cross-checked against SSO and OAuth logs.
• Plugins, MCP servers, tools and API integrations. Evidence: an export of OAuth grants and tool configurations with scopes recorded.
• Open-source libraries and packages, including ML frameworks. Evidence: a software bill of materials generated from each build.
• Training, fine-tuning and evaluation datasets. Evidence: a dataset register with source, owner, license and sensitivity.
• RAG sources, embedding models and vector databases. Evidence: an approved-source register and the vector store access list.
• Logging, monitoring and observability services. Evidence: a data-flow diagram showing where prompts, outputs and traces land.
• Build systems, pipelines and external infrastructure. Evidence: a pipeline inventory showing third-party actions pinned to commit hashes.

B. Provenance and integrity

• Record the source, owner, version and purpose of each important dependency. Evidence: register entries reviewed on a set cadence.
• Verify package and model artifacts against trusted sources. Evidence: pinned versions, lock files, an internal mirror and verification logs.
• Use SBOMs or an AI bill of materials where appropriate. Evidence: CycloneDX or SPDX output. CycloneDX supports machine learning models and datasets in its ML-BOM, and version 1.7 is published as ECMA-424, 2nd edition.
• Evaluate signatures, hashes, attestations and build provenance according to the artifact and threat model. Evidence: a written policy per artifact type and verification at deploy time.
• Approve new or changed dependencies. Evidence: change tickets and an allowlist.
• Track upstream vulnerabilities and advisories. Evidence: feeds mapped to the inventory and time-to-triage records.

A checksum shows that an artifact matches a known value. It does not show that the artifact is benign or that the value came from a trustworthy source. Hashes are only as good as the channel that supplies them.

C. Data and privacy

• Identify sensitive data each dependency can access. Evidence: a data-flow map and classification. Teams can discover where sensitive data enters AI workflows across prompts, uploads, vector stores and logs, instead of relying on the architecture diagram.
• Review retention, model-training terms, subprocessors and processing locations. Evidence: the signed data processing agreement and a dated subprocessor list.
• Minimize the data sent to external services. Evidence: redaction or anonymization settings and a reviewed sample of outbound payloads.
• Restrict access to confidential files and retrieval sources. Evidence: test results showing a user cannot retrieve documents they cannot open normally.
• Check where prompts, outputs, tool payloads and traces are stored. Evidence: a log storage inventory with retention settings. Our guide to PII governance across AI pipelines covers this in depth.
• Set deletion, retention and access policies. Evidence: a written policy and a successful deletion test.

D. Agent permissions and tool execution

• Define each agent's allowed tools and actions. Evidence: a permission map.
• Apply least privilege and separate development from production. Evidence: distinct identities and an IAM policy review. Restrict credentials and sensitive systems. Evidence: a scan showing no long-lived keys in agent environments.
• Require human approval for high-impact or irreversible actions. Evidence: gate configuration and sample audit-log entries.
• Test how the agent responds to malicious retrieved content. Evidence: red-team cases and results, rerun after model or prompt changes, because outputs are non-deterministic.
• Establish safe handling when a tool or inspection control fails. Evidence: a test showing the agent stops or drops to read-only when a filter or policy service is unavailable.

E. Deployment and monitoring

• Test generated code before production. Evidence: required status checks covering unit, integration and security tests.
• Review dependencies and artifacts before release. Evidence: a signed release checklist, composition report and provenance verification.
• Monitor unexpected outbound connections and secret access. Evidence: egress and secrets-manager logs with a tested alert.
• Track material model, plugin, library and vendor changes. Evidence: a change log, version pinning and regression results after vendor updates.
• Establish vulnerability handling and incident response. Evidence: an AI-specific runbook and a tabletop exercise record.
• Reassess risk after significant architecture changes. Evidence: a dated reassessment tied to the change ticket.

For a wider governance program, what enterprises get wrong about AI risk assessments.

How to Assess and Prioritize AI Vendor Risk

A useful vendor assessment starts with what the vendor can touch, not how well known it is. Work through four questions for each vendor, model, plugin or dependency:

1. What data can it reach, and how sensitive is that data?
2. What is its role in the workflow, and how much autonomy and tool access does it have?
3. What happens if it is compromised? Estimate the blast radius.
4. How good is the evidence? Look at subprocessors, retention and training terms, vulnerability disclosure and patching practice, advisories and incident history, update and artifact integrity, logging and audit evidence, and continuity and exit options.

Rating criteria. Rate a dependency high if it can reach regulated or confidential data, can change systems or identities, sits in the build or deploy path, or would be hard to replace. Rate it medium if its data access or actions are limited and compensating controls have been tested. Rate it low if it touches no sensitive data, is read-only and is easy to swap out. Raise the rating one level when evidence is missing, such as no advisory process or no audit logs. Lower it only on tested evidence. Avoid invented numeric scores; they imply precision the inputs do not support.

The highest-priority risks are not always the most prominent vendors. A small tool with broad OAuth access can matter more than a major model provider that only receives sanitized text, and a library used by dozens of pipelines can matter more than either. Access, data sensitivity, dependency concentration and blast radius decide priority. For a wider view of layered controls and how to evaluate security vendors, read our article on AI security solutions: what enterprises need to know.

Regulation: Obligations, Practices, Recommendations and Predictions

Keep four categories apart, because teams often blur them.

Existing legal obligations. The EU AI Act (Regulation (EU) 2024/1689) is risk-based, and obligations depend on the system's classification, your role (provider, deployer or other), the intended use and the provision's application date. Prohibitions and AI literacy have applied since February 2025, and obligations for providers of general-purpose AI models since August 2025. The Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since July 27, 2026, postponed the high-risk requirements to December 2, 2027 for Annex III systems and August 2, 2028 for Annex I products. For high-risk systems, Article 15 requires appropriate cybersecurity, including, where appropriate, measures against data poisoning, model poisoning, adversarial examples and confidentiality attacks. Article 25 addresses responsibilities along the AI value chain, including agreements between providers and suppliers of components. The Act does not prescribe an AI-BOM, a particular signing scheme or any one control for every AI system, and most enterprise AI uses are not high-risk. Where personal data is processed, GDPR also applies, including its rules on processors, subprocessors and security of processing. Verify current dates and text on EUR-Lex, and take legal advice for your situation. See also our overview of the EU AI Act.

Recognized security practices. The NIST AI Risk Management Framework (Govern, Map, Measure, Manage), NIST's secure software development practices for generative AI (SP 800-218A), the OWASP Top 10 for LLM Applications (which lists supply chain, poisoning, prompt injection and excessive agency), ISO/IEC 42001 for AI management systems, and SLSA for build provenance. These are voluntary frameworks or standards unless a contract or regulator makes them binding. Our compliance comparison of the NIST, EU and ISO governance frameworks can help you choose one to map to.

Recommended enterprise controls. The checklist and framework in this guide are our recommendations for managing risk. They are not legal requirements.

Predictions. Expect customers, auditors and regulators to ask for dependency inventories, vendor evidence and agent permission records more often. That is a reasonable expectation, not a stated requirement.

A Five-Step Implementation Framework

This is a working framework built from the controls above. It is not a certified methodology or industry standard. If you need a recognized structure, map it to the NIST AI RMF or ISO/IEC 42001.

1. Inventory AI dependencies

Action: Build the inventory in checklist section A, including shadow AI found through OAuth and SSO logs. Owner: Security architecture with the AI platform team. Artifact: An AI dependency inventory, with an SBOM or AI-BOM where appropriate. Verify: Reconcile it against CI-generated SBOMs, an OAuth grant export and cloud billing, and ask a sample of teams what tools they use that are not listed.

2. Apply least privilege

Action: Map the identities, scopes and tools of every agent and integration, and remove standing credentials. Owner: Identity and platform security, with each service owner. Artifact: A permission map and an exceptions register. Verify: Run a test in which an agent attempts an out-of-scope action, such as reading a production secret, and confirm it fails and is logged.

3. Add verification gates to AI code pipelines

Action: Implement the layered gates from Example A. Owner: DevSecOps and engineering leads. Artifact: A pipeline release checklist and the required-status-check configuration. Verify: Submit test pull requests with a known-bad dependency, a planted secret and an unexpected outbound call, and record which layer caught each one.

4. Govern RAG data sources

Action: Approve, tier and monitor every source that feeds retrieval, and enforce document permissions at query time. Owner: Knowledge owners, the AI platform team and security. Artifact: An approved-source register with owner, trust tier, refresh cadence and permission model. Verify: Plant a canary document with a harmless instruction in a staging source and confirm instruction isolation and tool authorization prevent any action. Test permissions with a low-privilege account.

5. Match deployment to data sensitivity

Action: Choose public API, dedicated hosting, private or sovereign deployment by data sensitivity and regulatory need, not by assumption. Add data minimization before data leaves your boundary. Owner: The CISO or risk lead, with privacy and legal. Artifact: A vendor risk record per important dependency and a deployment decision record. Verify: Reassess on a schedule and after major changes, and run a rollback or exit drill for your most critical vendor.

Where Questa AI Fits

The controls above govern what enters your AI supply chain. A separate question is what sensitive data leaves your environment when people and systems send prompts, documents and tool payloads to AI services. That is the part Questa AI addresses. Its privacy layer detects personal, financial, health and other sensitive entities and anonymizes them before content reaches a model, and Questa Blackbox runs this detection and anonymization inside a customer's own network, according to Questa's product documentation. Sending less sensitive data to a model, plugin or logging service can limit the damage if one of them is compromised or misused.

This addresses the data-exposure side of supply chain risk only. Anonymization reduces exposure but is not absolute, because it depends on detection quality and context. It does not replace dependency and software integrity checks, secrets management, least-privilege access, code review and CI/CD controls, RAG provenance and retrieval authorization, vulnerability management, vendor risk assessment, or monitoring and incident response.

If you are connecting privacy protection to a broader AI risk-management program, see how Questa AI works.

Frequently Asked Questions

Conclusion

AI supply chain risk is the accumulated exposure of every dependency between your data and the decision an AI system influences. The organizations that manage it well are not those with the fewest dependencies. They can name their dependencies, limit what each can reach, verify what can be verified, watch for change, and roll back when something upstream breaks. Start with the inventory and the permission map; they make every other control easier to justify and test.