Move beyond generic compliance claims. Verify data residency, inspect model weight handling, demand evidence of prompt injection defences, and audit the vendor’s own AI usage. Treat the model as code, not a black box service.
1. Verify Data Residency and Isolation
Many vendors claim their infrastructure is secure, but this rarely explains where your data sits while the model processes it. You need to know if your inputs share memory with other clients or if they are processed in a dedicated environment. Multi-tenancy introduces noise and potential cross-contamination risks that standard encryption does not fully mitigate.
Check the vendor’s architecture diagrams for logical isolation guarantees. Ask specifically whether your data is used for model improvement or fine-tuning by default. If the answer is yes, you must have an explicit opt-out mechanism that is technically enforced, not just contractually promised.
2. Inspect Model Weight Handling
Model weights are the mathematical parameters that define how an AI system makes decisions. These weights are valuable intellectual property and can also be a vector for data leakage if they are stolen or corrupted. You must understand how the vendor stores, updates, and transports these weights.
Request documentation on the encryption standards used for weights at rest and in transit. Verify that weight updates are signed and verified to prevent tampering. If the vendor uses third-party hosting for model inference, ensure the same security controls apply to that external environment.
3. Demand Evidence of Prompt Injection Defences
Prompt injection occurs when a user tricks the model into ignoring its original instructions. This can lead to data exfiltration or unintended actions. Many vendors claim their models are robust against these attacks, but this is often untested in real-world scenarios.
Ask for penetration testing reports that specifically target prompt injection vectors. Look for evidence of input sanitisation and output filtering mechanisms. Do not accept a generic statement of safety; require technical details on how the vendor detects and blocks malicious prompts.
4. Audit the Vendor’s Own AI Usage
It is ironic but common for AI vendors to use AI tools in their own development and operations without disclosing the risks. If the vendor uses generative AI to write code or generate documentation, their own supply chain may be vulnerable to training data leakage.
Inquire about the vendor’s internal policies on using generative AI tools. Ensure they have controls to prevent sensitive code or customer data from entering public AI models. This aligns with best practices for preventing training data leakage in your own environment.
5. Evaluate Explainability and Transparency
You cannot secure what you cannot understand. If the vendor’s model is a black box, you cannot assess why it made a specific decision or if it was manipulated. Explainable AI techniques allow you to trace decisions back to input features, providing a layer of auditability.
Require the vendor to provide feature importance scores or decision trees for critical outputs. This helps you identify bias or unexpected behaviour. Without this transparency, you are relying on the vendor’s word that the model is behaving correctly.
See also: Training Data Leakage Explained: How AI Memorises Your Secrets · AI-Generated Phishing: Risks and Protection for Small Businesses
6. Assess Supply Chain Dependencies
AI models are rarely built from scratch. They rely on open-source libraries, pre-trained models, and third-party data sources. Each of these dependencies introduces potential vulnerabilities.
Ask the vendor to disclose all significant third-party components used in their model. Review the security posture of these providers. A breach in a minor library can compromise the entire AI system, making this a critical part of your assessment.
7. Define Incident Response Protocols
When an AI system fails, the consequences can be immediate and severe. You need to know how the vendor responds to security incidents involving their model. Standard incident response plans often do not cover AI-specific threats like model poisoning or adversarial attacks.
Ensure the vendor has a dedicated AI incident response plan. Ask for their communication timelines and remediation steps. You should also understand how they notify you of incidents that may affect your data or operations.
8. Review Contractual Liability and SLAs
Standard service level agreements often exclude AI-specific failures. If the model hallucinates or provides incorrect data, you need to know who is liable.
Negotiate clauses that define responsibility for AI errors and data breaches. Ensure the contract includes right-to-audit provisions for security controls. This legal framework supports your technical assessments and provides recourse if things go wrong.
| Practice | Why it matters |
|---|---|
| Verify Data Residency | Prevents cross-tenant data leakage and ensures compliance with data sovereignty laws. |
| Inspect Model Weights | Protects intellectual property and prevents tampering with the core logic of the AI. |
| Demand Injection Defences | Mitigates the risk of attackers manipulating the model to steal data or execute commands. |
| Audit Vendor AI Usage | Reduces the risk of your data leaking through the vendor’s own use of public AI tools. |
| Evaluate Explainability | Allows you to audit decisions and identify bias or manipulation in the model’s output. |
| Assess Supply Chain | Identifies vulnerabilities in third-party libraries and data sources used to build the model. |
| Define Incident Response | Ensures rapid and effective response to AI-specific security incidents. |
| Review Liability | Clarifies responsibility for AI errors and provides legal recourse for breaches. |
9. Continuously Monitor Model Performance
Security is not a one-time event. AI models can degrade over time or be subtly manipulated through data poisoning. You need to continuously monitor the model’s performance and behaviour.
Implement logging and alerting for unusual model behaviour. Regularly review the model’s outputs for signs of drift or bias. This ongoing monitoring helps you detect and respond to security issues before they become critical.
Consider integrating these practices with your existing security frameworks. For example, using MITRE ATLAS can help you map AI-specific tactics and techniques to your defensive controls. Similarly, understanding AI transparency principles can guide your evaluation of vendor disclosures.
Key takeaways
- Generic SOC 2 certificates do not prove AI-specific data isolation.
- Model weights are intellectual property that can leak if not encrypted at rest and in transit.
- Prompt injection risks require technical verification, not just policy assertions.
Treat AI vendors as code suppliers, not just service providers. Verify their technical controls through evidence, not just claims, and continuously monitor the model’s behaviour for signs of compromise.
Frequently asked questions
How do I verify a vendor’s claim of data isolation?
Request architecture diagrams and technical documentation that show logical separation. Ask for evidence of dedicated resources or encryption keys for your tenant.
What is prompt injection and why is it dangerous?
Prompt injection tricks the model into ignoring its instructions, potentially leading to data theft or malicious actions. It is dangerous because it bypasses traditional security controls.
Should I avoid vendors using open-source models?
Not necessarily, but you must assess the security of the open-source components. Ensure the vendor patches vulnerabilities and monitors for threats in the underlying libraries.
How often should I re-assess an AI vendor?
Re-assess annually or whenever the vendor makes significant changes to their model or infrastructure. Continuous monitoring is also recommended for high-risk applications.
How this guide was produced: written by the AppSec Scoop editorial team with AI assistance, checked against the public references listed below, and reviewed when the facts change. See our editorial policy or report an error.
