ML Threat Modeling: Assume Your Data Is Compromised
Threat modeling for ML means assuming your training data is already compromised. This is crucial for services using public or user-supplied datasets. The main footgun is trusting data sources, as data poisoning can silently corrupt your model's behavior.
Why it exists
Traditional security practices don't cover the unique vulnerabilities of ML systems. The skills of security engineers and data scientists don't naturally overlap, so a specific process is needed to bridge this gap and address new attack vectors like data poisoning, which can compromise a model without ever touching its code.
The mental model
Assume your data is compromised from the start. Treat your training data stores, data providers, and the data itself as part of your threat modeling scope, not as trusted inputs. The core shift is to move from a "garbage in, garbage out" mindset to a "malicious data in, compromised model out" one.
How it works
Threat modeling for ML is a structured conversation between security and data science teams, centered on the data lifecycle. It focuses on asking new questions. For example: How would you know if your training data was poisoned? What telemetry do you have to detect a skew in data quality? If you train on user-supplied inputs, what validation are you performing? How do you secure the connection to online data stores?
When to use it
Use this process for any product that interacts with or is built on an ML model. It is especially critical when your model trains on data from untrusted sources, such as public datasets, user-generated content, or third-party data feeds. The more your model's behavior depends on its training data, the more important this is.
When not to use it
This process extends, not replaces, traditional security. You must still address conventional threats to your software stack and infrastructure; failing to do so makes ML-specific attacks trivial. While always useful, the urgency is lower for static models trained on a small, well-understood, and internally-generated dataset that never changes.
One canonical example
A service uses an ML model to filter spam comments, periodically retraining on new user submissions. An attacker submits thousands of subtly crafted, non-spam comments containing specific keywords. The model retrains on this poisoned data and starts misclassifying legitimate comments with those keywords as spam, creating a denial-of-service attack. A proper threat model would have prompted questions about input validation and monitoring for data distribution shifts.
Interview question
For which scenario is ML threat modeling, with its core assumption of compromised data, most critically important?
- a.A model that uses a novel or experimental architecture, making its behavior unpredictable.
- b.A model whose training data is sourced from public repositories or user-generated content.Correct
- c.A model trained exclusively on internally generated, static, and well-curated datasets.
- d.A model deployed in a highly regulated industry, regardless of its data source.
Why? this is the answer
The card explicitly states that this process is "especially critical when your model trains on data from untrusted sources, such as public datasets, user-generated content, or third-party data feeds." While models in regulated industries (D) require robust security, the specific focus of ML threat modeling on data compromise makes untrusted data sources the most critical scenario. Option C describes a scenario where urgency is lower, and option A describes a different type of risk.
Just read this? Test yourself on what you have been reading.
Read the original → learn.microsoft.com
You just looked this up. Could you explain it out loud?
That is the part interviews actually test. Tezvyn takes questions like this one and gives you what the interviewer is really checking, the answer that lands, and the mistake that ends the conversation, in the four minutes before your next meeting.
The iPhone app is on the way
We are building it. Until it lands, nothing here is held back from you: every interview card, your saved cards, streaks and the job board all work in Safari, plus hundreds of free practice quizzes of thirty questions each. Sign in and it all carries over to the app the day it arrives.
Want it as an icon? Tap Share at the bottom of Safari, then Add to Home Screen. It opens full screen and the cards you have read stay available offline.
We are hiring for this. Open roles that interview on mlops — each one lists the topics its interview covers.
See open roles