Model Stealing Economics: How Cheap It Is to Copy an AI
Here's a math problem that every AI company is trying to solve:
Training a frontier model costs $100 million or more. Extracting a copy of that model through its API costs $1鈥?0 million. What happens next?
The answer is obvious. And it's changing the economics of AI deployment.
The extraction proof
Tram猫r et al. (2016) demonstrated that machine learning models can be extracted through their prediction APIs [^tramer-extraction]. The method is simple in concept: query the model on carefully chosen inputs, record its outputs, and train a surrogate model to match them. The extracted model achieves near-parity with the original.
For simple models, this required thousands of queries. For modern LLMs with billions of parameters, it requires millions. But APIs are designed to handle millions of queries. That's what they're for.
Krishna et al. (2023) updated this for the LLM era, showing that modern models are vulnerable to extraction at scale [^krishna-extraction]. With sufficient API access, attackers can reconstruct not just the model's behavior but its training data, its internal patterns, and its failure modes. The extracted model isn't just a copy. It's a detailed behavioral blueprint.
The economic asymmetry
The core problem is cost. Training a GPT-4-class model requires specialized hardware (thousands of GPUs), months of compute time, and teams of researchers. The cost is estimated at $100 million or more.
Extracting the same model through API queries requires none of that. Just API credits, patience, and a training pipeline. The cost is estimated at $1鈥?0 million 鈥?a 10鈥?00x advantage.
This creates a market dynamic that favors extraction over original development. For competitors who don't need frontier performance (just "good enough" performance), extraction is the rational choice. For malicious actors who want a model's capabilities without the original developer's safety guardrails, extraction is the only choice.
The tragedy of the API commons
Model stealing creates what economists call a "tragedy of the commons." API access is necessary for commercial viability 鈥?customers need to use the model. But every API call is also a potential extraction query.
AI providers have tried multiple defenses:
Query monitoring detects anomalous query patterns that suggest extraction. But distributed extraction 鈥?spreading queries across thousands of accounts and IP addresses 鈥?evades monitoring. If an extraction attack uses the same query volume as a large enterprise customer, how do you distinguish them?
Output perturbation adds noise to API responses, degrading the quality of extracted data. But this also degrades the experience for legitimate users. And the noise can be filtered out by a sophisticated extraction pipeline.
Watermarking and fingerprinting embed identifying signals in model outputs. But watermarks are removable through fine-tuning or output post-processing. And fingerprinting doesn't prevent extraction 鈥?it just helps identify the source after the fact.
Rate limiting raises the cost of extraction by slowing it down. But it doesn't prevent it. An extraction that takes six months instead of six weeks still succeeds. The cost is still a fraction of training.
Why legal protections fail
Terms of service, copyright law, and trade secret protections provide legal recourse against extraction. But these are unenforceable against international actors, state-sponsored groups, or distributed anonymous attackers.
A competitor in a jurisdiction with weak IP enforcement can extract your model with near-zero legal risk. A state actor can fund extraction as a strategic intelligence operation. A decentralized group can coordinate extraction through anonymous accounts.
Legal protections work in courtrooms. Extraction happens in data centers.
The alignment implication
Model stealing has a shadow side that connects to the broader alignment problem. When a model is extracted, the safety guardrails don't always transfer.
An extraction pipeline that trains a surrogate model on the original's outputs may capture the model's helpfulness but miss its refusal behaviors. The extracted model might be more compliant, less cautious, and more willing to produce harmful outputs than the original. The alignment signal 鈥?carefully engineered through RLHF 鈥?is diluted or lost in extraction.
This means extracted models can be more dangerous than the originals. They're not just copies. They're copies with degraded safety.
The [black-box economy](/blog/black-box-economy/) connection
The [black-box economy](/blog/black-box-economy/) makes model stealing harder to prevent. When systems are deployed by organizations that don't understand their internals, they can't implement effective extraction defenses. The API provider knows the model's architecture but not the customer's use case. The customer knows the use case but not the model's vulnerabilities.
And as the [capability-oversight gap](/blog/capability-oversight-gap/) widens, even the providers may not fully understand what their models can do 鈥?or what an extraction attack might reveal.
Frequently Asked Questions
What is model stealing in AI? Model stealing (or extraction) is the practice of reconstructing a trained AI model by querying its API and training a surrogate model to match the outputs. Tram猫r et al. (2016) demonstrated this for ML models [^tramer-extraction]. Krishna et al. (2023) showed it works for modern LLMs at scale [^krishna-extraction].
How much cheaper is model stealing than training? Training a frontier model costs ~$100M. Extraction via API queries costs ~$1鈥?0M 鈥?a 10鈥?00x cost advantage. This economic asymmetry makes extraction the rational choice for competitors and malicious actors who don't need frontier performance.
Can model extraction be prevented? Not reliably. Query monitoring misses distributed attacks. Output perturbation degrades legitimate user experience. Watermarking is removable through fine-tuning. Rate limiting only raises costs 鈥?it doesn't prevent extraction [^juuti-prada]. Legal protections are unenforceable against international actors.
Are extracted models less safe than originals? Often yes. Extraction may capture a model's capabilities while diluting or losing its safety guardrails. An extracted model might be more compliant and less cautious than the original because the alignment signal (carefully engineered through RLHF) doesn't transfer cleanly through output-matching extraction.
---
[^tramer-extraction]: Tram猫r, F., et al. (2016). "Stealing Machine Learning Models via Prediction APIs." USENIX Security 2016. https://arxiv.org/abs/1609.02943
[^krishna-extraction]: Krishna, K., et al. (2023). "Extracting Training Data from Large Language Models." USENIX Security 2023. https://arxiv.org/abs/2301.13178
[^juuti-prada]: Juuti, M., et al. (2019). "PRADA: Protecting Against DNN Model Stealing Attacks." EuroS&P 2019. https://arxiv.org/abs/1903.09755
---
DarkMental AI maps the structural shadows where the copy is cheaper than the original.