The Conflation Problem

The AI field routinely conflates three distinct concepts under the umbrella of 'open AI': open weights, open source, and open training data. Each represents a different layer of accessibility, with different legal, ethical, and practical implications. Treating them as equivalent produces both inflated claims about model transparency and systematic underestimation of the structural problems in the AI data commons.

The Three Layers

Layer 1 — Open weights means the trained model parameters are publicly downloadable (e.g. Llama 4, Mistral, DeepSeek R1). This enables inference, fine-tuning, and deployment without proprietary API dependency. It does not imply reproducibility of training.

Layer 2 — Open source in the classical sense (OSI definition) means the full training pipeline — code, architecture, hyperparameters, training scripts — is available under a license that permits study, modification, and redistribution. Very few frontier models qualify. Most 'open-weight' releases are proprietary at Layer 2.

Layer 3 — Open training data means the data on which the model was trained is available, licensed for reuse, and legally unencumbered. This is where the commons is most severely closed. The web, which has been the primary fuel for language model pretraining, is systematically closing to scraping (Longpre et al., 2024: consent in crisis). Meanwhile, virtually all frontier models were trained on copyright-protected material — books, code, journalism — without explicit license or compensation.

The Hypothesis

Conflating the three layers systematically understates the AI training data crisis and overstates the openness of the current AI ecosystem. Specifically:

1. A model can be fully open-weight and open-source while being trained on legally encumbered, non-replicable data — creating a false appearance of reproducibility.
2. The economic and legal sustainability of the AI commons depends primarily on Layer 3, which receives the least policy and research attention.
3. The content value chain collapse — where journalism, academic publishing, and creative work are consumed as training fuel without compensation or attribution — is structurally analogous to the extraction dynamic in the original open-source commons (Poetz et al., 2012; Bartók/folk music analogy: ordinary human creativity collected, transformed, distributed as a cultural product the originators cannot access).

Falsifiability

• If models trained on licensed, consent-based data demonstrate no measurable capability degradation relative to unrestricted-corpus models, Layer 3 openness is tractable at scale.
• If opt-in data commons (e.g. Common Crawl successors with consent infrastructure) accrue sufficient volume to sustain competitive pretraining, the crisis resolves without policy intervention.
• If courts establish that pretraining on copyright material constitutes fair use, Layer 3 ceases to be a structural barrier.

References

• Longpre et al. (2024). Consent in Crisis: The Rapid Decline of the AI Data Commons. arXiv:2407.14933
• Longpre et al. (2025). Economies of Open Intelligence. arXiv (forthcoming)
• OSI Open Source Definition: https://opensource.org/definition
• Poetz, M. & Schreier, M. (2012). The Value of Crowdsourcing. Journal of Product Innovation Management, 29(2), 245–256.