In the early days of generative AI, it seemed as if the labs were so heads-down building large language models that they didn’t think much about the raw material they were using. After all, big data sets like Common Crawl already existed, and trawling the open web was how search engines had worked for decades.
Chatbots were arguably just a variation on that idea. Who knew that their method for sourcing information would get so controversial?
It turns out, everybody. In a brief made public September 17 in The New York Times’s lawsuit against OpenAI and Microsoft, internal documents show people at both companies saying things that suggest they understood exactly what they were taking, and how it would look.
Microsoft’s Brent Hecht, a director of applied science, warned in a memo in early 2023 that “millions of people around the world will soon consider large models ‘hoovering up’ all their work to be an astonishing theft,” and called it “the largest theft of labor in human history.”
When a researcher described getting around the Times paywall, OpenAI President Greg Brockman replied, “ah nice.” And Nick Turley, OpenAI’s head of ChatGPT, called chatbots an “existential threat” to publishers.
We’ll see how much this factors into the ongoing case, but it underscores what everyone knows: Content has value, even if it’s scraped “for free” from the internet. That value can vary with the type of content, how unique it is, and what the customer of that content wants to do with it. But everybody agrees it’s not zero.
{“blockType”:”mv-promo-block”,”data”:{“imageDesktopUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to u003Cemu003EThe Media Copilotu003C/emu003Eu003C/strongu003E”,”dek”:”Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for u003Cemu003EThe Media Copilotu003C/emu003E. To learn more, visit u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”text”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
By the way, that includes the government. Related to the case from The Times, the Department of Justice took the unusual step of filing a statement of interest on whether AI companies training their models on publishers’ content is fair use, which would effectively give them a pass on doing so. The DOJ said it is, shocking absolutely no one.
The administration has been quite clear that it sees any concession on the copyright question as helping the Chinese win the AI race, since they won’t abide by the same rules. President Donald Trump’s own summary: “China’s not doing it.”
Follow the answers, not the archive
All of this speaks only about training models, not AI search, or more broadly, inference. In fact, when talking specifically about what AI systems produce, the DOJ conceded that “an output reconstructing and disseminating an original copyrighted work may not be transformative,” referring to one of the pillar conditions of the fair-use doctrine. That’s close to a description of AI search, which is basically a machine for summarizing current reporting.
Also, the unsealed documents speak directly to another fair-use pillar: harm to the market for the original work. Turley wrote that OpenAI’s products “are largely substitutive, period,” and Microsoft CEO Satya Nadella testified that using chatbots “has substituted” for visiting the original sources.
The Times filed its lawsuit in December 2023, when the conversation was largely about training. And while the latest back-and-forth shows the legal case is far from over, the ambiguity is why it’s been difficult for a market to emerge for training data: It’s hard to justify investing in a framework to pay for something when a lot of indicators suggest it might be free in a few months. Training is where the lawsuits are. Inference is where the money is.
The other side of the coin here is the inference market: providing AI answers about real-time content. The better your AI provides those answers, the more valuable it will be. So it stands to reason AI companies should recognize that value and be lining up to pay for the quality content that will improve their products.
Except that hasn’t really happened. Despite a few licensing deals here and there, the AI industry has been mostly uninterested in building a marketplace or payment technology where they can buy content in real time for a fair price. Brian Morrissey at the Rebooting suggests this is because most content isn’t unique enough—even if you have the world’s best enchilada recipe, the AI only needs one, really.
Commoditization means a race to the bottom, and the bottom in this case turns out to be pretty close to zero.
The middlemen already cashed in
It’s a good theory if all you look at is the publisher deals. But the broader market tells a different story. I’ve written about the class of data brokers, essentially AI “middlemen,” who scrape the internet at scale and then resell the data.
Going by names like Exa, Parallel, and Tavily, their customers aren’t just AI companies, but a whole cadre of enterprise businesses, including ad agencies, investment firms, even other publishers. One estimate, cited in Matthew Scott Goldstein’s widely circulated report on the scraper economy, pegged the market at about $1 billion.
So there’s a market for inference; it’s just that the money is mostly bypassing the media. That’s a problem, and it’s getting worse: The bot-protection company DataDome put out a report that showed “bad” bot traffic grew 124% in a year, more than nine times faster than human traffic, meaning more bots are visiting sites and scraping data even when they’re told not to. Scraping alone jumped 185%.
An interesting quirk in the report: Meta, which just released its personal AI agent, Muse, is responsible for 46.3% of all AI bot traffic DataDome tracked in the first half of the year, well ahead of OpenAI. Meta also has deals with several publishers, including USA Today, CNN, Fox News, People Inc., and others. So the AI world’s biggest data harvester is also a buyer. It just gets to decide when.
Redirecting the market for inference back toward publishers will take some doing, but it’s already starting to happen. Parallel has introduced a way to pay publishers for their contributions to agent tasks, with The Atlantic and Fortune among its first partners.
Companies like Cloudflare, TollBit, and ProRata have all built payment rails for “good” bots to pay for what they take, and Cloudflare just shifted to paying publishers when their content shapes an AI answer, not just when it’s fetched.
Even Google is opening up payments to some publications when their content contributes significantly to answers in AI Overviews, AI Mode, and Gemini. It’s extremely early days, but the idea of AI inference working similarly to the YouTube Partner Program is actually beginning to look attainable.
A buyer’s market
So we have a market, and we have ways to pay. Everything’s there for the publishers to be compensated as essentially data suppliers to AI systems. The open question is who sets the price? Is it the publishers, the exchange, or the buyer?
Right now it’s the buyer. If you’re looking to buy content, there are more than a dozen data brokers willing to sell it to you for the lowest possible price, and the exchanges are nascent, with wildly inconsistent pricing. Even Google’s program pays whatever Google decides; publishers in the pilot described the math to Digiday as “quite black box,” and one called the offers “lowball.”
Jonathan Roberts, People Inc.’s chief innovation officer, described the situation this summer as having 30 Napsters for content, but no Spotify.
At this point, you could say we have a few baby Spotifys. But Spotify didn’t grow up and get artists paid just because it created a reliable exchange. In the case of the music industry, rights holders also had enough collective weight to insist on it. Publishers have the first part forming, but almost none of the second.
The government is clearly sitting this out, so the leverage has to come from somewhere else. Publishers could act collectively on licensing, and SPUR, a coalition of publishers building standards for how AI systems license and track their content (the Associated Press recently joined), might be the beginnings of this.
They can also work toward making access more controlled through better protections and their own agentic tools, but that would require a major shift, since DataDome’s report also shows 65.3% of the more than 21,000 popular websites it tested didn’t stop a single one of its test bots.
AI systems have improved considerably over the last several months, but their answers are still only as good as the information they have access to. Even Microsoft’s Nadella knows this, saying in the court documents, “anything that is paywalled should be licensed.” The market just hasn’t been told what “ideally” costs.
If publishers want to set those terms, they’re going to need to come up with leverage, either by banding together or shoring up their defenses. Otherwise buyers will continue to think it’s all just for the taking. And right now, they’re not wrong.
{“blockType”:”mv-promo-block”,”data”:{“imageDesktopUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://images.fastcompany.com/image/upload/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to u003Cemu003EThe Media Copilotu003C/emu003Eu003C/strongu003E”,”dek”:”Want more about how AI is changing media? Never miss an update from Pete Pachal by signing up for u003Cemu003EThe Media Copilotu003C/emu003E. To learn more, visit u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”text”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}