Internal Critique and Legal Maneuvers: AI Scraping Under Scrutiny
A Microsoft director's internal memo, revealed in court filings, labeled the practice 'an astonishing theft.'3
The allegation surfaces as the same companies offer publishers a new way to block such scraping, while facing multiple lawsuits alleging copyright infringement.1,2
- A Microsoft director's internal memo, revealed in court, called AI data scraping 'the largest theft of labor in human history.'3
- The New York Times alleges in its lawsuit that OpenAI and Microsoft bypassed paywalls and stripped copyright notices to build AI training datasets.3
- Microsoft data showed its Copilot system reduced click-through rates to New York Times articles by up to 93% compared to traditional search, according to an internal presentation.3
- Microsoft, OpenAI, Google, and Anthropic have pledged a new technical standard allowing publishers to opt out of future AI training scraping, though it applies only to future models and scraping remains the default.2
- The factual basis for the scale of the alleged 'theft,' the specific methods of any paywall bypassing, and the companies' formal responses to the unredacted allegations remain unresolved.3
A Microsoft executive wrote in a January 2023 internal memo that the scraping of online content to train artificial intelligence models constituted 'the largest theft of labor in human history,' according to newly unredacted filings in a copyright lawsuit.3
The memo by Brent Hecht, Microsoft's director of Applied Science, was filed under seal by The New York Times in its lawsuit against OpenAI and Microsoft.3
The disclosure comes as Microsoft, OpenAI, Google, and Anthropic have pledged to provide a standardized technical method for publishers to opt out of having their content used for future AI training.2
The Unsealed Memo and Internal Data
In a January 2023 internal memo, Microsoft's director of Applied Science, Brent Hecht, wrote that the scraping of online data for AI training represented 'an astonishing theft of unprecedented proportions' and 'the largest theft of labor in human history,' according to unredacted court filings.3
The same unredacted material, filed by The New York Times, included a Microsoft internal presentation by Hecht which showed the company's Copilot 'answer engine' reduced click-through rates to New York Times domain articles by as much as 93% when compared to traditional Bing search results.3
Allegations in the New York Times Lawsuit
The New York Times' lawsuit, filed in December 2023, alleges OpenAI and Microsoft used millions of its articles without authorization or payment to train AI models like ChatGPT.2
The unredacted filings further allege the companies bypassed paywalls undetected, built training datasets via mass scraping from sources including the Bing Index and Common Crawl, and deliberately stripped copyright notices from training data.3
In deposition testimony cited in the filings, Microsoft CEO Satya Nadella said paywalled content should be licensed and that he would have required OpenAI to retrain its models had he known of paywalled scraping.3
The Industry's Voluntary Opt-Out Pledge
In October 2026, OpenAI, Google, Microsoft, and Anthropic pledged to give media outlets and websites a single standardized way to prevent their content from being scraped to train future AI models.2
Microsoft said in a blog post, reported by Tech Xplore, that the new setting would apply only to future model training, would not prevent other forms of crawling, and that scraping would remain allowed by default until a publisher opts out.2
Microsoft stated it would implement the protocol beginning February 1, 2027, while Google planned implementation sometime in 2027, and Anthropic said it had already started rolling it out.2
Related Legal Action Against Brave Software
Separately, The New York Times and other newspaper companies filed a lawsuit against Brave Software in U.S. District Court on November 2, 2026.1
The lawsuit alleges Brave bypasses paywalls and sells scraped news content to AI companies including Anthropic, Perplexity, Mistral, xAI, and Cohere.1
The plaintiffs claim, citing third-party revenue estimates, that Brave made more than $100 million by copying and reselling publishers' work without consent.1
The suit accuses Brave of direct, contributory, and vicarious copyright infringement and of violating the Digital Millennium Copyright Act.1
Ars Technica staff testing the Brave browser for Mac found its optional 'paywall blocker' feature, enabled by default, allowed reading a paywalled New York Times article.1
Legal Context and Unresolved Questions
TechCrunch reported that OpenAI and Microsoft did not return its requests for comment on the unredacted filings.3
The outlet also reported that judges have been largely favorable to AI companies' fair-use arguments in similar cases, and that the Trump administration recently filed a legal brief defending OpenAI's unlicensed training use.3
Microsoft director Brent Hecht writes internal memo calling AI scraping 'the largest theft of labor in human history.'3
The New York Times sues OpenAI and Microsoft, alleging unauthorized use of millions of articles to train AI.2
OpenAI, Google, Microsoft, and Anthropic pledge to offer publishers a standardized technical opt-out for future AI training scraping.2
Unredacted filings in the NYT lawsuit are reported, revealing Hecht's memo and internal data on reduced click-through rates.3
The New York Times and other newspaper companies file a lawsuit against Brave Software, alleging paywall bypass and resale of scraped content to AI firms.1
- The factual basis for the alleged paywall bypassing and copyright-stripping: Are there specific technical methods or internal communications cited in the sealed exhibits?
- The scale and monetary valuation of the alleged 'theft of labor': What metrics did Hecht or others use to support the 'largest in history' comparison?
- Microsoft's and OpenAI's formal, on-the-record responses to the specific allegations in the unredacted filings.
- The current status and potential outcomes of the NYT v. OpenAI/Microsoft lawsuit beyond the reported judicial trends.
- The technical details and limitations of the new 'standardized way' to opt out of scraping, and how it differs from existing methods like robots.txt.
- The proportion of training data for major AI models that was sourced from paywalled or otherwise access-restricted content.
- 1.
- 2.
- 3.