ImportantAI & ML

Owner of 1.5M-page site PatronView on a year of fighting scrapers: 214:1 bot-to-human page loads, 35K Claude crawls per referred user, Amazon bot referred none (Nick Gray/PatronView)

A website operator documented a year-long battle against web scrapers, revealing a 214:1 bot-to-human traffic ratio with AI crawlers (particularly Claude) generating 35,000 requests per referred user, while traditional commercial bots like Amazon's contributed no referral value. This case study exposes significant infrastructure costs, security vulnerabilities, and erosion of data assets that CIOs must address through updated bot management strategies, API governance, and licensing agreements with AI vendors. Organizations face a critical decision point: implement aggressive anti-scraping measures, negotiate data licensing with AI companies, or accept reduced ROI on content investments.

Nick GrayTechMeme2 min read
Read full article
Owner of 1.5M-page site PatronView on a year of fighting scrapers: 214:1 bot-to-human page loads, 35K Claude crawls per referred user, Amazon bot referred none (Nick Gray/PatronView)
Nick Gray / PatronView: Owner of 1.5M-page site PatronView on a year of fighting scrapers: 214:1 bot-to-human page loads, 35K Claude crawls per referred user, Amazon bot referred none — A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now. August 7, 2026