Cloudflare requires AI companies to separate training crawlers from search crawlers by September 15

TechCrunch reports that Cloudflare has set a September 15, 2026 deadline requiring AI companies to distinguish web crawlers used for traditional search from those used for AI training and agent services. Beginning that date, Cloudflare's default settings will block mixed-use crawlers from pages hosting ads on new customer accounts, new sites, and all existing free customers. The policy affects AI model providers' ability to access web content for training and agentic services.