Bot Swarms · August 2025
Perplexity's Stealth Crawlers Are Dodging No-Crawl Directives
Cloudflare caught Perplexity's crawlers disguising themselves as regular Chrome browsers and rotating IPs to dodge robots.txt blocks. Told no, the bots just changed outfits and kept scraping.
Source: Cloudflare