Perplexity AI: the answer engine with a lot of question marks
See all Stories
R
Cloudflare is offering to block crawlers scraping information for AI bots.
Tech giants are rewriting the rules on web scraping, blaming unnamed third parties for disregarding robots.txt, and seemingly claiming the right to reuse anything posted anywhere for AI.
Now, Cloudflare is telling customers on its CDN that it can find and block AI bots that try to get around the rules.
The upshot of this globally aggregated data is that we can immediately detect new scraping tools and their behavior without needing to manually fingerprint the bot, ensuring that customers stay protected from the newest waves of bot activity.
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
Loading comments
Getting the conversation ready...
Most Popular
Most Popular
- Sony’s PlayStation 5 is $200 off for the first time since December
- Anthropic’s most dangerous AI model just fell into the wrong hands
- You’re about to feel the AI money squeeze
- Elon Musk admits that millions of Tesla vehicles won’t get unsupervised FSD
- Microsoft launches ‘vibe working’ in Word, Excel, and PowerPoint
