AI Summary
A 2025 Akamai report reveals a 300% surge in AI crawler bot traffic driven by the data training needs of major tech companies, positioning these bots in a gray area between beneficial search engine crawlers and malicious scrapers. While AI bots can potentially offer source citations, they often provide “zero-click” answers that bypass website visits and generate massive request volumes, causing increased hosting costs or even server crashes. As a result, over 80% of Cloudflare customers have chosen to block these bots; website owners are advised to weigh the trade-offs between potential visibility and server performance, utilizing tools like Cloudflare or robots.txt to manage access based on their specific goals.Growth of AI Crawler Bot Traffic
According to a report by Akamai [1], there has been a 300% increase in AI crawler bot traffic in 2025. This is being driven by a demand for content to train models for generative AI, and includes bots run by Google, ByteDance (owner of TikTok), ChatGPT, and Claude. With no signs of slowing down, it’s expected that this trend will continue.
Helpful or Harmful?
The spectrum of bot traffic includes “good bots,” such as search engine crawlers that index websites for platforms like Google and Bing, and other helpful bots like those that monitor website health.
On the other side are “bad bots”. This category includes content scrapers that steal information, spammers that clog forms with junk, and malicious tools used for DDoS (Distributed Denial of Service) attacks.
Where do AI bots fit in? In our opinion, probably somewhere in the middle. You could argue that they can provide a new way for users to discover your site by being cited as a source in an AI search result. But AI results often provide “zero-click” answers, where the user gets the information they need directly from the AI without ever visiting your website at all.
In addition, AI bot scrapers generate a high volume of requests, putting a strain on your hosting and potentially increasing your hosting costs. In a blog post from Barracuda Networks [2], they reported that one website they tracked received over half a million requests from an AI bot in a single day. We’ve personally seen websites getting crawled by AI bots so aggressively that it caused the server to crash over and over again.
It’s worth noting that the vast majority of Cloudflare customers, over 80%, choose to block AI bots completely [4]. And the more popular a website is, the more likely it is to block AI bots.
What should a website owner do?
Deciding whether to block AI bots from your website depends on your specific goals for visibility/SEO, content protection, and server performance. While search engine bots like Google are essential for traditional search rankings, AI bots serve a different purpose and you may decide that they are a net negative for your site.
Thankfully, there are ways to block these AI bots. Cloudflare, for example, offers a one click feature to block AI bots[5]. AI bots can also be blocked individually via robots.txt, which can be easily implemented in WordPress using the Dark Visitors plugin[6]. Dark Visitors maintains an excellent database of AI agents on their website, so you can understand the purpose of each bot and make an informed decision on whether to block it or not.
References
- https://www.akamai.com/resources/infographic/fraud-and-abuse-report-2025
- https://blog.barracuda.com/2025/04/02/threat-spotlight-gray-bots-gen-ai-scraper-bots-targeting-web-apps
- https://radar.cloudflare.com/ai-insights?dateRange=52w
- https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/
- https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/