Social media is evolving. Are you adapting? Connect with a community of brand pros and content creators at Social Media Week, May 12-14 in NYC, to learn how to keep pace with new trends and technology. Register now to save 20% on your pass.
TollBit's latest product, Content Cache, aims to safeguard publishers' revenue streams from AI bots.
A recent report by TollBit, a two-sided marketplace for publishers and AI firms, found that AI companies -- including OpenAI, Perplexity, and Meta -- scraped websites an average of 2 million times in Q4 last year. The study, which analyzed 160 websites spanning national news, local outlets, and tech blogs, revealed that each page was scraped about seven times on average.
"The bot traffic generated by these AI platforms is almost the same as bot traffic of search engine bots that have existed for the last 20 years, which is insane," said Toshit Panigrahi, co-founder, TollBit.
AI bots are a departure from traditional SEO bots deployed by Google or Bing, which deliver traffic and revenue by regularly indexing publisher sites on their respective search engines for a fee. This breed of AI bots scrape content unpredictably, without providing the same return. Tolbit finds that AI search bots, despite their claims, are sending on average 95.7% less referral traffic than traditional Google search.
Gartner predicts traditional search volume will decline 25% by 2026, as AI chatbots siphon away queries -- eroding a key source of publisher traffic and revenue.
Companies are already seeing the impacts of this. In February, Online education company Chegg sued Google, arguing that its AI-generated summaries in Search's AI Overview have decimated its traffic and revenue.
"New bots that have appeared in the past year and a half are hammering sites and publishers are left paying for these bots that are accessing content and not getting any traffic back," Panigrahi said.
Available to all TollBit publisher partners -- titles like Penske Media Corporation and TIME -- Content Cache builds on publishers' existing infrastructure and redirects AI crawlers to the dedicated TollBit subdomain, where AI bots must license content via micropayments
"The subdomain for TollBit has acted as the front door for AI bots," Panigrahi explained. "If they come in through that link, they will be able to license and pay for that article through micro-payments."
TollBit is offering Content Cache to its publisher partners for free for up to 3 million bot accesses per month per site. The company did not directly respond to questions on how it will charge publishers in the future.
Some of TollBit's publisher partners had been requesting a tool like this, according to Olivia Joslin, co-founder, TollBit.
"AI bot traffic on the web is exploding, straining publisher sites and driving up costs," Joslin said. "The value of AI scraping is still unclear. If publishers aren't ready to block bots outright, this at least gives them a way to offset the costs."