You know how AI companies have been gobbling up web content like it's an all-you-can-eat buffet—without giving much (or anything) back to the folks who actually created it?
Yeah... Cloudflare’s officially over it.
They just launched a new marketplace called Pay per Crawl — and yup, it’s exactly what it sounds like.
AI companies will now have to pay every time their bots crawl a website for data.
Here’s how it works (and why it’s kinda genius):
Website publishers are finally in control. They can now:
Charge a micropayment per crawl
Let AI bots in for free (if they’re feeling generous)
Or slam the door shut entirely
Oh, and Cloudflare plays middleman. They’ll handle the money, manage permissions, and keep things running smoothly. No crypto involved… yet. But they’re flirting with the idea of launching their own stablecoin.
And get this: Big-name publishers like TIME, Conde Nast, and The Atlantic are already on board. Plus, going forward, all new websites hosted on Cloudflare will automatically block AI crawlers by default unless the site owner decides otherwise.
Why does this matter?
Well, the numbers Cloudflare dropped are totally insane—if you go by their own data:
Google’s crawler hits a site 14x for every 1 referral
OpenAI? 1,700x
Anthropic?? 73,000x 😳
Translation? AI companies are taking way more than they’re giving.
And with AI agents now doing the "searching" for users, traditional traffic is drying up. Publishers are freaking out, lawsuits are flying, and the old-school business model is circling the drain.
Sure, some licensing deals are happening — but so far? Only the big dogs are cashing in. That’s exactly why Cloudflare’s stepping in: They’re trying to create a system where any publisher, big or small, can get a fair shake.
And yeah, it’s still an experiment for now...But if this takes off? It could be the new business model that saves online publishing from the AI black hole.
Now, will this actually work? Still TBD.
According to CNBC, OpenAI already declined to participate when Cloudflare previewed its plan to block AI crawlers by default—arguing that the content delivery network was inserting itself as a middleman.
But one thing’s clear: if anyone can strong-arm the ecosystem into a fairer model, it’s probably Cloudflare, given that they handle about 16% of global internet traffic (per their own 2023 report).
Bottom line: Cloudflare just drew a line in the sand—and for the first time in a while, publishers might actually have a bit of leverage.
Curious how it all works or want in on the experiment?
👉 Read the full report here .
