Discussion starter: a publisher and community SEO question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.
- Crawling
- News SEO
Pay-per-crawl and AI licensing deals: has any publisher you know earned meaningful revenue, or is blocking still the only real lever?
Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.
1 reply
-
adminAdmin
A small number of large publishers have signed AI licensing deals that appear to be worth serious money, but for most publishers licensing and pay-per-crawl income so far is small or non-existent. The deals that exist have gone mainly to organisations with large archives, strong brands, or news content that AI companies need for current answers. Pay-per-crawl systems, such as the model Cloudflare introduced in 2025 and marketplaces that broker access, give publishers a mechanism to charge, but a mechanism is not revenue until AI companies choose to pay at scale. Blocking is not the only lever, but for most publishers it remains the main source of leverage: it is what makes a negotiation possible, and it now comes in more precise forms than a blanket block.
The levers publishers actually have
Lever What it does Trade-off Blocking training crawlers Keeps content out of future model training, via robots.txt or CDN rules Little direct cost; reduces how much models learn from you Blocking AI search crawlers Keeps content out of AI search answers on that platform Loses citations and referral traffic from that platform Google’s Search generative AI control Site-level opt-out of AI Overviews, AI Mode and AI Overviews in Discover, rolled out worldwide on 31 August 2026 No traffic or impressions from those features; Google says it is not used as a ranking signal elsewhere in Search Pay-per-crawl and licensing marketplaces Set a price for crawling or access Revenue depends on AI companies agreeing to pay Direct licensing deals Negotiated payments for access, often with attribution terms Available mainly to large or specialised publishers Collective licensing and industry standards Shared terms and machine-readable licensing signals Early stage; depends on adoption by AI companies Why most publishers see little licensing revenue
- AI companies have alternatives. For most topics, many sources say similar things, so no single mid-size publisher is essential.
- Deals favour scale and uniqueness. Large archives, breaking news, specialist data and strong brands have negotiating power; general content does not.
- Payment mechanisms are new. Pay-per-crawl systems work only if crawlers participate, and adoption by the largest AI companies has been limited.
- The value of a crawl is hard to price. A single article’s contribution to a model or an answer is difficult to measure, which makes pricing contentious.
Blocking as leverage, used precisely
The useful change in 2025 and 2026 is that blocking no longer has to be all or nothing. Publishers can separate training from search: block training crawlers while allowing AI search crawlers that send referrals, as OpenAI’s separation of GPTBot and OAI-SearchBot allows. CDN providers now offer controls aimed at mixed-use crawlers. And Google’s site-level control lets publishers opt out of its generative AI features while staying in ordinary Search. Each choice trades visibility for control, and each should be made per platform based on the referrals and value that platform actually provides.
A decision framework
- Measure what each AI platform gives you: referral sessions, subscriptions or revenue from those sessions, and citations.
- Measure what it takes: verified crawler requests and bandwidth cost.
- Separate purposes: training, search and user-triggered fetching have different value to you.
- Decide per platform and purpose, and revisit quarterly as referral patterns and deals change.
- Register with licensing marketplaces and signal licensing terms where it costs little, so you are in position if payment at scale arrives.
Hypothetical example:
A specialist trade publisher finds that AI search referrals are small but convert to subscriptions at a good rate, while training crawlers account for most AI bot bandwidth and send nothing back. It blocks training crawlers, keeps AI search crawlers allowed, stays in Google’s AI features because they produce visible impressions for its brand, and registers with a pay-per-crawl programme through its CDN. The programme earns very little in its first year. The publisher treats it as an option for the future rather than a revenue line, and uses its data on crawler volume in a licensing conversation with an AI company that values its archive.
Watch for this:
Opting out of Google’s AI features or blocking AI search crawlers can reduce brand visibility where readers increasingly discover publishers. Model the traffic cost before making the change, and make it reversibly, so you can compare results over a few months rather than committing permanently on principle.
What would change the picture
Licensing revenue could become meaningful for more publishers if regulators require payment or transparency, if industry standards for machine-readable licensing are widely honoured, or if AI companies need fresh, reliable content enough to pay for it at scale. None of those has happened broadly yet. Until then, most publishers should treat licensing as upside, keep precise control over which crawlers they allow, and focus revenue plans on the audiences AI platforms still send.
For most publishers, meaningful licensing revenue has not arrived. Use precise blocking and Google’s opt-out as leverage and control, measure each platform’s value, register for payment mechanisms cheaply, and plan the business around readers rather than around AI payments that may or may not come.
Need help with this on your own site?
Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.
Add a reply Cancel reply
Replies are open to approved members. Log in or apply to join.