Discussion
  • AI Search

Lighthouse now has an agentic browsing audit that checks llms.txt and WebMCP, while Search docs say skip llms.txt. How are you advising clients on that split?

Published 1 reply

Discussion starter: an AI search question, posted with an answer from Niraj Raut to open the thread. If you have dealt with this on a site, reply with what you saw, especially where it differs.

Disclosure: Niraj Raut, who posted this answer, runs the SEO consultancy linked at the end of it.

1 reply

  • admin
    Admin
    Original poster

    The split is smaller than it looks, because the two Google teams are answering different questions. Search Central answers “do I need llms.txt to appear in Google Search, including AI Overviews and AI Mode?” The answer is no: Google’s AI optimisation guide says you don’t need machine-readable files, AI text files or Markdown versions, and that Search ignores them. Lighthouse answers a different question: “how easily can a browser-based agent understand and operate this page?” llms.txt is one small check in an experimental category that is mostly about accessibility, form semantics and layout stability.

    So my advice to clients is: do not sell llms.txt as SEO, and do not report the Agentic Browsing result as a search metric. Fix the accessibility and stability findings because they help users, agents and conversion at the same time. Add llms.txt only where there is a real non-search consumer for it (developer documentation is the clearest case). Treat WebMCP as a pilot for sites with high-value forms and engineering capacity, not a default recommendation.

    What each Google source actually says

    Source Question it answers Position
    Search Central guide to generative AI features (last updated July 2026) What helps a site appear in Google Search and its AI features? Foundational SEO. No llms.txt, special markup, Markdown copies or AI-specific rewording needed. Search ignores these files, and creating them does not harm the site. The guide separately points to agent-friendly site practices and the Universal Commerce Protocol.
    John Mueller, on the Lighthouse check Why is Lighthouse checking it? “It’s not done for search. There’s more to websites than just SEO.” He added that for non-developer sites he doesn’t think it makes much sense, even with more agentic traffic.
    Lighthouse Agentic Browsing scoring (Lighthouse 13.3, May 2026) Can an agent in a browser use this page? Experimental, “based on proposed standards”. The focus is to “gather data and provide actionable signals rather than a definitive ranking.”
    web.dev, Build agent-friendly websites (April 2026) How should developers build for agents? Semantic HTML, a clean accessibility tree, no invisible overlays, stable layouts, and WebMCP as a proposed standard. No mention of llms.txt.

    There is some genuine untidiness. Search Engine Journal reported that llms.txt files briefly appeared on several Google developer properties in December, apparently from a CMS rollout rather than a Search decision, and that Gary Illyes and Amir Taboul said at a Search Central Live event that Google was not pursuing llms.txt. That is inconsistency between product teams, not a hidden Search signal.

    What the Agentic Browsing category checks

    Per Google’s scoring page and third-party write-ups from DebugBear and No Hacks, the category groups its audits roughly like this:

    • Agent-centric accessibility: a filtered set of accessibility checks that matter to machines (names and labels, tree integrity, visibility).
    • Stability: Cumulative Layout Shift, because agents that act on screenshots or element positions break when layout moves.
    • WebMCP integration: whether the page registers tools, either declaratively (annotated HTML forms) or imperatively (JavaScript), and whether those tool definitions are valid. WebMCP entered a Chrome origin trial in June 2026, so on most sites today these audits are likely to return not applicable.
    • llms.txt: checks for a machine-readable summary at the domain root. Where a file exists, write-ups report it being checked for format problems (no H1, too short, no parseable links). One site’s plain-text link list failed until it was rewritten as Markdown links, which matches the llms.txt proposal’s format. Google’s llms.txt audit page says a missing file (a 404) is marked not applicable, because providing one is optional. So the audit can only fail a file you have chosen to publish. That behaviour could change between releases of an experimental category.
    • Agentic Resource Discovery: Lighthouse 13.5 (September 2026) added an audit that validates an ARD catalogue (by default /.well-known/ai-catalog.json) against a proposed schema. It sits alongside llms.txt under agent discoverability. Treat it the same way: another proposed standard, not a Search signal.

    The scoring matters for client conversations. There is no weighted 0 to 100 score, just a pass ratio and per-audit results. Nobody should be chasing a green number here.

    Two different machines reading the same site

    A useful way to explain the split to a client: Search Central is the rulebook for getting listed in a directory. Lighthouse’s new category is closer to an access audit of the shopfront. The access audit tells you whether a particular kind of visitor, one who navigates by labels rather than by eye, can get through the door and use the counter. Passing it does not move you up the directory. Failing it may mean that when such a visitor arrives, it cannot finish the job.

    Google Search pipeline

    Googlebot discovers, crawls, renders and indexes HTML, then ranking and AI features select from the index. An llms.txt file adds nothing this pipeline uses, which is exactly what the Search docs say.

    Agent operating a page

    A browser agent reads the accessibility tree, screenshots and any registered WebMCP tools to click, fill forms and complete tasks for a user. Mislabelled buttons, shifting layout and ambiguous forms cause failed tasks, not lower rankings.

    This is the same pattern practitioners already know from performance. A Lighthouse lab score was never the input to Google’s page experience assessment; Search Console’s Core Web Vitals report uses field data from real users. Lighthouse is a diagnostic tool, and its categories do not map one to one onto ranking signals.

    How I would split the advice by site type

    Site type llms.txt WebMCP Agent accessibility and stability
    Local service or lead generation Skip. No evidence of a consumer that matters. Not yet. Watch the spec. Yes. Quote and booking forms are where agents and humans both fail.
    Publisher or content site Skip, or add only if near-zero cost and someone owns maintenance. No clear use case. Yes, through normal accessibility work.
    Ecommerce Skip for SEO. Watch. The more concrete agentic channel right now is Google’s Universal Commerce Protocol and Merchant Center, not page-level tools. As of September 2026, UCP checkout is available to select merchants, for products eligible in the US, Canada and Australia. Yes, especially variant selectors, cart and checkout.
    SaaS, API or developer docs Reasonable. This is the case Mueller’s comment leaves open, and developer docs are the most plausible place for a coding assistant to use it. Candidate pilot for complex in-app flows. Yes.
    Complex applications (bookings, quotes, insurance, finance) Low priority. Pilot on one high-value form if engineering can follow a changing spec. Yes, first.

    The deciding factor is not the audit result. It is whether an agent completing a task on the site has commercial value, and whether the client has the engineering capacity to maintain something built on a proposed standard.

    Watch for this:

    Once clients see a failed or missing audit in a Google tool, the pressure is to “fix” it. The risks are small but real: an llms.txt with stale links that contradicts the live site, parallel Markdown copies nobody maintains, and reporting time spent on a pass ratio that has no documented link to traffic or revenue. Keep Agentic Browsing results out of SEO KPI reports and put them in the accessibility or product backlog, where they belong.

    How to find out whether agents matter for a specific client

    This is still an evidence-poor area, so I would measure demand before building for it:

    1. Server logs: segment requests by user agent for known AI crawlers and user-triggered fetchers. If you publish an llms.txt, log who requests it. Requests only from SEO tools and auditors tell you something useful.
    2. Analytics: GA4 added an AI Assistant default channel group in May 2026. It undercounts (no-referrer traffic still lands in Direct), but the trend and the landing pages show where AI-driven visits start.
    3. Task tests: run an agent through the three or four journeys that make money (request a quote, book, buy) and record where it fails. Repeat after accessibility fixes. This is a more meaningful measure than the pass ratio.

    Hypothetical example:

    A B2B software client wants llms.txt “because Lighthouse says so”. Fast experiment: publish a well-formed llms.txt on the docs subdomain only, then count requests by user agent for 90 days. Metric: requests from AI agents or assistants rather than SEO tools. Target: a steady, non-trivial share, with sessions or signups traceable to docs pages. If you see that, extend it and keep it maintained. If you only see auditing tools, leave it as a low-maintenance file and stop spending time on it.

    What would change my advice: Google Search documentation saying it uses these files, major agents publicly consuming WebMCP tools outside origin trials, or client logs showing agent traffic that converts. None of those is true today, based on what Google has published.

    A one-question filter for every Agentic Browsing finding

    For each failed or missing audit, ask who benefits if you fix it. If a human user benefits too (unlabelled controls, a broken accessibility tree, layout shift), it goes into the normal backlog and is judged on conversion and accessibility, not on the pass ratio. If the only beneficiary is a hypothetical agent, as with llms.txt or an ARD catalogue on a non-developer site, it waits until your logs show something actually reading it. WebMCP stays a pilot on one form where a completed agent task is worth money, with budget for the spec to change underneath you.

    Need help with this on your own site?

    Niraj Raut works with businesses in Nepal, Australia, the UK, Europe and the US on technical SEO, ecommerce SEO, local SEO and AI search.

    See SEO services

Add a reply

Replies are open to approved members. Log in or apply to join.