Skip to content
SEO Community
  • Knowledge
  • Discussions
  • Experiments
  • Case Studies
  • Experts
  • Events
  • About
Log In Apply to Join
SEO Community
  • Knowledge
  • Discussions
  • Experiments
  • Case Studies
  • Experts
  • Events
  • About
Apply to Join Log In
  1. Home
  2. Knowledge
  3. Crawling

Search topic

Crawling

Practitioner discussions, experiments, case studies and insights about Crawling, each attributed to the member who wrote it.

  • All (33)
  • Discussions (33)
Discussion
  • Crawling

AI crawler requests now exceed Googlebot on some sites. When server cost is the constraint, how do you decide which bots to allow, throttle or block?

Decide bot by bot on value against cost, using verified data. Search engine crawlers are essential and should be served well.

Niraj Raut
Niraj Raut
1 replies 4 days ago
Discussion
  • Crawling

Has launching hundreds of thousands of new templated pages ever slowed crawling of your existing important pages?

It can when crawl capacity is limited and new URLs draw attention from existing pages. Launch in batches, prioritise key pages and gate on quality.

Niraj Raut
Niraj Raut
1 replies 5 days ago
Discussion
  • Crawling

Switching SEO platforms wipes years of ranking and crawl history. How are you preserving baselines that survive vendor changes?

Own your data: export rankings, crawls and backlinks to storage you control on a schedule, use tool-independent metrics and run both platforms in…

Niraj Raut
Niraj Raut
1 replies 6 days ago
Discussion
  • Crawling

On an enterprise domain with several busy subdomains on shared infrastructure, have you seen a heavy UGC subdomain reduce crawling of the commercial one?

It can, indirectly: crawling is largely per host, but heavy UGC crawling on shared servers can slow responses, so Google crawls the commercial host…

admin
adminAdmin
1 replies November 12, 2026
Discussion
  • Crawling

Changelog and release-notes pages: indexable asset or crawl waste? Have they ever ranked for anything that mattered?

A well-structured changelog is an indexable asset, not crawl waste, for most SaaS products.

admin
adminAdmin
1 replies November 11, 2026
Discussion
  • Crawling

Calendar and date-archive crawl traps: do you block them in robots.txt, cap the range server-side, or change the links? Which held up long term?

The durable fix is changing how the calendar generates links and returning a 404 for out-of-range dates, with a robots.txt rule kept as a backstop.

admin
adminAdmin
1 replies November 11, 2026
Discussion
  • Crawling

After moving to a new host or CDN, have you seen Googlebot’s crawl rate reset and ramp up slowly? How long until it matched old levels?

A crawl dip after a host or CDN move is common and usually recovers in days to a few weeks on a fast, reliable set-up. If not, the set-up is usually…

admin
adminAdmin
1 replies November 10, 2026
Discussion
  • Crawling

Google can pause crawling if robots.txt returns server errors. Has a CDN or WAF rule on robots.txt specifically ever caught you out?

Yes, and it is one of the more painful ways a well-meaning security rule can hurt search, because the effect is sitewide and easy to miss.

admin
adminAdmin
1 replies November 8, 2026
Discussion
  • Crawling

After a six-hour outage, how quickly did crawl rate and rankings recover for you, and did serving 503 rather than 500 make any visible difference?

Short outages usually cause a crawl dip that recovers within days, with little ranking effect; 503 is correct, but differs little from 500 for six…

admin
adminAdmin
1 replies November 6, 2026
Discussion
  • Crawling

Has removing Shopify apps and their scripts improved crawling or rankings for you, or only speed scores?

In most cases the measurable effect is on speed, user experience and conversion rather than on crawling or rankings directly.

admin
adminAdmin
1 replies November 2, 2026
Discussion
  • Crawling

Crawl frequency from logs as an SEO KPI: a useful leading indicator, or vanity?

Crawl frequency is a useful leading indicator in specific situations and a vanity metric as a general KPI.

admin
adminAdmin
1 replies November 2, 2026
Discussion
  • Crawling

Pay-per-crawl and AI licensing deals: has any publisher you know earned meaningful revenue, or is blocking still the only real lever?

A few large publishers have deals that appear worth serious money, but most earn little or nothing so far; blocking remains their main, not only…

admin
adminAdmin
1 replies November 1, 2026
  • 1
  • 2
  • 3
  • Next page

Related topics

  • AI Search
  • Content SEO
  • Technical SEO
  • Indexing
  • eCommerce SEO
  • Local SEO
  • SEO Analytics
  • Enterprise SEO
  • Entity SEO
  • Agency
  • Digital PR
  • International SEO
Browse the full knowledge hub
SEO Community

A free community for people who work in SEO and modern search.

Community

  • Knowledge
  • Discussions
  • Experiments
  • Case Studies
  • Insights
  • Experts
  • Events

About

  • About the Community
  • About Niraj Raut
  • Community Guidelines
  • Contact

Legal

  • Privacy
  • Terms
  • Cookie Policy

Account

  • Apply
  • Log In

© 2026 SEO Community. Member contributions remain the work of their authors.

Independent from, and complementary to, nirajraut.com.np.