TL;DR: Crawl management means steering Google’s crawler towards the pages you want fetched and away from URLs that waste its time. For most sites that is basic housekeeping: a reliable server, clean internal links, no dead ends, and the right tool for each job. Google’s crawl budget guidance is aimed at very large or fast-changing sites, so on a smaller site an unindexed page is more likely a content, linking or technical problem than a crawl budget one.
Before a page can appear in Google, Google has to find it and fetch it. Crawl management is the work of making that easy for the pages that matter and pointless for the ones that don’t. It covers what Google can reach, what it finds when it gets there, and how quickly your server answers.
It sits one step before indexing, so it can’t guarantee a page will be indexed or rank. A page can be crawled and never indexed. If you want the basics of how the two stages differ, start with what crawling and indexing mean in SEO.
What does crawl management involve?
Google defines crawl budget as the set of URLs it can and wants to crawl. Its documentation splits that into two parts, and crawl management works on both.
- Crawl capacity is how much crawling your server can take. Google says server errors and slow responses lower the limit, so Google crawls less.
- Crawl demand is how much Google wants to crawl. It varies with a site’s size, how often it changes, page quality and relevance. Google also says that, without guidance, it tries to crawl all or most of the URLs it knows about, so every junk URL competes for attention with a page you care about.
In practice that means two jobs: keep your server healthy so capacity isn’t held back, and tidy up what Google knows about so its attention lands on useful pages.
Do you need to worry about crawl budget?
Probably not. Google describes its crawl budget guide as an advanced guide for three kinds of site: those with over a million unique pages that change about weekly, those with over 10,000 unique pages that change daily, and those where a large share of URLs show as “Discovered – currently not indexed” in Search Console. Google calls those numbers a rough estimate, not exact thresholds.
For most businesses, we’d look at content quality, internal linking, accidental noindex tags and technical errors before suspecting crawl budget. If your site does fit one of Google’s three descriptions, our guide to managing crawl budget on larger websites goes much deeper than this post.
Which tool does which job?
Most crawl management mistakes come from using one tool to do another tool’s job. This is how the main ones differ.
| What you want | What to use | The catch |
| Stop Google fetching a section | robots.txt | It isn’t a way to keep a page out of Google. Google can still index a blocked URL if other sites link to it. |
| Keep a page out of search results | A noindex tag or header | Google has to crawl the page to see the tag, so it still costs a crawl. If robots.txt blocks the page, Google never sees the noindex. |
| Remove a page for good | A 404 or 410 status | Google treats these as a signal that the page is gone and gradually crawls it less. A “soft 404”, an error page that returns 200 OK, keeps being crawled. |
| Move a page to a new address | A permanent (301) redirect | Google treats it as a signal that the new URL should be the canonical one. Point the old URL straight at the final destination. |
| Consolidate duplicate URLs | A redirect or rel=canonical | These are signals, not commands, so Google may still choose a different version. Don’t use robots.txt for this. |
| Help Google find your pages | An XML sitemap | It doesn’t guarantee crawling or indexing. Small, well-linked sites may not need one. |
If you want more detail on any of these, we cover what robots.txt is and how it works and HTTP status codes and SEO separately, and how often to submit a sitemap if that is your question.
What wastes crawling in practice?
These are the problems worth checking for, roughly in the order they tend to matter.
- Server errors and slow responses. Google says 5xx and 429 errors make its crawlers slow down temporarily, and it speeds up again gradually once the server responds normally.
- Soft 404s. Google says these keep being crawled and waste budget. Search Console flags them when a page looks like an error or is empty but returns 200 OK.
- Duplicate and near-duplicate URLs. Filters, parameters and product variations can produce many addresses for the same content. Consolidate them with redirects or canonicals rather than leaving Google to crawl them all.
- Long redirect chains. By default Google’s crawlers follow up to 10 redirect hops, so a short chain won’t stop a page being crawled. But every hop is another request and a slower journey for visitors, so link and redirect straight to the final URL. Our guide to redirect mapping covers the planning side.
Wildcat Example: On a new site we worked on for a UK business offering outsourced and fractional support services, Google was slow to index the commercial pages, even though we couldn’t find any technical fault. Improving E-E-A-T signals across the site didn’t make a difference straight away. Over the following months we built links through a mix of directories and guest posts, and the pages were indexed within a few weeks. Stronger internal linking helped too, because it gave Google more routes to discover them.
Nothing in that case was stopping Google from fetching the pages. What changed was how easily Google could find them and how much reason it had to treat them as worth indexing. It’s one site, and several changes overlapped, so it’s an example of the pattern rather than a rule.
How can you tell whether crawling is a problem?
Start with the Page indexing report in Search Console. Two statuses look alike but mean different things. “Discovered – currently not indexed” means Google knows the URL exists but hasn’t crawled it yet. “Crawled – currently not indexed” means Google fetched the page and has not added it to the index. The first can point to a crawling constraint. The second is a question about the page itself, because Google has already fetched it, so more crawling won’t fix it.
“Most crawl problems we find on smaller sites are housekeeping, not budget. On one new site, the developers moved to www. but left every internal link on the old address, so around half of Google’s requests hit a redirect. We pointed every link straight at the live URL.”
Jamie Stowe, Senior SEO Account Manager
If you suspect your server is the bottleneck, the Crawl Stats report shows how Google’s requests have been answered, including response types and average response time. Google describes it as aimed at advanced users and says sites with fewer than a thousand pages generally don’t need it.
Top Tip: Before you block a section of your site “to save crawl budget”, check Google is actually running short. Google says freed-up crawl budget isn’t passed to other pages unless a site is already hitting its crawl capacity limit, so unless your site is already hitting that limit, a block tidies things up but is unlikely to speed up crawling of the rest. If the pages you care about are already indexed, crawl budget is unlikely to be your problem.
Crawl management FAQs
Is crawl management the same as crawl budget?
No. Crawl budget is the amount of crawling Google is prepared to do on your site. Crawl management is what you do to influence where that crawling goes. The housekeeping is worth doing on any site, but only the largest sites need to think about the budget itself.
Does Google crawl a site on a set schedule?
No. Google weighs how often to recrawl on a URL-by-URL basis, with its systems aiming to recrawl documents often enough to pick up changes, so there isn’t one crawl frequency for a whole site. Our post on how often Google crawls your site covers what that looks like for a typical business.
Does a sitemap make Google crawl my site more?
It helps Google discover URLs, but Google says it doesn’t guarantee everything in it will be crawled and indexed. Google also says small sites (roughly 500 pages or fewer) and sites that are comprehensively linked internally may not need one.
How Wildcat Handles Crawl Management
If you think crawling is holding your site back, our technical SEO service starts with a Technical SEO Audit early in the campaign. That includes reviewing how easily Google can move through your site, covering internal linking, site structure and sitemap formatting, and auditing indexing and caching. Where a site is large enough for crawl budget to matter, the service can also include work to maximise it. The audit findings set the priorities for the first few months of technical work.
To get started, tell us what you need through our contact page and we’ll arrange a free consultation.
