A ‘crawl budget’ is how much time a search engine such as Google will spend per day going through the URLs of your website and discovering what your pages contain. Your website may not get crawled in full in one day, especially if it is a larger website as the search engines only dedicate so much time to each webpage. So, how do you manage this budget on larger sites?
Key Takeaways:
Here’s a few top tips for managing your crawl budget on larger websites:
- Optimise your site speed, especially on mobile.
- Submit XML sitemaps.
- Monitor crawl reports to keep on top of issues.
- Use a logical website structure and internal linking strategy.
- Use a Robots.txt file.
- Use canonical tags properly.
- Use the ‘noindex’ tag for pages that don’t need to be crawled.
What is a Crawl Budget?
A crawl budget is time and resources used by search engines to go through your website urls and appropriately index these so they can be served on the search engine results pages. Learn more about how often Google would typically crawl your website in our blog ‘How often does Google crawl my site?’.
Factors That Influence Crawl Budget
There are factors that influence the crawl budget a website has, these are especially important to manage if you have a large website such as an ecommerce website with a high volume of pages.
Factors affecting crawl budget are:
Crawl Rate Limit
The crawl rate limit determines how frequently a search engine bot can crawl a website, without overwhelming your website server. There are several factors to this that will be considered by the search engine, these are:
- Server Performance – To prevent overloading the server, a search engine will reduce the number of pages crawled if your server is slow to load, or gives 5xx errors (server unable to handle the request).
- Crawl Health – If a website responds fast to the requests from the search engines this may increase the crawl budget of the website.
Crawl Demand
This is determined by how often a search engine wants to crawl your website. So, even if you have a lot of pages and a good crawl limit, it may only crawl pages that are frequently updated or considered important. Some factors that influence crawl demand include:
- Page popularity – High traffic, good quality backlinks and healthy engagement rates can influence how often a page is crawled.
- Page Updates – Pages that are kept up-to-date will be crawled more frequently to ensure the most up to date information is captured.
- Canonicals & Redirects – If a page is giving mixed signals such as re-direct loops or conflicting canonicals, it may reduce the demand for crawling that page due to the confusion this would cause. To read more on redirects see our blog, Everything you need to know about redirects.
Why Does a Crawl Budget Matter More for Larger Websites?
Larger websites will naturally face more challenges when it comes to their crawl budget due to:
- A higher volume of pages
- A higher risk of duplicate content being detected
- Potential low-value pages being crawled before high-value pages
It is important to be aware of this so you can optimise your website in the right places. If your pages can’t be crawled, then they can’t be indexed and served on the search engines in the first place! Learn more about this in our blog What is crawling and indexing in SEO.
Ways to Optimise Your Website for a Healthier Crawl Budget
There are ways in which you can optimise your website to ensure you have effective crawl management, these include:
- Page loading speeds – Optimse your website to ensure you have adequate loading speeds. Pay particular attention to mobile loading speeds as Google indexes on a mobile first basis. This can help the crawlers to index more pages. Learn how to audit your page speed in our dedicated blog, How to audit your site speed and performance.
- Submit XML sitemaps – Sitemaps give the search engines a structure to follow and can help them prioritise the key pages you want them to index.
- Monitor crawl reports – Using tools such as Google search console you can identify pages where there may be issues and can rectify these quickly.
- Website structure and Internal linking – Using a logical website hierarchy can help a crawler understand your website better and good internal linking structures and the use of breadcrumbs can lead the crawlers to your most important pages.
- Use a robots.txt file – You can disallow search engines from crawling certain pages, such as admin and checkout pages, with a robots.txt file so you are not wasting crawl budget.
- Use canonical tags appropriately – You can guide search engines to a preferred page and also prevent duplicate content by using canonical tags.
- ‘Noindex’ tag – Use the ‘noindex’ tag for pages that you don’t want to rank for and that you don’t want to appear on the search engine results pages.
Manage Your SEO Effectively With Wildcat Digital
SEO strategies contain way more than just keywords and content, there are many building blocks that make up the intricacies of a good SEO campaign. At Wildcat Digital we are a team of experts on PPC, paid social, SEO, and technical SEO. If you’re needing help to increase your visibility and boost your website traffic, get in touch with our team today to discuss your goals.
Thea Chapman
SEO Account Manager
