TL;DR: There’s no fixed schedule. Google decides how often to crawl each page on its own, based mainly on how often the page changes, how popular it is and how well your server copes. Some new pages are picked up the same day they’re published; for others, Google says crawling can take anywhere from a few days to a few weeks. You can see exactly when Google last crawled any page with the URL Inspection tool in Google Search Console, and you can make a recrawl more likely, but you can’t force one. Crawling more often doesn’t, on its own, improve your rankings.
If you’ve updated a page and Google still shows the old version, or published something new that hasn’t turned up yet, the question is how often Google comes back. Here’s what decides that, and how to check your own pages.
How Often Does Google Crawl a Website?
Googlebot, Google’s crawler, works page by page rather than site by site, so a page that changes often and is well linked will usually be visited far more often than an old page nothing links to.
Google says crawling and indexing “take some time and rely on many factors”, and that it can’t make predictions or guarantees about when, or if, a page will be crawled. Even when you ask Google to recrawl a page, its own guidance says this can take anywhere from a few days to a few weeks.
You’ll see figures like “every three to four weeks” quoted online. Google doesn’t publish an average, so treat any single number as one site’s experience, not a rule for yours. Some sites see new pages crawled the same day they’re published, and Google’s crawl budget guide treats that as a sign a site has nothing to worry about.
What Decides How Often Google Crawls Your Pages?
Google’s crawl budget guide describes two sides to this: how much Google wants to crawl your pages, and how much your server can comfortably handle. In practice, four things matter most.
How often the page changes. Google says its systems want to recrawl pages “frequently enough to pick up any changes”. A page that’s updated regularly gives Google a reason to keep coming back, and Google states plainly that if a site has important content that changes often, it crawls it often, regardless of the size. A page that hasn’t changed in years is likely to be checked less often.
How popular the page is. According to Google’s crawl budget guide, URLs that are more popular on the internet tend to be crawled more often. Google doesn’t spell out exactly what counts as popular, so don’t read it as your visitor numbers. For most businesses, we recommend focusing on the part you control most directly: how well the page is linked, from your own site and from others.
How well your server copes. If your pages load quickly and reliably, Google says it may be able to crawl more of your site. If it starts hitting server errors, it backs off: Google automatically slows crawling when it sees a significant number of 500, 503 or 429 errors, and speeds up again once they’re fixed.
Whether Google can find the page at all. Google uses links to find new pages to crawl, and recommends that every page you care about is linked from at least one other page on your site. A robots.txt file tells crawlers which URLs they can access, so anything blocked there won’t be crawled (our guide to what robots.txt is covers how to check yours). Your XML sitemap helps too, but it’s a hint rather than an instruction.
The most common cause of pages not being crawled that I see, is Google not actually being able to find the page. Many clients will upload a page, but not add it to the navigation menu or internally link from any other page on the site. This makes it hard for Google to actually find the page in the first place.
Jamie Stowe, Senior SEO Account Manager
How Do I Check When Google Last Crawled My Site?
You don’t need to guess. Google Search Console shows you directly, at page level and site level.
For a single page, use the URL Inspection tool. Paste the page’s address into the search bar at the top of Search Console. The “Last crawl” field shows the last time Google crawled that page, and everything else the tool reports is based on that crawled version. If you’ve changed the page since, “Test live URL” shows you what Google would see now.
For the whole site, use the Crawl Stats report. You’ll find it under Settings, then Crawl stats, and it’s only available for a property that covers your whole site, such as a domain property. The Crawl Stats report shows how many requests Google has made to your site over the past 90 days, how your server responded, and whether Google has had any trouble reaching it (the host status). Google describes it as a report for advanced users and says sites with fewer than a thousand pages shouldn’t need it. It’s most useful when you’ve noticed a site-wide drop and want to know whether server problems coincide with it.
Top Tip: Before you request indexing, compare the “Last crawl” date in URL Inspection with the date you last changed the page. If Google crawled it after your change, asking for another crawl won’t achieve much: Google has already seen the new version, and what you’re waiting on is indexing, not crawling. If the last crawl is older than your change, add a link to the page from one Google already crawls often (check when your homepage or a main category page was last crawled, in the same tool), and make sure your sitemap’s lastmod date shows when the page really changed.
How Can I Get Google to Crawl a Page Sooner?
You can’t force a crawl, but you can make one more likely.
- Request indexing for the pages that matter most. In URL Inspection, choose “Request indexing”. This is best kept for a handful of important new or updated pages. There’s a limit on how many requests you can make, and Google says requesting a recrawl of the same URL again won’t get it crawled any faster. A request doesn’t guarantee the page will be indexed either.
- Link to the page from somewhere Google visits often. A new page linked from your homepage, a main category page or a related, well-established post gives Googlebot a direct route to it on its next visit. A page with no internal links relies on your sitemap or on links from other sites.
- Keep your sitemap accurate. For lots of new or updated URLs at once, a sitemap is the better route than individual requests. Google only uses the lastmod date if it’s consistently and verifiably accurate, and ignores the changefreq and priority values completely, so a sitemap that claims every page changed today can mean Google stops relying on your dates altogether. You don’t need to resubmit it every time: our post on how often to submit a sitemap covers how often resubmitting is needed.
- Fix server errors and slow responses. If the Crawl Stats report shows server errors or availability problems, sort those first. Google slows its crawling to protect a struggling server, so no amount of linking or requesting will fully make up for it.
“One of my clients, a local Audiologist, had a lot of pages that weren’t being indexed by Google. We conducted a technical SEO audit and tried to crawl the site using Screaming Frog. Our crawl also didn’t find these pages, despite many of them being in the navigation menu. After a manual review, we discovered that the slide-out burger menu that the site used was being implemented by Wix in JavaScript, and therefore the links weren’t available to crawl in the HTML. By replacing this menu with a static navigation bar, as well as adding internal linking throughout the site, we quickly saw results with multiple first-page rankings on pages that previously didn’t rank at all.”
Jamie Stowe, Senior SEO Account Manager
Crawling FAQs
Does Google crawling my site more often improve my rankings?
Not in itself. Google says improving your crawl rate won’t necessarily lead to better positions in search results. Crawling is how Google finds and refreshes your pages. Where they rank depends on what it finds.
Do I need to worry about crawl budget?
Probably not. Google’s crawl budget guide is aimed at sites with over a million pages that change weekly, or over 10,000 pages that change daily. If your new pages are crawled soon after publishing, it isn’t your problem. If you do run a large site, our post on managing crawl budget for larger websites goes into more detail.
If Google has crawled my page, does that mean it’s indexed?
No. Crawling is Google fetching the page; indexing is Google deciding to store it so it can appear in results. A page can be crawled and not indexed, and Google says submitting an indexing request doesn’t guarantee the page will be added. Google also can’t predict or guarantee when, or if, a page will be indexed, so there’s no fixed schedule for that either. Our guide to crawling and indexing in SEO explains the difference.
How Wildcat Handles This
Questions about crawling usually come up as part of the technical SEO work we do for clients. We review a site’s crawlability early in a campaign as part of a technical SEO audit, looking at the things that shape how Google crawls it: internal linking, site structure, the XML sitemap, indexing, and site speed.
To look at this properly, the most useful starting point is access to your Google Search Console property, since that’s where Google’s own crawl data lives. What you get back is a clear picture of whether Google is reaching and revisiting the pages that matter to your business, what’s getting in the way if it isn’t, and which fixes to prioritise, rather than a list of every technical warning a tool can produce.
If important pages on your site seem to be slow to appear or update in Google, you can arrange a free consultation, and we’ll talk through what’s going on.
Chloe Robinson
Content Strategist & Team Leader
