Crawl Speed and Frequency

Crawl budget is the number of pages on your website a web crawler will crawl within a given timeframe.

Each time you hit the publish button, Google needs to crawl and index the content for it to start appearing in search results. Given the scale and volume of content on the internet, crawling becomes a valuable resource that needs to be budgeted and rationed for the most efficient use.

To put it simply, it’s hard for Google to crawl and index every single page on the internet everyday. So, Google crawls each website according to its assigned budget.

2.6.2 How Is Crawl Budget Assigned to Websites?

Crawl budget is assigned to websites based on two factors — crawl limit and crawl demand.

Crawl Limit

This is a website’s capacity and/or willingness to be crawled.

Not every website is built to be crawled everyday. Crawling involves Googlebot sending requests to your website’s server that, if done too frequently, may stress the server’s capacity.

Also, not every publisher wants their site crawled continuously.

Crawl Demand

Crawl demand is a measure of how often a particular page wants to be (re)crawled. Popular pages or pages that are updated frequently need to be crawled and recrawled more frequently.

2.6.3 Why Should Publishers Care About Crawl Budget?

If Google can’t crawl and index your content, that content will simply not show up on search results.

That being said, crawl budgets are generally only a concern for medium to large publishers who have more than 10,000 pages on their website.  Smaller publishers shouldn’t need to worry overly about crawl budgets.