Does your website need crawl budget work?

Start with a missing or delayed page, rather than a target number of Google visits. If pages are being crawled on the day they are published and your site does not change rapidly at scale, Google says keeping the sitemap current and checking Page Indexing is generally enough. Who the guidance is for.

A sitemap is a list of pages you provide to search engines. Page Indexing is the Search Console report Google recommends for monitoring indexing. Crawling and indexing are separate steps, so a successful crawl does not guarantee inclusion in search. Google's explanation.

A hypothetical shop example

Imagine a furniture retailer launching a seasonal collection. The owner wants the new product pages found, while the website also offers several ways to sort the same collection. I would ask the team to review those two groups separately before buying another SEO tool.

Write down the important pages, when they changed and what appears wrong. Keep this list small enough to inspect individually. If you cannot yet identify a crawling problem, start with a broader SEO audit and decide where the evidence points.

What actually determines crawl budget?

Crawl budget is the collection of URLs Google can and wants to crawl. A URL is a page or resource address. Google identifies two main elements: crawl capacity limit and crawl demand. Google's definition.

Capacity: what your server can handle

The server is the system that delivers your website. Google calculates a capacity limit to avoid overwhelming it, taking account of simultaneous connections and how long they remain open. Stable responses can allow that limit to rise. Crawl capacity explained.

Demand: what Google wants to revisit

For Googlebot, the crawler used for Google Search, demand depends on factors including site size, update frequency, quality and relevance. Known duplicate URLs can consume crawling time. Low demand can mean less crawling even when capacity remains available. Crawl demand explained.

Use those two categories in your project brief. Put slow responses under delivery problems and unnecessary page variants under URL inventory. Ask for evidence in each category before approving a larger hosting bill or a sitewide blocking rule.

How do you read the Search Console Crawl Stats report?

Open your website property in Google Search Console, then choose Settings and Crawl stats. The report requires a Domain property or a URL-prefix property at the root level. A property limited to a folder does not meet that requirement. Report access requirements.

Start with Host status, which describes availability problems Google encountered. Then inspect response codes, average response time and example URLs. The report also groups requests by file type, crawl purpose and Googlebot type. Report navigation.

A crawl request is not the same as a unique page. Repeated requests count separately, and totals include resources hosted on your site. Example URLs are a sample, so absence from that list does not prove a page was never requested. How the data is counted.

Save the reporting period and your observations before changing anything. Separate what you can see from what you suspect. For example, write “response time rose after the release” before asking the developer to investigate whether the release caused it.

Which URLs should you consolidate, block or remove?

Give each URL group a clear purpose. Google recommends consolidating duplicate content, controlling unwanted crawling and returning appropriate responses for permanently removed pages. Treat these as different decisions. URL inventory recommendations.

Use a short decision list

• Repeated content: identify versions that can be consolidated.
• Unwanted sorting variants: consider crawl blocking when consolidation is unsuitable.
• Permanently removed pages: check for a 404 or 410 response.
• Long redirect chains: review unnecessary intermediate steps.

These treatments follow Google's crawling recommendations. Use the duplicate content guide to prepare the first group for review, with a preferred destination and a reason for each decision.

Do not confuse noindex with crawl blocking

Noindex tells Google not to index a page after requesting it. It therefore does not prevent that request. Robots.txt controls crawling, but Google warns against temporarily blocking pages to redirect crawl budget elsewhere. Google's blocking guidance.

Before changing rules, list pages that must remain accessible. Check proposed patterns against both wanted and unwanted examples. Hand the developer the robots.txt setup guide alongside those examples, rather than a vague instruction to block filters.

In the hypothetical furniture shop, I would review price sorting separately from useful category pages. Ask the merchandising team which pages answer a distinct customer need. Record that decision before choosing the technical treatment.

For retired addresses, review the actual response rather than just the message visitors see. Google advises correcting soft 404 problems and avoiding long redirect chains. Removal and redirect guidance. Assign relevant cases to your redirect implementation checklist.

How does server speed affect crawling?

Consistently healthy responses can support increased crawl capacity. Slow responses, server errors such as 5xx codes, and rate limiting such as 429 responses can reduce it. Response codes are the server's short technical messages about a request. Server health and crawl capacity.

Send your hosting provider the affected dates and example addresses. Ask whether the issue concerns the whole website or a particular page group. Request a diagnosis before deciding that a more expensive package is the answer.

Include unchanged content in the review

Google recommends supporting 304 responses where appropriate. This means “not modified” and lets Google reuse a stored version of unchanged content, saving server resources. Faster loading and rendering may also let Google read more content. Loading and caching recommendations.

Ask the developer to document how the proposed change will be checked. Include an unchanged page and a genuinely updated page in that review. Keep the business requirement clear: current product information must still be delivered correctly.

Also maintain the sitemap and include lastmod information for updated content, as Google recommends. Lastmod records the last modification. Sitemap guidance. Use the XML sitemap guide for the implementation task.

What should you do this week?

Choose one documented problem and one owner. The following is a suggested working plan, not a promise that Google will recrawl or index pages within a week.

1. Collect evidence: save the reporting period, affected pages and current indexing observations.
2. Group the problem: separate availability failures, repeated URLs, removed content and discovery concerns.
3. Agree on treatment: write what should change and which pages must remain accessible.
4. Apply a focused correction: record the implementation date and the person responsible.
5. Review delivery: check the changed behavior, then schedule another look at crawling and indexing.

A practical developer brief

Use this wording: “Please investigate this page group using the attached report examples. Explain the cause, proposed change, pages affected and checks after release. Include a way to restore the previous behavior if the change affects important pages.”

For the follow-up, compare equivalent reporting periods and keep campaign launches in your notes. Review whether the identified errors remain and what happened to the important pages. Avoid making a larger request count your only success condition.