What should your robots.txt file actually do?

Use it to control crawler access to selected parts of your website. A crawler is an automated visitor that reads pages; Googlebot is Google’s crawler. Google describes robots.txt primarily as a way to manage crawler traffic and avoid unnecessary requests. Read Google’s introduction.

Before opening an editor, write down the outcome you want. “Stop crawling this duplicate section” is a useful brief. “Improve our SEO” needs more detail before anyone changes access rules.

Make a keep-open list first

Hypothetical example: a small furniture shop wants Google to read its product pages, delivery information, and showroom details. Put those pages on a checklist before considering any blocked folders. Ask the person editing the file to check those exact paths afterwards.

Your initial worksheet can be simple:

• The section being considered for a restriction.
• The business reason for restricting it.
• One actual page inside that section.
• One important page that must remain accessible.
• The person responsible for checking the change.

You may hear the phrase crawl budget, meaning the capacity devoted to crawling a site. For this task, focus on a concrete access problem. Google specifically identifies unimportant or similar pages as possible candidates for crawl restrictions. See its stated use cases.

How do you write user-agent, disallow and allow rules?

Start a group with a user-agent directive, which names the crawler receiving the instructions. Put each instruction on its own line. The following meanings come from Google’s rule reference.

A small robots.txt example

Hypothetical example: your shop keeps duplicate browsing pages in a folder called /duplicates/. You want that folder skipped, except for a public explanation page inside it. The paths below are teaching examples, so replace them with paths you have checked.

User-agent: *
Disallow: /duplicates/
Allow: /duplicates/about.html

The asterisk selects a general crawler group, although Google’s AdsBot crawlers require explicit naming. The allow rule creates the narrower exception. Paths not blocked are allowed by default. Google explains these rules and the AdsBot exception.

Do you need to explicitly allow Google bots?

Unblocked files are already available for crawling by default. If you deliberately need a separate Googlebot group, this is a simple example. Default access and named groups are documented here.

User-agent: Googlebot
Allow: /

Do not casually append this to an existing file. Google selects the most specific matching group, so a named Googlebot group does not simply inherit the general group’s restrictions. Review the complete file together. Check group selection before editing.

Compare path spelling carefully. Uppercase and lowercase paths are treated differently, and a single slash after Disallow blocks the whole site for the selected group. Google includes both case rules and full-site blocking examples.

Where should the file go, and what format does it need?

Publish it at /robots.txt directly beneath the website’s root address. Use plain text saved with UTF-8 encoding. A file inside a blog or uploads folder is not the root robots.txt file. Google’s location and format requirements.

Notepad or TextEdit can create the file. Avoid word processors such as Word because their formatting can introduce unwanted characters or a different file format. Google recommends a text editor.

1. Save a copy of any existing file.
2. Prepare the replacement in a plain text editor.
3. Check that its name is exactly robots.txt.
4. Choose UTF-8 when saving.
5. Ask your hosting provider where the website root is if you cannot identify it.

A subdomain is a separate named part of a domain, such as a dedicated shop host. The main website’s rules do not automatically cover it. A robots.txt file applies to its own protocol, host, and port. Google defines that scope explicitly.

If this work is part of a website move, add a robots.txt check to your domain migration plan. Record which host you checked rather than writing only “robots done.”

Can robots.txt hide a page from Google search?

No. Website crawling control and search visibility are different decisions. A blocked address can still appear in search if Google discovers links pointing to it. For excluding a page, Google points to noindex or password protection. Read its visibility warning.

Noindex vs disallow

Noindex is an instruction to leave a page out of search results. Disallow restricts crawler access. Write your intended outcome in ordinary language before asking a developer to choose the mechanism. Google distinguishes these purposes.

Hypothetical example: a consultant has a confidential proposal intended for one client. The brief should say “only the client and our team should access this.” A request to add the proposal to robots.txt does not express that security requirement.

Robots.txt cannot force every bot to obey. For private information, Google recommends other protection methods, such as password protection. See the limitations of crawler instructions.

Also review proposed restrictions on images and other page resources. Google warns against blocking resources it needs to understand the page. Its resource guidance explains why. Include that check when following your image SEO checklist.

What should you check on WordPress and in the sitemap line?

For a WordPress robots.txt change, begin by asking who currently manages the file. Request the existing text and the proposed replacement together. Avoid approving a template without knowing which existing rules it replaces.

Give the site manager a specific brief

Please preserve a copy of the current robots.txt file. Explain every proposed restriction, list an affected page, and confirm that our product, service, and contact pages remain accessible to the intended crawlers.

The upload method depends on the website and server setup. Some hosted platforms expose search settings instead of direct file editing. Google therefore advises checking the provider’s instructions. See its platform and upload guidance.

Add the actual sitemap address

A sitemap identifies content you want crawled. Its optional robots.txt entry must use the full address, rather than a relative path. Google documents the Sitemap instruction.

Sitemap: [replace with your complete sitemap address]

This line is a placeholder, not a finished configuration. Find your real sitemap, open it, and replace the bracketed text. If you have not prepared one, follow the XML sitemap guide first.

How do you test the change and manage crawler load?

Check public access first, then check the rules. Google recommends opening the file in a private browser window and using the Search Console robots.txt report for files already accessible on the site. Follow its testing instructions.

1. Open your website’s /robots.txt path in a private window.
2. Confirm that the expected plain text appears.
3. Review the robots.txt report in Google Search Console.
4. Compare your important page paths against the selected crawler’s rules.
5. Check one blocked path and every intentional exception.
6. Keep the previous version and record why you changed it.

Google describes crawl restrictions as a way to avoid excessive crawler requests, including requests for unimportant pages. That does not make the file protection against bots that ignore it. Its introduction explains both points.

Ask your hosting contact to compare request activity before and after the change. Look for the particular section you intended to restrict. Do not treat the mere presence of a file as proof that the original problem has been resolved.

Include campaign pages in the handover

If you are also working through a QR code campaign guide, add the campaign’s destination page to your review list. Do the same for public offer pages in your referral marketing plan. Decide their intended visibility explicitly before approving a folder restriction.