Robots.txt Tester
Test whether URLs are allowed or blocked by your robots.txt rules instantly with this free online robots.txt tester.
Keep going with other free EnjoySEOs tools. No signup, no limits.
No tools match your search. Try a different keyword.
Understand and Test Your Robots.txt Rules
Use the Robots.txt Tester above to check how your pasted rules affect individual URLs for a selected crawler. This guide explains the main directives, how to interpret test results, and what to review before changing your website’s crawl rules.
What Is a Robots.txt Tester?
A robots.txt tester checks whether a URL or path matches an Allow or Disallow rule in a robots.txt file. It helps website owners, developers, and SEO professionals find rules that may prevent search-engine crawlers from accessing important pages or resources.
The robots.txt file is a plain-text set of crawler instructions usually published at the root of a website, such as https://example.com/robots.txt. It can apply different rules to different user-agents, including a general group for crawlers that do not have a more specific matching group.
This online tester works with the robots.txt content you paste into the tool. It checks the entered paths against those rules in your browser; it does not automatically fetch the live robots.txt file from a domain or confirm what a crawler has actually accessed. Copy the current file from your site first when you want to check its published configuration.
Useful for SEO checks
Test whether important landing pages, blog posts, product URLs, images, or site sections are unintentionally blocked from crawling.
Useful before publishing changes
Review a proposed robots.txt file with sample URLs before uploading it, especially after a redesign, migration, or CMS configuration change.
Important Robots.txt Directives
Robots.txt uses groups and directives to describe which URL paths a compliant crawler is requested to access or avoid. These are the rules you will most often need when testing a file.
| Directive | Purpose | Example |
|---|---|---|
User-agent |
Identifies the crawler group to which the following rules apply. | User-agent: * |
Disallow |
Requests that a matching crawler avoid the specified path. | Disallow: /private/ |
Allow |
Allows a matching path, including a more specific path inside a disallowed directory when the crawler supports the directive. | Allow: /private/public.pdf |
Sitemap |
Provides the absolute URL of an XML sitemap. It is normally written outside the user-agent groups. | Sitemap: https://example.com/sitemap.xml |
* |
In a supported path pattern, matches a sequence of characters. | Disallow: /*?sort= |
$ |
In supported path patterns, marks the end of the URL path pattern. | Disallow: /*.pdf$ |
Crawl-delay |
Not part of the standardized core rules and not supported by every crawler. Google ignores this directive. | Check the documentation for the specific crawler. |
A crawler generally uses the most specific matching user-agent group. If no specific group matches, it can use the User-agent: * group. When multiple rules match a URL, the most specific matching rule is important; for Google-style matching, a more specific Allow rule can permit a path that a broader Disallow rule would otherwise block.
Example Robots.txt File
This sample gives all crawlers a general set of rules, keeps a WordPress administration directory from being crawled while allowing a useful AJAX endpoint, blocks a private path, and declares a sitemap. Replace the example domain and paths with values that reflect your own site structure.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /private/
Disallow: /*?sort=
Disallow: /*.pdf$
Sitemap: https://example.com/sitemap.xmlYou can test paths such as /wp-admin/, /wp-admin/admin-ajax.php, /private/report, and /documents/guide.pdf in the tester above. The outcome depends on the exact rules in the text you provide, so review the matched rule shown with each result.
Disallow: / can prevent compliant crawlers from crawling the entire site for the selected user-agent.How to Use the Robots.txt Tester
- Get the current file. Open the robots.txt URL for the exact site origin you want to check and copy its contents. If you are testing a draft, use the proposed text instead.
- Paste the robots.txt content. Place the directives in the tool’s Robots.txt content field. You can also load the sample file to see how the tester behaves.
- Select a user-agent. Choose the crawler you want to test, such as Googlebot or Bingbot, or enter a custom user-agent token. The selected group matters because different crawlers may have different rules.
- Enter URLs or paths. Add one URL or path per line. You can enter a path such as
/blog/article/or a full HTTP or HTTPS URL. The tool tests up to 200 non-empty lines at a time. - Run the test and inspect the results. Click Test URLs. Review each Allowed or Blocked result and the matching rule or reason, then adjust the draft and test again as needed.
- Save carefully. Use Copy or Download to keep the text. Before publishing, double-check the final file and test important URLs after the file has been uploaded to your site's root directory.
What Do the Results Mean?
| Result | Meaning | What to check next |
|---|---|---|
| Allowed | No matching block was found, or an applicable Allow rule won. | Confirm the live file is the same as the text tested and that the URL uses the intended host and protocol. |
| Blocked | A matching Disallow rule applies to the tested path. | Review the rule and crawler group before changing it. Make sure the blocked area is genuinely intended to be excluded from crawling. |
| Invalid | The tester could not interpret that input as a supported URL or path. | Use an HTTP/HTTPS URL or a path starting with /, and remove pasted labels or other extra text. |
| Warning | The parser found a line that may be malformed, unsupported, or ignored by some crawlers. | Check the relevant directive and verify its behavior for the crawler you are targeting. |
Robots.txt vs. Noindex: Crawling Is Not Indexing
One of the most important SEO distinctions is that robots.txt controls crawling, not guaranteed removal from search results. A search engine may know a URL exists through links even when its crawler cannot fetch that page. In that case, the URL may still appear in results without the page content or a normal snippet.
| Method | What It Does | Best Used For |
|---|---|---|
| robots.txt Disallow | Asks compliant crawlers not to crawl matching URLs. | Managing crawler access and reducing crawling of selected URL patterns. |
| Meta robots noindex | Instructs a crawler that can fetch the page not to index it. | Pages that should be crawlable so search engines can read the noindex instruction, but should not appear in search results. |
| Password protection / access control | Restricts access to authorized users rather than relying on crawler cooperation. | Private, confidential, or account-only content. |
If your goal is to keep a page out of search results, do not simply block it in robots.txt and assume it will disappear. Choose an appropriate indexing directive or access restriction, and make sure crawlers can access the page when they need to read a noindex instruction.
Common Robots.txt Mistakes to Avoid
1. Blocking the Entire Website by Accident
A rule such as User-agent: * followed by Disallow: / broadly blocks crawling for the general group. Check staging or development rules carefully before copying them to a production website.
2. Using Robots.txt to Hide Sensitive Content
A disallow rule does not stop a visitor from opening a URL and does not guarantee that the URL will be absent from search results. Use authentication and server-side permissions for private or sensitive material.
3. Blocking Resources That Search Engines Need
Restricting important CSS or JavaScript files can make it harder for search engines to render and understand a page. Review resource paths before adding broad directory-level rules.
4. Forgetting How User-Agent Groups Work
A crawler-specific group may take precedence over the general wildcard group. Test with the actual crawler token that matters instead of checking only User-agent: *.
5. Testing Draft Text but Not Verifying the Published File
A local test validates only the text entered into the tester. After deployment, check the live robots.txt file at the correct origin and retest key URL patterns against its actual contents.
6. Forgetting the File Location or Sitemap URL
Publish robots.txt at the root of the intended site origin, for example https://example.com/robots.txt, rather than inside a subdirectory. A Sitemap directive should use an absolute HTTP or HTTPS URL to the XML sitemap.
Frequently Asked Questions
Is This Robots.txt Tester Free?
Yes. You can paste robots.txt rules, choose a user-agent, test URL paths, inspect the results, and copy or download the text without paying for the tool.
Does the Tester Fetch My Website’s Live Robots.txt File?
No. It checks the content entered in the tool. To test the live configuration, open the robots.txt file for your site, copy its contents into the editor, and then test the relevant URLs.
Can Robots.txt Prevent a Page from Being Indexed?
Not reliably. Robots.txt can prevent crawling, but a blocked URL may still be indexed if it is discovered through other signals. Use an appropriate noindex directive or access restriction when your goal is to keep content out of search results.
Should Every Website Have a Robots.txt File?
A site can be crawled without one, but a robots.txt file can be useful when you need explicit crawl rules or want to point crawlers to a sitemap. Create one carefully and avoid adding rules unless you understand their effect.
Does Robots.txt Apply to Every Subdomain?
No. The file applies to its site origin. For example, the robots.txt file for example.com is not automatically the same file as the one for shop.example.com. Check the exact hostname and protocol you want to control.
Can All Bots Be Forced to Obey Robots.txt?
No. Robots.txt is a convention followed by compliant crawlers, not a security barrier. A malicious or non-compliant bot can ignore the rules, so sensitive endpoints must be protected on the server.