SEO tool · Runs in your browser
Robots.txt Generator
Create a robots.txt file: block folders, add your sitemap, and allow or block AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.
Robots.txt Checker
Test if robots.txt blocks a URL for Googlebot, Bingbot or AI crawlers like GPTBot and ClaudeBot. See the rule that matches and the sitemaps listed.
This tool asks our server for public information only. Nothing you enter is stored.
How to use the Robots.txt Checker
- Enter the website address and press Fetch to load its robots.txt.
- Choose a crawler and type the path to test, such as /blog/my-post/.
- Press Test path and read the verdict and the rule that matched.
About this tool
Enter a website and the checker downloads its live robots.txt, shows which search and AI crawlers may visit it, and lists its rules and sitemaps. Then test any path for one crawler: you see Allowed or Blocked and the exact Allow or Disallow line that decided it. The checker follows Google's rules: the most specific (longest) matching rule wins, and the * and $ wildcards work. Crawlers include Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended and CCBot.
How to read a robots.txt file
robots.txt is a plain text file at the root of a site (example.com/robots.txt) made of groups. Each group starts with a User-agent line naming a crawler, followed by Disallow lines for paths to skip and Allow lines for exceptions. A crawler uses the group that names it, or the * group if none does. Sitemap lines can go anywhere. In this example every crawler may visit the site except /admin/ and the search pages, /admin/help/ stays open, and GPTBot may crawl nothing:
User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/help/
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlFixing "Blocked by robots.txt" in Search Console
Test the URL from the report here to find the Disallow line that blocks it. If the page should be in Google, remove or narrow that line, or add an Allow for the path, upload the new file, then press Validate fix in Search Console. If you want the page out of Google, unblock it and add a noindex tag instead: Google can't see a noindex on a page it isn't allowed to crawl, so a blocked page can still appear in results with no description.
Is robots.txt still a thing?
Yes, more than ever. Search engines still read it, it became an official internet standard (RFC 9309) in 2022, and AI companies now publish their crawler names so site owners can choose: GPTBot and ClaudeBot collect training data, while OAI-SearchBot and Claude-SearchBot fetch pages for AI search answers. Google-Extended controls whether Google may use your content for Gemini, without affecting Google Search.
Testing tools, generators and site builders
Google retired the old robots.txt Tester in Search Console at the end of 2023. Its robots.txt report (under Settings) shows the file Google last fetched and any errors, and this page tests any path for any crawler. To write a new file, use the robots.txt generator below. On Wix the file is made for you and you add rules in the SEO settings with the Robots.txt Editor; on Shopify you edit the robots.txt.liquid theme template.
GuideHow to Rank in ChatGPT Search and Google AI Overviews (A Practical GEO Guide)
Frequently asked questions
Is robots.txt legal?
robots.txt is not a law; it is a voluntary standard (RFC 9309). Major search engines and AI companies say they follow it, but it doesn't stop anyone technically, so never rely on it to hide private pages; put them behind a login. Whether ignoring it breaks a law depends on the country and the case.
How do I fix a blocked by robots.txt error?
Find the blocking Disallow line with this tester, remove it or add an Allow for the page, upload the file, then press Validate fix in Search Console.
Is robots.txt still a thing?
Yes. Every major search engine reads it, and it's now how you choose which AI crawlers, such as GPTBot, ClaudeBot or PerplexityBot, may use your site.
How do I read a robots.txt file?
Find the User-agent group for the crawler you care about (or *), then read its Disallow and Allow lines. When several match a URL, the longest rule wins.
Does Disallow remove a page from Google?
No. It stops crawling, not indexing. A blocked page can still be listed if other sites link to it. Use a noindex tag on a crawlable page to remove it.
Where does robots.txt go?
At the root of each host: https://www.example.com/robots.txt. A subdomain such as shop.example.com needs its own file.
More seo tools
All 31 free tools →
SEO and AI search
SEO tool · Runs in your browser
llms.txt Generator
Create an llms.txt file for your website in a minute: name, summary, details and sections of links in the right Markdown format. Free, with an example.
SEO tool · Runs in your browser
Meta Tag Generator
Generate HTML meta tags for SEO and sharing: title, description, robots, canonical, Open Graph and Twitter cards, with a Google preview. Free.
SEO tool · Checks live sites
Open Graph Checker
Preview how a link looks when shared on Facebook, WhatsApp, LinkedIn and X, and check every Open Graph and Twitter Card tag.