SEO tool · Checks live sites
Robots.txt Checker
Test if robots.txt blocks a URL for Googlebot, Bingbot or AI crawlers like GPTBot and ClaudeBot. See the rule that matches and the sitemaps listed.
Robots.txt Generator
Create a robots.txt file: block folders, add your sitemap, and allow or block AI crawlers such as GPTBot, ClaudeBot and PerplexityBot.
Runs in your browser. Your data never leaves your device.
How to use the Robots.txt Generator
- Enter your sitemap address and the folders to block.
- Choose the rules for AI crawlers.
- Download robots.txt and upload it to your site's root folder.
About this tool
Build a correct robots.txt without learning the syntax. Choose the folders to keep crawlers out of, add your sitemap address, and decide separately for AI search crawlers and AI training crawlers. Copy the file or download it and upload it to the root of your site.
GuideHow to Rank in ChatGPT Search and Google AI Overviews (A Practical GEO Guide)
Frequently asked questions
Where does robots.txt go?
At the root of the domain, so it opens at https://yoursite.com/robots.txt.
More seo tools
All 31 free tools →
SEO and AI search
SEO tool · Runs in your browser
llms.txt Generator
Create an llms.txt file for your website in a minute: name, summary, details and sections of links in the right Markdown format. Free, with an example.
SEO tool · Runs in your browser
Meta Tag Generator
Generate HTML meta tags for SEO and sharing: title, description, robots, canonical, Open Graph and Twitter cards, with a Google preview. Free.
SEO tool · Checks live sites
Open Graph Checker
Preview how a link looks when shared on Facebook, WhatsApp, LinkedIn and X, and check every Open Graph and Twitter Card tag.