Create and validate robots.txt files for your website. Block AI scrapers, control crawler access, and add sitemap references.
Enter any page URL on your domain to simulate crawling. The engine parses wildcards (*), matches prefixes, and calculates rule weight precedence (longest rule matches win) to verify crawl eligibility.
Click a preset to start with a recommended configuration, or build custom rules from scratch. The 'SEO Friendly' preset is recommended for most websites.
Add User-agent rules for specific bots or use * for all bots. Add Disallow paths for sections you want to hide from crawlers like /admin or /private. Add Allow rules for exceptions within disallowed sections.
Add your sitemap URL in the Sitemap section. This helps search engines discover all your pages faster. Most sites have their sitemap at /sitemap.xml.
Use the URL Tester to verify specific URLs are crawled or blocked as expected. Enter any page URL to see exactly which rule applies to it.
Click Download to get your robots.txt file, then upload it to the root of your website (e.g. yoursite.com/robots.txt). Verify it is accessible in a browser before submitting to Google Search Console.
Tools are great for checkups, but real results require custom, persistent strategy. Get a comprehensive SEO audit directly from a world-class expert.