Robots.txt Generator
Quick Presets
Sitemap URL
Disallow Paths
Paths you want to block from all crawlers (e.g. /admin/, /wp-login.php)
Allow Paths
Override a disallow rule for specific paths (leave empty if not needed)
Block Bad Bots
Free Robots.txt Generator
A robots.txt file tells search engine crawlers which pages and sections of your website they are allowed or not allowed to index. It is one of the most important technical SEO files and should exist at the root of every website.
Our generator lets you build a robots.txt with presets for common site types (WordPress, e-commerce, Google-only), add custom allow and disallow rules, and block common bad bots that scrape your content without contributing to traffic.
robots.txt Syntax Reference
| Directive | Example | Meaning |
|---|---|---|
| User-agent: * | User-agent: * | Applies to all robots |
| User-agent: name | User-agent: Googlebot | Applies to specific bot |
| Disallow: /path/ | Disallow: /admin/ | Block this path from crawling |
| Allow: /path/ | Allow: /public/ | Override disallow for this path |
| Sitemap: url | Sitemap: https://site.com/sitemap.xml | Tell crawlers where sitemap is |
| Crawl-delay: n | Crawl-delay: 10 | Seconds between requests (not Googlebot) |
Frequently Asked Questions
Where do I upload the robots.txt file?
Upload robots.txt to the root of your website — it must be accessible at https://yoursite.com/robots.txt. In WordPress, it is already there. For custom sites, upload it via FTP or your hosting file manager to the public_html folder.
Does robots.txt prevent pages from appearing in Google?
Disallowing a page in robots.txt prevents Google from crawling it, but not necessarily from indexing it if another site links to it. To fully remove a page from Google, use the noindex meta tag inside the page's HTML head section.
Should I block WordPress admin and login pages?
Yes. Disallowing /wp-admin/ and /wp-login.php prevents search engines from crawling admin pages that have no SEO value and helps keep login pages out of search indices. These pages are still accessible — robots.txt only affects crawlers, not browsers.