Create a perfect robots.txt file for your website in seconds. Control crawlers, block bad bots, add your sitemap — then download and upload instantly.
A robots.txt file is a plain text file placed at the root of your website — accessible at https://yoursite.com/robots.txt — that instructs search engine crawlers and bots which pages or sections of your site they are allowed or not allowed to visit. It follows the Robots Exclusion Protocol (REP), an industry-standard used by Google, Bing, Yahoo, and thousands of other crawlers since 1994. Every time a search engine bot visits your website for the first time, the very first file it requests is your robots.txt. Without one — or with a misconfigured one — you risk wasting Google's crawl budget on unimportant pages, accidentally exposing sensitive admin areas, or even blocking your entire site from being indexed.
A properly configured robots.txt is one of the foundational elements of technical SEO. Here is why every website needs one:
Sitemap: directive helps Google discover all your pages faster.Your robots.txt file must live in the root directory of your website — the same folder that contains your homepage. It is only valid at the top-level domain. Key rules:
https://yoursite.com/robots.txthttps://yoursite.com/blog/robots.txt — crawlers won't find it hereyoursite.com/robots.txt in a browser after uploadA misconfigured robots.txt can cause serious SEO damage. These are the most common mistakes:
/wp-content/ entirely.Disallow: / for all bots means Google cannot index anything.Sitemap: to help Google discover your pages./Admin/ and /admin/ are treated differently.These two are often confused but serve very different purposes:
Select a preset that matches your website platform — Default, WordPress, Shopify, Google Only, or Block AI Bots — to instantly populate the rule builder.
Add or remove Allow/Disallow rules for each User-agent. Use the rule builder to target specific bots and paths with precision.
Enter your sitemap URL (e.g. https://yoursite.com/sitemap.xml) so Google can discover all your pages directly from robots.txt.
Click Generate to create your file. The live validator checks for common errors and warnings before you download.
Download the robots.txt file and upload it to the root of your website via FTP, cPanel, or your CMS dashboard.
Use the robots.txt Tester in Google Search Console to verify your rules work correctly before they go live.
| Website Type | Block These Paths | Always Allow |
|---|---|---|
| WordPress Blog | /wp-admin/, /?s=, /tag/, /page/ |
/wp-admin/admin-ajax.php, /wp-content/ |
| E-commerce (Shopify) | /cart, /checkout, /account, /orders |
All product & category pages |
| SaaS / Web App | /dashboard/, /api/, /settings/, /login |
Marketing & landing pages |
| News / Media | /print/, /amp/ (if duplicate), /search |
All article & category pages |
| Portfolio / Brochure | /staging/, /admin/, /cgi-bin/ |
All public-facing pages |
See your robots.txt file built in real-time as you configure rules. No manual editing required.
Live validation catches common mistakes before you upload — like accidentally blocking your whole site.
One click adds disallow rules for 30+ known scraper bots, spam crawlers, and AI training bots.
WordPress, Shopify, and other CMS-specific presets get you the right configuration instantly.
Everything runs in your browser. Your site URL and rules are never sent to any server.
No account, no watermark, no limits. Generate and download as many robots.txt files as you need.
robots.txt file is a plain text file at your website root that tells search engine crawlers which pages to crawl and which to skip. You need it to protect crawl budget, keep sensitive pages out of search results, block bad bots, and point search engines to your sitemap. Without it, Google may waste time crawling admin pages instead of your important content.robots.txt helps Google allocate its crawl budget efficiently — directing it to your important pages. Blocking unnecessary pages like admin areas, duplicate content, and filter pages means Google spends more time crawling your valuable content. However, accidentally blocking important pages with Disallow: / can completely prevent them from appearing in search results.robots.txt to the root directory of your website — the same folder that contains your index.html or homepage file. It must be accessible at exactly https://yourdomain.com/robots.txt. You can upload via FTP using FileZilla, via your hosting cPanel File Manager, or through your CMS dashboard (Yoast SEO for WordPress, etc.). After uploading, visit the URL in a browser to confirm it is accessible.Disallow in robots.txt tells crawlers not to visit a URL at all — they will not crawl it. Noindex is a meta tag placed on the page itself that tells crawlers not to include the page in search results even if they do crawl it. An important implication: if you Disallow a URL, Google cannot read its noindex tag — so Google may still list the URL in search results based on links pointing to it. For pages you want excluded from search, use noindex without blocking in robots.txt.User-agent rules for known AI and scraper bots. Common ones include GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended (Google AI training), anthropic-ai (Anthropic/Claude), and Bytespider (TikTok). Use our Bad Bot Blocker or Block AI Bots preset to add disallow rules for all of them automatically with one click.robots.txt is a public file that anyone can read by visiting yoursite.com/robots.txt. Ironically, listing sensitive paths in robots.txt can reveal them to hackers looking for admin panels or private directories. To actually protect sensitive content, use server-level authentication (password protection), firewalls, or access control rules — not robots.txt.search.google.com/search-console/robots-testing-tool. After any change, always test a few of your most important URLs to make sure they are not accidentally blocked.