Robots.txt Generator
Create a custom robots.txt file for your website. This file tells search engines and AI crawlers which pages they can or cannot crawl on your site.
Enter your website’s full URL including https://
Enter the URL of your XML sitemap
Common Restrictions:
AI & LLM Crawlers (optional):
Control access for popular AI crawlers like GPTBot, Perplexity, Claude, and Google-Extended.
Other Specific Bots (optional):
Use carefully. Disallowing Googlebot or Bingbot can remove your site from search results.
Add any additional paths you want to block from search engines