Robots.txt Generator
Create clean, search-engine compliant robots.txt files with custom user-agents and sitemap directives.
User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /private/ Sitemap: https://burnjet.com/sitemap.xml
Quick Reference & Instructions
Simple steps, pro tips, and execution details
Provide Inputs
Type, paste, or select your values in the form fields below.
Instant Live Analysis
Calculations and formatting happen automatically with zero delay as you type.
Copy or Use Output
Copy results or apply the clean output directly to your projects.
How It Works
Configures crawl permissions for Googlebot, Bingbot, and other web crawlers.
Frequently Asked Questions
Common questions about calculations, assumptions, and edge cases.
Disallow tells crawlers not to request a URL at all, but the page can still get indexed (often shown with no description) if other sites link to it. A noindex tag lets crawlers fetch the page but tells them not to include it in search results - use noindex when you need guaranteed removal.