How to Create a Robots.txt File Correctly

A robots.txt file tells search engine web crawlers (like Googlebot and Bingbot) which paths on your server they are allowed or forbidden from requesting. Configuring it correctly protects your server bandwidth and crawl budget.

Generate a compliant robots.txt file with the Burnjet Robots.txt Generator.

Standard Robots.txt Directives #

• User-agent: Specifies which crawler the rule applies to (* for all bots, or Googlebot, Bingbot).
• Disallow: Specifies directory paths bots should not crawl (e.g., /admin/, /api/).
• Allow: Explicitly permits crawling of sub-paths within a disallowed directory.
• Sitemap: Provides the absolute URL to your XML sitemap.

Example: Recommended Modern Robots.txt #

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /preview/

Sitemap: https://burnjet.com/sitemap.xml

Step-by-Step: Using the Burnjet Robots.txt Generator #

Step 1: Open the Tool #

Visit Burnjet Robots.txt Generator.

Step 2: Choose Presets or Add Custom Rules #

Select whether to allow all crawlers or block sensitive directories.

Step 3: Add Your Sitemap URL #

Input your full XML sitemap path (e.g., https://example.com/sitemap.xml).

Step 4: Download or Copy #

Save the generated text as robots.txt and upload it to your website's public root directory.

Related SEO Tools #

• XML Sitemap Generator: Build your site's URL index.
• Meta Tag Generator: Configure page-level noindex tags.