Step 1
Choose a template
Select a starting point such as Allow All, Block All, Block Admin Areas, SEO Optimized, Development/Staging Site, or Custom Configuration.
Free SEO Tool
Create a custom robots.txt file for your website in seconds. Choose a beginner-friendly template, add your sitemap, control crawler access, and copy or download a ready-to-use robots.txt file.
Use this robots.txt generator to create crawler rules for WordPress, Next.js, ecommerce websites, staging sites, blogs, and custom business websites. Whether you want to allow all search engines, block admin areas, add a sitemap, or prevent staging pages from being crawled, this tool helps you build a clean robots.txt file without writing everything from scratch.
Sample output
User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlA robots.txt file tells search engine crawlers which parts of your website they can access. It is commonly used to manage crawl behaviour, reduce unnecessary crawler requests, block admin or internal paths, and point search engines to your sitemap.
This free robots txt generator helps you create a valid robots.txt file quickly. Start with a template, customise your rules, add your sitemap URL, then copy or download the final file.
Do not use it to hide private files, passwords, customer data, or sensitive pages. Some crawlers may ignore robots.txt rules, and blocked URLs can still appear in search results in some cases if they are discovered through other links.
Use authentication, noindex tags, or server-level protection when you need to protect private content.
Choose a template, customise crawler rules, add your sitemap, then copy or download a ready-to-use file.
Use the robots.js option if you want to add robots rules directly into a Next.js project instead of manually uploading a static robots.txt file.
User-Agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Step by step
Creating a robots.txt file is simple. Follow these steps to generate and publish your file correctly.
Step 1
Select a starting point such as Allow All, Block All, Block Admin Areas, SEO Optimized, Development/Staging Site, or Custom Configuration.
Step 2
Choose which user agents should follow the rule. For most websites, User-agent: * applies the rule to all major crawlers.
Step 3
Add paths you do not want crawlers to access, such as admin folders, internal search result pages, temporary files, or staging content.
Step 4
Include your sitemap location so search engines can discover your important pages more easily.
Step 5
Copy the generated output or download it as a robots.txt file.
Step 6
Place the file at the root of your domain, such as https://example.com/robots.txt. Do not upload it to a random folder like https://example.com/assets/robots.txt.
Step 7
Check the file in your browser and review your rules carefully before publishing changes to a live website.
A robots.txt file is a plain text file that gives instructions to web crawlers. Search engines use crawlers to discover and understand pages across the web. With robots.txt, you can tell crawlers which areas of your website they are allowed or not allowed to request.
For example, you might allow search engines to crawl your public pages while blocking admin areas, search result pages, cart pages, checkout pages, or duplicate content sections.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThis tells all crawlers that they can access the website and also gives them the sitemap location.
Examples
Use these examples as a starting point. Always customise the domain, paths, and sitemap URL for your own website.
Use this if you want search engines to crawl your entire website.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlUse this for development or staging websites that should not be crawled. Do not use this on a live website unless you intentionally want to block crawling.
User-agent: *
Disallow: /This is a common WordPress robots.txt setup. It blocks the admin area while allowing admin AJAX requests that some WordPress themes and plugins may need.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlThis example works for a simple Next.js website where all public pages can be crawled and the sitemap is available.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlMany websites block internal search result pages because they can create thin, duplicate, or low-value URLs.
User-agent: *
Disallow: /search/
Disallow: /*?s=
Disallow: /*?q=
Sitemap: https://example.com/sitemap.xmlFor ecommerce websites, you may want to block carts, checkout pages, customer account pages, and internal search pages.
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xmlIf you run a WordPress website, your robots.txt file should usually allow search engines to crawl your public posts, pages, categories, and important assets. You may want to block admin pages, login pages, internal search results, and unnecessary parameter URLs.
Avoid blocking important folders such as theme assets, JavaScript, CSS, or images unless you have a specific reason. Search engines may need those files to understand how your pages render.
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlFor a Next.js website, you can use either a static robots.txt file or a robots.js / robots.ts file depending on how your project is structured.
For more advanced projects, use the Next.js metadata route to generate crawler rules dynamically. This is useful when your sitemap URL, environment, or domain changes between development, staging, and production.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlYou should use a robots.txt file when you want to manage how search engines and other crawlers access your website.
Robots.txt should not be used as a privacy or security solution. It does not password-protect content, remove indexed pages, or guarantee that every bot will follow your rules.
For sensitive content, use proper authentication, server-level restrictions, password protection, or remove the content from public access. For pages that should not appear in search results, use a noindex directive instead of relying only on robots.txt.
Robots.txt and noindex are different. Robots.txt tells crawlers whether they are allowed to crawl a URL. Noindex tells search engines not to include a page in search results.
If you block a page with robots.txt, search engines may not be able to crawl the page and see a noindex tag on it. That is why robots.txt is not always the right tool for removing a page from search results.
Follow these best practices before publishing your robots.txt file.
User-agent: *
Disallow: /This blocks all crawlers from your entire website. It is useful for staging websites, but dangerous for live websites that need to rank in search engines.
Your robots.txt file should be available at https://example.com/robots.txt.
The rule Disallow: / blocks the whole site. Only use it when you intentionally want to stop crawling, such as on a staging website.
Robots.txt is publicly accessible. Anyone can visit your robots.txt file and see the paths you have listed.
Adding a sitemap line helps search engines discover your XML sitemap.
If you block important service pages, blog posts, or landing pages, search engines may have trouble crawling them.
This tool is useful for:
After generating your robots.txt file, you may also need to create or update your sitemap, review site performance, or improve the technical setup around your website.
FAQ
A robots.txt generator is a tool that helps you create a robots.txt file without manually writing every crawler rule. You can choose a template, add blocked paths, include your sitemap, and copy or download the final file.
Upload your robots.txt file to the root of your website. For example, if your website is https://example.com, the file should be available at https://example.com/robots.txt.
Robots.txt controls crawling, not guaranteed indexing. If you want to prevent a page from appearing in search results, use a noindex tag or remove the page from public access.
Yes, adding your sitemap URL to robots.txt is a common best practice. It helps search engines find your XML sitemap more easily.
A common WordPress setup blocks /wp-admin/ and /wp-login.php while allowing /wp-admin/admin-ajax.php. You should also include your sitemap URL.
Yes. This tool can generate a standard robots.txt file and also provide robots.js style output for Next.js projects.
Not every website needs a complex robots.txt file, but most websites benefit from having a clean file that allows important pages, blocks unnecessary paths, and includes the sitemap URL.
No. Robots.txt is not a security tool. It is publicly visible and some crawlers may ignore it. Use authentication, password protection, or server-level restrictions for private content.
Disallow tells crawlers not to crawl a specific path. For example, Disallow: /wp-admin/ asks crawlers not to access the WordPress admin area.
User-agent: * means the rule applies to all crawlers that follow robots.txt instructions.
For a staging site, you can use User-agent: * and Disallow: /. However, password protection is safer because robots.txt does not guarantee privacy.
You can add rules for specific AI crawlers, but robots.txt depends on crawler compliance. It should be treated as a request, not a guaranteed block.
A robots.txt file is only one part of technical SEO. If you need help with website structure, indexing, analytics, performance, or conversion-focused website improvements, Website Builder Central can help you build a cleaner and more search-friendly website.