Crawler Rules
Robots.txt Generator
Create rules to manage web crawlers and define which parts of your site search engines can access.
Target Host / Domain
User-Agent
Allow all crawlers full access
Disallowed Paths (comma separated or new lines)
/admin, /private, /tmp
Sitemap XML URL
Generate robots.txt
Content copied to clipboard!