Generate a valid robots.txt file to control how search engines crawl your website.
Configure rules and generate your file.