WebTools - The very handy web based tools
Login
Register
Robots.txt Generator
Generate a custom robots.txt file for your website to control search engine crawlers.
User-agent:
Enter the user-agent (e.g., *, Googlebot, Bingbot).
Disallow (one per line):
Enter paths to disallow, one per line (e.g., /admin/, /private/).
Allow (one per line):
Enter paths to allow, one per line (e.g., /public/, /images/).
Sitemap URL:
Enter the URL of your sitemap (optional).
Generate Robots.txt
Generated Robots.txt: