Crawler groups
User-agent lines define which crawler receives the rules that follow.
robots.txt content
EditorPaste your robots.txt content above (plain text, not a file).
The essentials
What is robots.txt?
robots.txt is a plain text file that gives web crawlers instructions about which parts of a website they can request. It must live at the root of the host, such as example.com/robots.txt, and it can contain separate rules for different crawlers.
Site owners use it to reduce unnecessary crawling, keep bots away from duplicate or low-value pages, and guide search engines toward important content. It does not hide pages from people and should never be used as a security measure.
Read the complete beginner's guidePath matching
Allow and Disallow patterns are compared against each URL path and query.
Specific rule wins
The longest matching path decides access, with Allow winning equal ties.
Why validate
Small rules can have a big impact
A missing slash, an overly broad wildcard, or a crawler-specific group can change which pages bots reach. Testing real URLs before publishing helps catch accidental blocks and confirms that exceptions work as intended.
Common mistakes to check
- Blocking the entire site with Disallow: /
- Putting full domain names inside path rules
- Expecting wildcard rules to apply to a more specific crawler group
- Using robots.txt to protect private or sensitive pages
FAQ
Frequently asked questions
Everything you need to know before testing crawler access on your website.