Robots.txt Tester

Test your robots.txt file and check whether your URLs are allowed or blocked for different user-agents.

Result

Enter the details and click "Test" to see the result.

How to use this tool

1. Enter the robots.txt

Copy and paste the content of your robots.txt file, or load it from a URL with the "Load from URL" button.

2. Specify the URL

Enter the full URL you want to test (example: https://yourdomain.com/path/page.html).

3. Select the user-agent

Choose the bot you want to simulate (Googlebot, Bingbot, etc.) or use a custom one.

4. Test and analyze

Click "Test" to see whether the URL is allowed or blocked for that specific user-agent.

💡 Tips

  • If there is a group for the specific bot (for example Googlebot), only that group counts and the * group is ignored
  • User-agent: * applies to bots that don't have their own group
  • Patterns with * match any sequence of characters, and $ marks the end of the URL
  • Disallow: with no value allows everything
  • If a URL matches both Allow and Disallow, the longest rule wins, and if they're the same length, Allow wins