SynthTrail

Robots.txt versus what agents actually do

A robots.txt file asks agents to stay away from certain paths. Whether they do is a different matter. SynthTrail shows the paths agents really opened, so you can compare behaviour with your rules.

Install the snippet See a sample trail

$9.99 a month. No contract. Cancel any time.

The problem: rules nobody checks

You write a robots.txt, add some paths and assume they are respected. Well-behaved crawlers honour it. Many agents acting for a user, and any that choose not to, do not. Without evidence, you do not know which is which.

How to compare

Open a session trail and read the list of paths the agent opened. Then check them against the paths in your robots.txt. A path you disallowed that shows up in a trail was opened regardless of your request.

What this page is not

This is not a live testing tool: SynthTrail does not fetch your robots.txt and score agents automatically. It gives you the real trail to compare with. And remember that robots.txt is not security; protect private content with sign-in.

A simple routine

Publish a clear robots.txt
Review agent session trails each week
Note disallowed paths that were opened
Protect anything private with sign-in, not a request

Follow the trail on your own site

Paste one snippet and see the synthetic visitors that use your site. Card details are entered on Stripe, never on this site.

Install the snippet