robots.txt ↔ standard (for voluntary compliance to opt-in) by Web Crawler/bot to access to URL routes
disallow bots to whole site
User-agent: *
Disallow: /