Skip to main content

Web Crawler ↔ 🤖 Program to systematically browse & index all public web pages (for Search Engine)

aka Crawl

References​

  1. SEO ↔ (Design Pattern) to optimize Web Page max exposure to Search Engine's Web Crawler

  2. @docusaurus/plugin-sitemap ↔ docu plugin to gen sitemap for site for SEO Web Crawler | pack with classic preset

  3. sitemap ↔ XML Website map for 🤖 Web Crawler

  4. robots.txt ↔ standard (for voluntary compliance to opt-in) by Web Crawler/bot to access to URL routes

  5. Bing Penalty ↔ take Bing Report errors & even warnings VERY seriously - even short/long titles & descriptions - bing will outright refuse to Crawl your site due to "low quality". It doesn't matter if Google indexes your site well - Bing has very different, very strict rules for "quality" (purely based on meta tags/markup) & will outright refuse to index any page that fails it's rules (below).