# Default crawler policy User-agent: * Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association # AI / answer-engine crawler policy, June 2026 # Explicitly allow public content for discovery while preserving the same protected-path rules as the default group. User-agent: GPTBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: OAI-SearchBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: ChatGPT-User Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: OAI-AdsBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: ClaudeBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Claude-Web Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Claude-SearchBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Claude-User Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Google-Extended Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: PerplexityBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Perplexity-User Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Meta-ExternalFetcher Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: Amazonbot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: DuckAssistBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: YouBot Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / User-agent: MistralAI-User Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /*/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*filter. Disallow: /*/?*filter. Disallow: /*?*sort_by= Disallow: /*/?*sort_by= Disallow: /*?*pf_*=* Disallow: /*/?*pf_*=* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /*/*?*oseid=* Disallow: /*/*preview_theme_id* Disallow: /*/*preview_script_id* Disallow: /search Disallow: /apple-app-site-association Allow: / # Training-only crawlers: no citation, no referral, full catalogue crawl. # Measured over 30 days: zero sessions referred, while ChatGPT sent 76. User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Diffbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: FacebookBot Disallow: / # Microsoft Copilot uses standard Bingbot; Google Search and Gemini retrieval use standard Googlebot where applicable. Sitemap: https://developdevice.com/sitemap.xml Sitemap: https://developdevice.com/sitemap_agentic_discovery.xml Sitemap: https://developdevice.com/pages/video-sitemap-v8