— LLM Crawler Permissions for AI Search Visibility —

Allow reputable AI crawlers that improve search, product discovery,

and visibility in AI-driven search engines.

User-agent: Google-Extended Allow: /

User-agent: GPTBot Allow: /

User-agent: PerplexityBot Allow: /

User-agent: ClaudeBot Allow: /

— Block high-volume scrapers or non-beneficial crawlers —

User-agent: CCBot Disallow: /

Default: allow all other compliant crawlers

User-agent: * Allow: /