Comprehensive List of AI Search Engine Crawlers
Comprehensive List of AI Search Engine Crawlers This is a comprehensive list of AI-related crawler user agents, including search engine crawlers known to feed AI systems. Major AI Company Crawlers OpenAI GPTBot ChatGPT-User OAI-SearchBot Anthropic ClaudeBot Claude-Web Anthropic-ai Google Googlebot GoogleOther Google-CloudVertexBot Google-Extended Microsoft/Bing bingbot msnbot Meta FacebookBot Meta-ExternalAgent Meta-ExternalFetcher ByteDance Bytespider Other Commercial AI […]