🔨 List all IP ranges from: Google (Cloud & GoogleBot), Bing (Bingbot), Amazon (AWS), Microsoft, Oracle (Cloud), GitHub, Facebook (Meta), OpenAI (GPTBot) and other with daily updates.
-
Updated
Sep 10, 2024 - Shell
🔨 List all IP ranges from: Google (Cloud & GoogleBot), Bing (Bingbot), Amazon (AWS), Microsoft, Oracle (Cloud), GitHub, Facebook (Meta), OpenAI (GPTBot) and other with daily updates.
Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.
A crawler that crawls search engine! 😎 Usable for collecting site with dorks and wildcards. Also provides output in web interface with more than 3 API endpoints!
Add a description, image, and links to the bingbot topic page so that developers can more easily learn about it.
To associate your repository with the bingbot topic, visit your repo's landing page and select "manage topics."