web-robots

Here are 3 public repositories matching this topic...

Simple robots.txt template. Keep unwanted robots out (disallow). White lists (allow) legitimate user-agents. Useful for all websites.

A Python notebook showcasing the use of Machine Learning for the task of bot detection, with an emphasis on e-commerce sites.

A simple trap for web crawlers

Add a description, image, and links to the web-robots topic page so that developers can more easily learn about it.

To associate your repository with the web-robots topic, visit your repo's landing page and select "manage topics."