Add new bots #36
Merged
Add new bots #36
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Typhoeus wraps libcurl in order to make fast and reliable requests. See: https://github.com/typhoeus/typhoeus
The citoid node.js service generates citation data given a URL, DOI, ISBN, PMID, PMCID or QID. It has a companion extension, Citoid, which aims to provide use of the citoid service to VisualEditor. See: https://www.mediawiki.org/wiki/Citoid
7siters is some kind of link and domain analysis database operating their own spider. See: https://7ooo.ru/siters/
sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. --- This is definitely not a human user agent, and actually you would be well advised to ban any IP address that made requests declaring this user agent! See: https://github.com/sqlmapproject/sqlmap
Pattern is a web mining module for the Python programming language.
---
This seems to be an academic spider. The spider's user agent looks
like this in my logs:
Pattern/2.6 +http://www.clips.ua.ac.be/pattern
Because the word "Pattern" is not very unique and could appear in
some legitimate human user's user agent I suggest using anchoring
the string to the beginning of the line and matching at least one
digit for the version.
See: https://www.clips.uantwerpen.be/pattern
Apparently the Turnitin.com plagiarism scanning service uses both the TurnitinBot and Turnitin user agents. Right now COUNTER-Robots does not block the second one. See: https://turnitin.com/robot/crawlerinfo.html
Drupal is a content management system, not a user. Sometimes people
write plugins that perform harvesting of content into the CMS so we
should ignore these requests. Drupal uses the following user agent:
Drupal (+http://drupal.org/)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds patterns to match the following robot user agents:
Relevant URLs for each user agent included in the patterns file.
Note: I had added a few of these in #34 but only a few were merged. I will re-submit them again here.