Skip-Gram Based Taxon Tagger for the TextImager Pipeline
A Java-based gazetteer tagger, developed for the BIOfid project. Recognizes biological entities provided with large lists (gazetters) in texts.
Utilizes a Java ConcurrentHashMap-backed tree-search algorithm parallelized with Java 8 streams that tags arbitrary texts of n words in O(c · n) time by looking up (→ c) each word in a previously created tree.
Each node in the tree represents a word from the given input lists.
All leaves must have a label (usually an URI); any node in the tree may have a label.
Also allows to create skip-grams and abbreviations from input terms.
The tagger is highly suspectible to false positives, such as vernacular names that double as common names of people (espcially prominent in German, eg. Schneider). Please keep this in mind while curating input lists/gazetteers.
This repository is in the process of being replaced by a Rust implemenation: gazetteer-rs
The tool was used in the creation of all BIOfid corpora. Please cite