Tokenize2 is a plugin which allows your users to select multiple items from a predefined list or ajax, using autocompletion as they type to find each item. You may have seen a similar type of text entry when filling in the recipients field sending messages on facebook or tags on tumblr.
Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules …
A fast, simple, multilingual tokenizer
Taiwanese Hokkien Transliterator and Tokeniser
Text segmenter and tokeniser for Danish, English and other languages. Reads an RTF or flat text file and outputs the text, one line per sentence & optionally tokenized.
A Lightweight Word Piece Tokenizer
Javascript port of HappyFunTokenizer.py by Christopher Potts and HappierFunTokenizing.py by H. Andrew Schwartz
Taiwanese Hokkien Transliterator and Tokeniser
Add a description, image, and links to the tokeniser topic page so that developers can more easily learn about it.
To associate your repository with the tokeniser topic, visit your repo's landing page and select "manage topics."