umLabeller

umLabeller is an inspection tool for characterizing the semantic compositionality of subword tokenization, based on the morphological information retrieved from UniMorph. Given a word w and its subword tokenization, s = (s₁, ..., s_n) | ∀i s_i ∈ V, umLabeller assigns one of four categories: vocab, alien, morph, or n/a:

• vocabulary subword: the given word w is a subword in the vocabulary as w ∈ V;
• alien composition: the given subword sequence s is an alien subword composition if we find at least two subwords s_i and s_j in s that are not meaningful with respect to the meaning of w;
• morphological composition: the subword sequence s is morphological if it is neither a vocabulary nor an alien subword composition;
• n/a: UniMorph has no information on the word.

umLabeller can characterize over half a million English words and is compatible with most modern tokenizers.

Examples

input word	subword tokenization	output label
jogging	_j ogging	alien
neutralised	_neutral ised	morph
stepstones	_steps tones	alien
swappiness	_sw appiness	alien
swappiness	_swap pi ness	morph
jogging	_jogging	vocab

Installation

To install from the source, please use the following commands:

!git clone https://github.com/unimorph/umLabeller.git

cd umLabeller

!pip install .

Note: The instructions above have been tested on Google Colab.

Usage

from umLabeller.umLabeller import UniMorphLabeller

uml = UniMorphLabeller()
print(uml.auto_classify('stepstones',['Ġsteps','tones']))

Output:

alien

License:

https://creativecommons.org/licenses/by-sa/3.0/

References

More details can be read in the following article:

Khuyagbaatar Batsuren, Ekaterina Vylomova, Verna Dankers, Tsetsuukhei Delgerbaatar, Omri Uzan, Yuval Pinter, Gábor Bella – Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge. https://arxiv.org/abs/2404.13292