Dev.ej/lexicon tokenizer

[joanise]

PR Goal?

Introduce the lexicon tokenizer

Fixes?

Fixes #401

Feedback sought?

This is a draft PR, seeking early feedback on the algorithm I implemented.

Priority?

medium-high

Tests added?

yes

How to test?

Run g2p convert "We'll let go, she'll stop." eng eng-ipa and see the output wil lɛt ɡoʊ, ʃil stɑp. instead of wi' lɛt ɡoʊ, ʃi' stɑp.

Confidence?

moderate only.

• unit testing tells me it's fine
• but I have not yet done any benchmarking to see if we need to optimize

Version change?

Probably yes, at least a patch bump, since this is a bug fix, maybe even a minor bump?

[github-actions]

CLI load time: 0:00.05
Pull Request HEAD: c3d73bfa8014c34c7dc004637465ca453f9e9cbf
Imports that take more than 0.1 s:
import time: self [us] | cumulative | imported package

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.29%. Comparing base (1262cbb) to head (c3d73bf).
Report is 8 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #405      +/-   ##
==========================================
+ Coverage   93.89%   94.29%   +0.40%     
==========================================
  Files          18       18              
  Lines        2587     2664      +77     
  Branches      580      598      +18     
==========================================
+ Hits         2429     2512      +83     
+ Misses         91       88       -3     
+ Partials       67       64       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[joanise]

Some benchmarking notes: I tested g2p'ing the UNDHR (https://www.un.org/en/about-us/universal-declaration-of-human-rights) text, and it's actually about 1% faster on this branch than on origin/main, so clearly this implementation is fine on "normal" text.

I created a degenerate case with a word surrounded by dozens of punctuation marks on each side, and in that case I can see on slow-down on this branch, which makes sense because I'm doing O(n^2) lookups, when n is the number of candidate tokens, and each lookup costs O(log k), where k is the lexicon size. I'm incline to say YAGNI on optimizing this: it's only a problem on artificial data. Alternatively, I could cap the number of non-alphabetic characters before or after alphabetic ones that I include in the lexicon lookups, as a defensive programming solution.

[roedoejet]

I'm incline to say YAGNI on optimizing this

Agreed. Thanks for looking into this though @joanise !

[roedoejet]

Just a couple small typos, but this looks good to me. Nice that you fit in a few optimizations as well. Thanks @joanise - tested and it works well on my machine.