-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Encoded sequences can clash with separator byte and cause assertion errors #85
Comments
I'll look into this, thanks Daniel. |
Confirmed. The encoded length of the suffix to strip clashes with separator character. Thinking on how to fix this. |
Daniel, the simplest workaround for this problem is currently to recompile the dictionary from scratch and substitute the separator character for something with a higher value. Good examples would be ',' or '\t' since these fall below 'A' in ascii range and would wrap around to much longer suffixes. I didn't invent this encoding scheme, it's a legacy from Janek Daciuk's work. A proper fix would involve some bookkeeping on the encoded string so that we don't accidentally trip on separator character (the input must not contain it, this is verified). |
Fixed in 2.1.2. Will publish asap. Thanks for a detailed repro, Daniel! |
Thanks for the fast fix, I can confirm it fixes the issue - also without re-building the dictionary, or do you still recommend that? |
The fix will work with your existing dictionary if you can/ are willing to upgrade. If you want to stick to 2.1.1, you'll have to rebuild the dictionary with a different separator, unfortunately. |
This causes the assert to fail in TrimSuffixEncoder.java:60:
To reproduce, you can get the
german_synth.dict
from http://search.maven.org/remotecontent?filepath=de/danielnaber/german-pos-dict/1.0/german-pos-dict-1.0.jarStacktrace is this:
In a less stripped down case this shows as (at least I assume this is the same issue):
The text was updated successfully, but these errors were encountered: