Fix indexation of CJK languages.

[mgautierfr]

Fix #802

[codecov]

Codecov Report

Patch coverage: 16.66% and project coverage change: -0.03 ⚠️

Comparison is base (c9d5e9a) 57.49% compared to head (0f0016c) 57.46%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #806      +/-   ##
==========================================
- Coverage   57.49%   57.46%   -0.03%     
==========================================
  Files          98       98              
  Lines        4526     4528       +2     
  Branches     1901     1903       +2     
==========================================
  Hits         2602     2602              
  Misses        674      674              
- Partials     1250     1252       +2     

Impacted Files	Coverage Δ
src/search.cpp	48.40% <0.00%> (ø)
src/writer/xapianIndexer.cpp	30.88% <0.00%> (-0.47%)	⬇️
src/writer/xapianWorker.cpp	32.50% <0.00%> (-0.84%)	⬇️
src/suggestion.cpp	45.38% <33.33%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[kelson42]

@mgautierfr Can you please provide an index size benchmark?
@xiaoyifang Could you please confirm this fix the issue you have open (hope you are able to build a test ZIM file)?

[veloman-yunkan]

Is the commit containing the fix final? If yes, why does it have [WIP] in its title?

[veloman-yunkan]

Also, do you have an explanation for why the codecov/patch flow has failed?

[mgautierfr]

Also, do you have an explanation for why the codecov/patch flow has failed?

For a unknown reason (maybe the assert at the beginning of function ?), codecov thinks the line are partially tested.
Maybe it is not totally counted as tested and, as we add new lines, the coverage decrease.

[veloman-yunkan]

Don't merge without squashing the fixup commit.
Don't merge without squashing the fixup commit.
Don't merge without squashing the fixup commit.
Don't merge without squashing the fixup commit.