llama : tokenizer unicode codepoint categories

[jaime-m-p]

Add all unicode categories to unicode-data.cpp.

Currently we are limited to high categories:

• C, L, M, N, P, S, Z.

This PR allows access to subcategories:

• Cn, Cc, Cf, Co, Cs, Ll, Lm, Lo, Lt, Lu, Mc, Me, Mn, Nd, Nl, No, Pc, Pd, Pe, Pf, Pi, Po, Ps, Sc, Sk, Sm, So, Zl, Zp, Zs.

Related PR: #8579, regex using Lu, Lt, Lm, Lo, etc.

TODO: Add more comments to explain the unicode regex collapse trick for all subcategories.

---

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[compilade]

Nice! This should also help fix (at least part of) Falcon's tokenization, because the Punctuation pre-tokenizer type uses the Po category and not the broader P one.

(ref: https://github.com/huggingface/tokenizers/blob/4ea2f235b0430f5db09f867b65306d6c0a5ec7ed/tokenizers/src/pre_tokenizers/punctuation.rs#L8, which uses Rust's is_ascii_punctuation and is_punctuation)

[ggerganov]

TODO: Implement unicode regex collapse trick for all subcategories.

Do you expect any problems with this? Probably we will run out of ASCII characters for the k_ucat_cpt map:

llama.cpp/src/unicode.cpp

Lines 647 to 651 in 50e0535

	static const std::map<int, int> k_ucat_cpt = {
	{ codepoint_flags::NUMBER, 0xD1 },
	{ codepoint_flags::LETTER, 0xD2 },
	{ codepoint_flags::PUNCTUATION, 0xD3 },
	};

Though we could dynamically generate the map based only on the used subcategories in the current regex

[ggerganov]

The src/llama.cpp conflict should be easy to resolve - just accept the new src/llama.cpp and apply the same changes to src/llama-vocab.cpp instead

[jaime-m-p]

TODO: Implement unicode regex collapse trick for all subcategories.

Do you expect any problems with this?

More problems than I thought:

• Need +29 collapse codepoints for subcategories.
• Ranges of collapse codepoints, ie: \p{L} --> \p{Ll} to \p{Lu} (Ll, Lm, Lo, Lt, Lu).
• Collapse codepoint for unicode whitespaces to fix the \s problem (std::regex ignores non-ASCII \s).
  • Take care of \S and regex lookaheads, ie: (?!\S).

[jaime-m-p]

I tested (subset of the brute-force tests) all available BPE models, including tekken. Same results as before this PR.
Also tested the original tekken regex and seems correct too.

The reimplementation is not very understandable without context.
I want to add more comments and try to explain all steps/blocks of code.

[ggerganov]

Most of the assert in unicode.cpp should be changed to GGML_ABORT or GGML_ASSERT