Perf improvement 16% by removing offsets.

[Narsil]

Offsets calculation are always calculated in Python land. By changing it to not being calculated, we win 16% of the runtime.

This is not the total extent of it because offsets are still calculated in bytes.

On M3.
Before:

==============
num_threads: 8, data size: 24.04 MB, documents: 10000 Avg Length: 1659
tiktoken 	59.51 MB  / s
huggingface 	25.36 MB / s
==============
num_threads: 8, data size: 1.11 MB, documents: 10000 Avg Length: 116
tiktoken 	7.48 MB  / s
huggingface 	20.93 MB / s

After:

==============
num_threads: 8, data size: 24.04 MB, documents: 10000 Avg Length: 1659
tiktoken 	60.38 MB  / s
huggingface 	30.31 MB / s
==============
num_threads: 8, data size: 1.11 MB, documents: 10000 Avg Length: 116
tiktoken 	5.94 MB  / s
huggingface 	24.78 MB / s
==============

On AMD EPYC 7R13 Processor:

Before:

num_threads: 32, data size: 24.04 MB, documents: 10000 Avg Length: 1659
tiktoken        29.88 MB  / s
huggingface     28.14 MB / s
==============
num_threads: 32, data size: 1.11 MB, documents: 10000 Avg Length: 116
tiktoken        2.85 MB  / s
huggingface     22.27 MB / s

After:

==============
num_threads: 32, data size: 24.59 KB, documents: 10 Avg Length: 1662
tiktoken        4.97 MB  / s
huggingface     4.73 MB / s
==============
num_threads: 32, data size: 24.04 MB, documents: 10000 Avg Length: 1659
tiktoken        29.78 MB  / s
huggingface     49.95 MB / s
==============
num_threads: 32, data size: 1.11 MB, documents: 10000 Avg Length: 116
tiktoken        2.85 MB  / s
huggingface     29.28 MB / s

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.