fix: add tiktoken fallback mechanism.

[danielbichuetti]

Related Issues

• fixes
  • Docker failure in CI
  • fix: temporarily remove tiktoken #3927

Proposed Changes:

Implemented fallback to HF FastTokenizer in Linux ARM64 and AARCH64

How did you test it?

Ubuntu 22.04, Windows, Mac M1

Notes for the reviewer

tiktoken wheels available in PiPy:
https://pypi.org/pypi/tiktoken/json

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added tests that demonstrate the correct behavior of the change
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[ZanSara]

Hey @danielbichuetti, thank you for this fix! We realized just yesterday night that tiktoken broke a few Docker builds 🙈 Hopefully this will fix it.

[sjrl]

Linking this to the open issue on the tiktoken library for visibility openai/tiktoken#23

[sjrl]

Just out of curiosity, would the workaround in this comment work for us as well? (Basically building the wheels from scratch). This way we could keep the tiktoken library in our docker images as well, but I'm not sure of the cons of this approach.

[ZanSara]

Yesterday with @silvanocerza we tried to apply it with no success. We could try harder, but I don't see the benefit right now. I think makes sense to disable the unsupported platforms for this release, and reconsider it later. Maybe there are going to be prebuilt wheels by then!

I will take care of adding a "self-destructing" reminder test in Haystack once CI has been moved to 3.8, so that we don't forget.