Avoid PyArrow type optimization if it fails

[mariosasko]

Adds a new variable, DISABLE_PYARROW_TYPES_OPTIMIZATION, to config.py for easier control of the Arrow type optimization.

Fix #2206

[lhoestq]

That's good to have a way to disable this easily :)
I just find it a bit unfortunate that users would have to experience the error once and then do DISABLE_PYARROW_TYPES_OPTIMIZATION=1. Do you know if there's a way to simply fallback on disabling it automatically when it fails ?

[mariosasko]

@lhoestq Actually, I agree a fallback makes more sense. The current approach is not very practical indeed and would require a mention in the docs.

[mariosasko]

Replaced the env variable with a fallback!

[lhoestq]

Good job ! I think this could also be part of a documentation page about "Processing text data" in an optimization section cc @stevhliu

[stevhliu]

Hmm if the fallback automatically happens without the user knowing it, then I don't think we really need to mention it. But if you really wanted to, I think the Improve performance section would be a great place for it!

[lhoestq]

Yea I think this could just end up in a note that says that datasets automatically picks the most optimized integer precision for your tokenized text data to save you disk space. Maybe later if we have a page on text processing we can add this note, but for now I agree it doesn't fit well into the doc.

In particular in the "Improve performance" section we mention what users can do to speed up their computations, while this behavior is just some internal feature that users don't have control over anyway.