Add script to convert T5X T5 (v1.0 and v1.1) checkpoints to PyTorch

[bastings]

What does this PR do?

Adds a script that can convert Google T5X (Flax) T5 and T5-v1.1 checkpoints into PyTorch checkpoints.
This allows users to convert non-standard checkpoints that have been trained with T5X and use them with the Transformers library in PyTorch.

Usage:

• In case you don't have gsutil, install according to https://cloud.google.com/storage/docs/gsutil_install
• Native T5X checkpoints are at https://github.com/google-research/t5x/blob/main/docs/models.md#t5-11-checkpoints. Example:
  gsutil -m cp -r gs://t5-data/pretrained_models/t5x/t5_1_1_small $HOME/
• Create a corresponding config.json for the downloaded checkpoint. Often one already exists, e.g. here we can use https://huggingface.co/google/t5-v1_1-small/blob/main/config.json
• Finally python3 convert_t5x_checkpoint_to_pytorch.py --t5x_checkpoint_path=$HOME/t5_1_1_small --config_file=config.json --pytorch_dump_path=$HOME/t5_1_1_small_pt

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case. Discussed with @thomwolf .
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests? The code is tested but not part of this PR, since the test requires manually downloading the T5X checkpoints from a cloud bucket.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten
@sanchit-gandhi
@ArthurZucker
@younesbelkada

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[bastings]

I could use some clarification on the following: I'm missing a configuration option for T5 for the 1.0/original T5 checkpoints to have an lm_head that shares parameters with the token embeddings.

Currently there is T5Model (which returns hidden states) and T5ForConditionalGeneration (which returns logits, used for T5 v1.1 models among others). The latter assumes there is an lm_head layer, but for the 1.0 checkpoints there is no such thing, it reuses the embedding matrix to map to the vocab space.

[patrickvonplaten]

Thanks a lot for adding this @bastings

cc @ArthurZucker

[ArthurZucker]

Hey @bastings, when there is no lm_head you have to set the tie_word_embeddings to True

[ArthurZucker]

It's very clean, thanks a lot for the addition.

[sanchit-gandhi]

Very cool PR @bastings! Thanks for the addition! Do you have a set of example args I could use just to try the script out once for myself? Thanks! 🙌

[bastings]

I added the instructions to the top docstring. Maybe it's ready? :-)

[ArthurZucker]

A last nit and we can merge! Thanks a lot for bearing with me 😄

[bastings]

Thanks! Committed your suggestion :)

[sanchit-gandhi]

LGTM! Thanks!

[sanchit-gandhi]

Once the quality tests are green (requires make fixup) we can merge!

[bastings]

Oh looks like the suggestion made it fail ;)

[ArthurZucker]

Ah, sorry then ahha, I guess the make stylewill correct that 😅

[bastings]

Ah, sorry then ahha, I guess the make stylewill correct that 😅

Fixed! :)