Update transformers requirement from <4.12,>=4.1 to >=4.1,<4.13

[dependabot]

Updates the requirements on transformers to permit the latest version.

Release notes

Sourced from transformers's releases.

v4.12.0: TrOCR, SEW & SEW-D, Unispeech & Unispeech-SAT, BARTPho

TrOCR and VisionEncoderDecoderModel

One new model is released as part of the TrOCR implementation: TrOCRForCausalLM, in PyTorch. It comes along a new VisionEncoderDecoderModel class, which allows to mix-and-match any vision Transformer encoder with any text Transformer as decoder, similar to the existing SpeechEncoderDecoderModel class.

The TrOCR model was proposed in TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models, by Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei.

The TrOCR model consists of an image transformer encoder and an autoregressive text transformer to perform optical character recognition in an end-to-end manner.

• Add TrOCR + VisionEncoderDecoderModel by @​NielsRogge in huggingface/transformers#13874

Compatible checkpoints can be found on the Hub: https://huggingface.co/models?other=trocr

SEW & SEW-D

SEW and SEW-D (Squeezed and Efficient Wav2Vec) were proposed in Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition by Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi.

SEW and SEW-D models use a Wav2Vec-style feature encoder and introduce temporal downsampling to reduce the length of the transformer encoder. SEW-D additionally replaces the transformer encoder with a DeBERTa one. Both models achieve significant inference speedups without sacrificing the speech recognition quality.

• Add the SEW and SEW-D speech models by @​anton-l in huggingface/transformers#13962
• Add SEW CTC models by @​anton-l in huggingface/transformers#14158

Compatible checkpoints are available on the Hub: https://huggingface.co/models?other=sew and https://huggingface.co/models?other=sew-d

DistilHuBERT

DistilHuBERT was proposed in DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT, by Heng-Jui Chang, Shu-wen Yang, Hung-yi Lee.

DistilHuBERT is a distilled version of the HuBERT model. Using only two transformer layers, the model scores competitively on the SUPERB benchmark tasks.

Compatible checkpoint is available on the Hub: https://huggingface.co/ntu-spml/distilhubert

TensorFlow improvements

Several bug fixes and UX improvements for TensorFlow

Keras callback

Introduction of a Keras callback to push to the hub each epoch, or after a given number of steps:

• Keras callback to push to hub each epoch, or after N steps by @​Rocketknight1 in huggingface/transformers#13773

Updates on the encoder-decoder framework

The encoder-decoder framework is now available in TensorFlow, allowing mixing and matching different encoders and decoders together into a single encoder-decoder architecture!

• Add TFEncoderDecoderModel + Add cross-attention to some TF models by @​ydshieh in huggingface/transformers#13222

Besides this, the EncoderDecoderModel classes have been updated to work similar to models like BART and T5. From now on, users don't need to pass decoder_input_ids themselves anymore to the model. Instead, they will be created automatically based on the labels (namely by shifting them one position to the right, replacing -100 by the pad_token_id and prepending the decoder_start_token_id). Note that this may result in training discrepancies if fine-tuning a model trained with versions anterior to 4.12.0 that set the decoder_input_ids = labels.

... (truncated)

Commits

• 62bf536 Release v4.12.0
• 5f3bf65 Fix EncoderDecoderModel docs (#14197)
• ac12a5a Fix EncoderDecoderModel classes to be more like BART and T5 (#14139)
• 1251072 Fix SEW-D implementation differences (#14191)
• 78b6a2e Add audio-classification benchmarking results (#14192)
• 1dc96a7 Add SegFormer (#14019)
• 123cce6 [modeling_utils] respect original dtype in _get_resized_lm_head (#14181)
• 88cd82e Update README.md
• e118db1 Update README.md
• 01b1466 [TPU tests] Enable first TPU examples pytorch (#14121)
• Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)