Fix ClapProcessor to merge feature_extractor output into the returned BatchEncoding

[mxkopy]

Info

• transformers version: 4.42.3
• Platform: Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.36
• Python version: 3.11.9
• Huggingface_hub version: 0.23.4
• Safetensors version: 0.4.3
• Accelerate version: 0.31.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.3.1+cu118 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: yes
• GPU type: NVIDIA GeForce RTX 3050 Laptop GPU

Problem

CLAP examples break for fused models. The ClapModel.__call__ implementation for fused models requires the is_longer argument to be set, but currently ClapProcessor only sets input_features, resulting in a NoneType error. See the following MWE and stacktrace:

from transformers import ClapModel, ClapProcessor
import torch


processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused", torch_dtype=torch.float16, device_map='cuda')
model = ClapModel.from_pretrained("laion/clap-htsat-fused", torch_dtype=torch.float16, device_map='cuda')
audio = torch.randn(1, processor.feature_extractor.sampling_rate * 40).tolist()
text  = 'test'
inputs = processor(text=text, audios=audio, sampling_rate=processor.feature_extractor.sampling_rate, return_tensors="pt")
inputs['input_features'] = inputs['input_features'].type(torch.float16)
inputs.to('cuda')
outputs = model(**inputs)

Traceback (most recent call last):
  File "/src/test.py", line 12, in <module>
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/clap/modeling_clap.py", line 2097, in forward
    audio_outputs = self.audio_model(
                    ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/clap/modeling_clap.py", line 1747, in forward
    return self.audio_encoder(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/clap/modeling_clap.py", line 911, in forward
    is_longer_list = is_longer.to(input_features.device)
                     ^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'to'

Note that if you pull from 'laion/clap-htsat-unfused' instead, the code runs fine.

Fix

Merge the output of the feature extractor rather than set individual keys.

This will break if ClapFeatureExtractor.__call__ returns something that can't be unpacked into the kwargs of ClapModel.__call__. However, I think it's reasonable to expect the outputs of ClapFeatureExtractor.__call__ to always constitute valid inputs to ClapModel.__call__.

@ArthurZucker

[LysandreJik]

cc @sanchit-gandhi @ylacombe

[sanchit-gandhi]

LGTM - thanks for this change @mxkopy 🤗 and congrats on your first contribution!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[LysandreJik]

Thanks for your PR @mxkopy!