Falcon Model Bug - ImportError: cannot import name 'top_k_top_p_filtering' from 'transformers.generation.utils'

[anirudTT]

Describe the bug

When running the models falcon to benchmark it throws a error.

To Reproduce
Steps to reproduce the behavior:

1. Clone PyBuda
2. Apply git submodule sync and git submodule update --init --recursive -f
3. Checkout branch vcanic/nvukobrat_rc_240617
4. Run make clean from the root of the pybuda repository.
5. Run the the code source env_for_wormhole_b0.sh , it compiles pybuda with all necessary submodules for wormhole_b0 arch, one should see activated python_env
6. When the compilation is finished, go ahead and clone benchmarking repository, and pip install -r requirements.txt , with the same activated python_env.
7. Next run the falcon model using : python benchmark.py -d tt -m falcon -c 7b-instruct --task hellaswag -df Fp16_b -mb 32 --loop_count 1 --save_output

Expected behavior
The model should without any errors , and produce a benchmarking results similar or close to previous results for the falcon model.

Bug log

backend//cluster_desc.yaml' with timeout 120s Detecting chips (found 1) INFO:benchmark.common.benchmark_run: PyBUDA is not installed as a wheel dist, get git hash from submodule. INFO:__main__: creating benchmarking run: falcon INFO:__main__: kwargs: {'training': False, 'microbatch': 32, 'task': 'hellaswag', 'device': 'tt', 'data_type': 'Fp16_b', 'config': '7b-instruct'} Traceback (most recent call last): File "benchmark.py", line 573, in <module> model, generator, eval_fn = models[args.model]["func"](benchmark_run=benchmark_run, **kwargs) File "/home/aramchandran/Work/benchmarking/benchmark/common/common.py", line 30, in wrapper model, generator, eval_fn = model_func(*args, benchmark_run=benchmark_run, **kwargs) File "/home/aramchandran/Work/benchmarking/benchmark/models/falcon/falcon.py", line 79, in falcon from benchmark.models.falcon.utils.model import Falcon File "/home/aramchandran/Work/benchmarking/benchmark/models/falcon/utils/model.py", line 16, in <module> from transformers.generation.utils import top_k_top_p_filtering ImportError: cannot import name 'top_k_top_p_filtering' from 'transformers.generation.utils' (/home/aramchandran/Work/pybuda/build/python_env/lib/python3.8/site-packages/transformers/generation/utils.py)

System:

• Device: Wormhole N150
• Host information: [e.g. Intel Xeon]
• OS : Ubuntu 20.04]

Additional context
Add any other context about the problem here.

[anirudTT]

Looks like the function 'top_k_top_p_filtering' from 'transformers.generation.utils' method was deprecated from transformers >= 4.39. Current build of pybuda makes use of transformers -= 4.41.0.

This issue is referenced in hugging face github repository within a issue and according to the developers simplest solution is to copy the method directly into source code.

Current Solution to fix this issue:

1. Copy over the method as into the folder benchmark/models/falcon/utils/
2. import/ Adding this method within the benchmark/models/falcon/utils/model.py

References:

• Fix import error from deprecation in transformers huggingface/trl#1415
• Can not import name top_k_top_p_filtering huggingface/trl#1409

[anirudTT]

Running after this commit fix :
23be2be
Should fix the error :)