Fix tqdm zip bug

[david1542]

This PR solves #5117, by wrapping the entire zip clause in tqdm.

For more information, please checkout this Stack Overflow thread:
https://stackoverflow.com/questions/41171191/tqdm-progressbar-and-zip-built-in-do-not-work-together

[albertvillanova]

Thanks for trying to fix this bug, @david1542.

However the proposed solution will not work: please note that pbar is also used if not batched:

datasets/src/datasets/arrow_dataset.py

Lines 2951 to 2952 in 07b7c38

	if not batched:
	for i, example in enumerate(pbar):

• there it is expected that iterating over pbar returns examples, not a tuple like with your implementation.

[david1542]

@albertvillanova Thanks for your comment. What do you think about creating 2 pbar for each case? I see the pbar_iterable is initialized differently. Maybe pbar can also be initialized like that.

[david1542]

@albertvillanova Another solution I implemented is to change pbar_iterable and add the zip to it. I updated the PR with this solution. Let me know what you think.

[albertvillanova]

Thanks for the improvement.

I think it is a good idea to include the indices in pbar_iterable.

You need to fix the style to pass the CI test on code quality. Please run:

make style

To be consistent, I would recommend to do the same (include the enumerate in pbar_iterable) for the non batched case.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[dudulasry]

@albertvillanova Done :) Let me know what you think.

[albertvillanova]

Hi @david1542, awesome work!

I'm just wondering if it could be possible to implement a regression test for this, but I don't see any straightforward way...

Otherwise, this PR seems good to me!

[david1542]

@albertvillanova Thanks :) I also don't see an easy way to test this. This was just a problem in the way tqdm was used. I'm not sure we should cover it in tests.

[Cochonaki]

Hi,

First of all, thanks for this PR.
It's the first time I join a discussion on GitHUB on problem resolution in libraries such as transformers, so I hope I comply to the best practices for an efficient communication...

I am running AutoTokenizer.from_pretrained in a Google Colab notebook for using with BERT base.
I am experiencing issue 5117.

Each time I run my notebook, I do:

! pip install transformers ! pip install datasets ! pip install huggingface_hub

As I understand, the issue has been resolved and the solution merged to the released version of the code?
So I expect that the bug is resolved in my notebook, however this is not the case.

Do I get something wrong?
Do I have to implement some change in the source code myself?

Thanks in advance for your help!

[david1542]

@Cochonaki Hi :) The problem was fixed but there wasn't a release since then. I believe a new release should come out in the upcoming weeks. Maybe someone from the core maintainers can answer that :)

cc: @albertvillanova

[armw4]

Baby Haiti Coffee SE is born NH watch

…

On Sun, Oct 23, 2022 at 02:39 Dudu Lasry ***@***.***> wrote: @Cochonaki <https://github.com/Cochonaki> Hi :) The problem was fixed but there wasn't a release since then. I believe a new release should come out in the upcoming weeks. Maybe someone from the core maintainers can answer that :) cc: @albertvillanova <https://github.com/albertvillanova> — Reply to this email directly, view it on GitHub <#5120 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB4E2NCT7QO7W3PTQGDIKDWETMQ7ANCNFSM6AAAAAARGRBY2M> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[albertvillanova]

Hi, @Cochonaki.

As @david1542 pointed out, we have not made a release since this bug was fixed. We will make one in the following weeks.

In the meantime, if you would like to incorporate the bug fix, you can install datasets from this repo main branch:

pip install git+https://github.com/huggingface/datasets#egg=datasets

[Cochonaki]

Thanks a lot @albertvillanova and @david1542, it works now!
I am really thankful for your help, that encourages me to participate more in this community.
See you around!

[albertvillanova]

Welcome!!! 🤗