[GH->HF] Part 2: Remove all dataset scripts from github

[lhoestq]

Now that all the datasets live on the Hub we can remove the /datasets directory that contains all the dataset scripts of this repository

• Needs [GH->HF] Load datasets from the Hub #4973 to be merged first
• and PR to be enabled on the Hub for non-namespaced datasets

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[albertvillanova]

Thanks! Good: this way all datasets are completely aligned.

Just one thought: HEHE, I would say that it is at least ironic that a library called datasets deletes all the datasets but nevertheless keeps the metrics (which indeed were moved to other library)... 🤣

[osbm]

So this means metrics will be deleted from this repo in favor of the "evaluate" library? Maybe you guys could just redirect metrics to that library.

[lhoestq]

We are deprecating the metrics in datasets indeed and suggest users to switch to evaluate (via a warning message)

We'll keep the current metrics as they are for now, but they'll be completely removed at one point

[lhoestq]

I guess this is ready to merge ?

It should break nothing except one rare case:

If someone is using an old version of datasets to try to load a recent dataset. Indeed in that case it fetches the main branch on github to see if it exists. But since we're removing all the datasets, forward fetching won't work anymore.

e.g. if someone uses "imagenet-1k" with a version of datasets that didn't have it at that time. I checked on kibana and one single user would be affected with 4k downloads/months. It should still work for them though thanks to the datasets cache

But if they delete their cache, the workaround is... 🥁 update datasets 😅

[lhoestq]

Let's merge this on monday if we can, to make sure contributors who wanted to merge their dataset PRs here could do it

[albertvillanova]

All pending PRs have been either merged or closed.

You can now remove all dataset scripts from GitHub.

[lhoestq]

Alright, merging !