upgrade datasets to 2.14 #1550
Assignees
Labels
Comments
|
I could take care of this. |
|
I will address this issue with the patch release 2.14.1 |
|
good idea. In particular, it contains:
I imagine we will have to refresh all the datasets with the |
This was referenced Jul 28, 2023
Merged
|
I am updating
We will be able to remove some authentication tweaks, where we had to use |
severo
added
P0
This was referenced Aug 11, 2023
I've launch the refresh of these: the list is here: |
|
For Refresh all the datasets with only one config I used the following query to get the datasets: |
|
For datasets with one config: We still have 25548, but I would like to continue force refreshing them incrementally, currently, we have a load of 859K pending jobs in the queue and would like to avoid overloading db. |
|
We have still 23402 datasets with one config to backfill. It is pending because of queue overload. |
Is it fixed now @AndreaFrancis? |
|
Not yet I think, but the queue is empty now so feel free to fill it :) |
|
I will continue with the refresh for datasets with one config |
|
I finished updating datasets with one config. Only the following records (331) are missing but those are because they don't exist in the hub. Maybe we should remove those records? |
I wonder, why don't they get deleted when calling webhook? https://github.com/huggingface/datasets-server/blob/main/services/api/src/api/routes/webhook.py#L77 maybe we have something buggy there or maybe the webhook was not called? |
No, in this case when calling either force-refresh or dataset-backfill, there is no action performed because https://github.com/huggingface/datasets-server/blob/main/services/admin/src/admin/routes/dataset_backfill.py#L50 Some of those records might be deleted once we implement TTL in cache collection. |
They are generally deleted, but it seems like not all the cases are processed correctly. It would be good to have some metrics about that (not a priority) and investigate why some of them are not deleted. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://github.com/huggingface/datasets/releases/tag/2.14.0
main changes:
tokeninstead ofuse_auth_tokendefaultinstead ofusername--dataset_name: we have to refresh all the datasets with only one configTODO:
StreamingRowsErrorTODO: 2.14.4
The text was updated successfully, but these errors were encountered: