kaggle croissant files api?

[luisoala]

@goeffthomas is there an api for kaggle croissant files similar to

croissant/health/crawler/spiders/huggingface.py

Line 31 in 505ea3c

return f"https://datasets-server.huggingface.co/croissant?dataset={dataset_id}"

?

when i do chrome dev tools for a croissant file on the kaggle web gui i get a gcs url like

https://storage.googleapis.com/kaggle-data-sets/4482993/7683348/croissant/metadata.json?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=gcp-kaggle-com%40kaggle-161607.iam.gserviceaccount.com%2F20240228%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20240228T140305Z&X-Goog-Expires=259200&X-Goog-SignedHeaders=host&X-Goog-Signature=78fbb6537a3cf1a6f8ac9f85dbb46aaa244a66b9dfce03d37405c3dd2bb9f6c6d40a7897e516841d5f5bb516e806252852752905b6f12bb9f9a67a4becd2664df514e33dde9af3931576acc78bfac24cf3991bc6021a7f138850bb757429360f8bfa44e4fecc85d936ae217d9bc2062162beee7cf394c24d63333fbcd13f81cef6263a5399837bbd0f51bb9277f4be5237320ae459ad596a33048eb9a8109f5b99882b7554efb41edb32e39ff3109e88e964982ed106523ba5d9166a90ec9b0d06276e1795fe299f5190d48a899d56ee8fe8f1b3694a2ece023a1318149d79a2975f9cc0ee6400fbab80b63aa1ab16b28e5cfd1d4d9518f89ccc7ceff745d004

(this is for dataset example https://www.kaggle.com/datasets/kanchana1990/airbnb-las-vegas-listings)

[goeffthomas]

@luisoala For any Kaggle Dataset URL, you should be able to append /croissant/download to the end and you'll get a signed URL for the metadata.json file (which we store in GCS).

[luisoala]

thx goeff! will try that for the crawler

[goeffthomas]

If you're working on a crawler for Kaggle, you'll want to be very careful. I had discussed this with @marcenacp since he was wondering about setting it up for Kaggle as well. When it comes to doing downloads (Croissant included), we have a daily rate limit of 5k per user. As a result, crawling all of Kaggle won't be possible. We've had some side convos about maybe us setting up an internal crawler that can bypass user rate limit issues, and then we'd find a way to get those results shared back for visibility.

Alternatively, a sampling approach could be possible, with an understanding that it's quite a small sample of our total datasets.

[luisoala]

got you thx for context! itd be nice to have an initial "internal" dump if thats an option (to map out the universe at its current state) and then we can see how to monitor it moving forward. i understand its not feasible to have unlimited crawls on demand(:

[goeffthomas]

I could look into that, but would still take a bit of work on our end to produce. Most of my remaining time allocated for Croissant is going toward launching/landing v1.0 (migrating off the v0.8 we already implemented) and ensuring all our datasets get that upgrade. If I have some free time toward the end of H1, I'd be happy to look at how to help enable this "Croissant health" effort (in one form or another).

[luisoala]

sounds good, step by step. i started with openml crawl dump to project the croissant datasets from there to one feature space for visual exploration and discovery. it would be nice to eventually crawl the entire croissant ecosystem to do this (btw i appreciate how rich the kaggle dataset description fields are:)

Screencast.from.02-29-2024.05.01.31.PM.webm