-
Notifications
You must be signed in to change notification settings - Fork 73
Datasets
The datasets listed below are all preloaded into the HackReduce hadoop clusters and ready for immediate use at the event. The [datasets/*] notice next to each title indicates the path where its located depending on where you want to access it:
- Hadoop HDFS: Can be found at
/datasets/* - Namenode local filesystem: Can be found at
/mnt/datasets/* - HackReduce Github project: Samples found in the
datasets/*folder of the project. Note: not all the datasets listed on this page will have samples in the Github project.
There's also the possibility of loading new data at the event, but this process could take a few hours. Please see Greg about loading new data into your clusters.
If you're looking for data sets, below are a few good places to start:
- Quora answer wiki for Where can I get large datasets open to the public?
- The Data Hub data catalog
- The Visua.ly Blog has a great article called 30 Places to Find Open Data on the Web.
If you're looking for Toronto data sets here are a couple of places to start:
- Buzzdata has many city-related data sets rounded up here.
- Global news's data set of Toronto parking tickets is here, again courtesy of Buzzdata.
Special thanks to Echo Nest for converting the whole 200+ GB HDF5 format of the dataset to TSV for us
- Quad dump (http://wiki.freebase.com/wiki/Data_dumps#Quad_dump) [datasets/freebase/quadruples]
- Simple topic dump (http://wiki.freebase.com/wiki/Data_dumps#Simple_Topic_Dump) [datasets/freebase/topics]
- Only the 1-gram and 2-gram datasets are available
- http://ngrams.googlelabs.com/datasets
- Data format documentation: http://dcc.icgc.org/pages/docs/ICGC_Data_Submission_Manual-0.6b-Unextended.pdf
- French: datasets/fre-eng/fre
- English: datasets/fre-eng/eng
- http://www.statmt.org/wmt09/translation-task.html
- Provided by the Mate1 team
- Includes (with all personally identifiable data excluded, of course):
- Profiles: datasets/mate1/profile
- Iinternal messages: datasets/mate1/internal_message
- Subscriptions: datasets/mate1/subscription
- Who's seen who: datasets/mate1/whos_seen_who
- Hot block list: datasets/mate1/hot_block_list
- Take a look at the datasets/mate1/*-cols.txt files for a description of the CSV fields for each dataset.
- Includes Bixi (branded differently in other cities) data for Toronto, Ottawa and Boston (Hubway).
- Updated by Julia Evans and Kamal Marhubi for HackReduce Montreal 2012
- Dataset location on the namenode local filesystem is: /mnt/bixidata
- Additional Boston Hubway data sets available from: http://hubwaydatachallenge.org including shapefiles, an aggregated rebalancing data sample, Apr - Sep 2012, and station status data, with available bikes and empty docks per station by minute back to August 2011 (30 million records)
- XML dump of all the bike station information queried every minute over a couple of months.
- Provided by Fabrice (http://twitter.com/f8full)
- Contains the root file with all the domain names and their associated nameservers for the "com" TLD.
- Data of the social graph, user id to names, and selected celebrity profiles. This does not contain actual tweets because of Twitter policies.
- http://an.kaist.ac.kr/traces/WWW2010.html
- Limited set of flight data containing origin, destination, departure time, return time, price and date.
- Only has flights originating from SEA
- Provided by Hopper
- Description of data formats: http://131.193.40.52/data/README.txt
- Data listing: http://131.193.40.52/data/
- Taken around of the time of Elizabeth Taylor's death in late March 2011, this dataset was a search of all tweets containing the word "taylor" in them.
- JSON format
-
Arxiv HEP-PH (high energy physics phenomenology) [datasets/citation-networks/hep-ph/{dates,graph}]: http://snap.stanford.edu/data/cit-HepPh.html
-
Arxiv HEP-TH (high energy physics theory) [datasets/citation-networks/hep-th/{dates,graph}]: http://snap.stanford.edu/data/cit-HepTh.html
-
U.S. patent dataset: http://snap.stanford.edu/data/cit-Patents.html
- All the data for the U.S. specifying origin/destination of orders from our system, including price and date.
- Web crawl corpus of 5 billion pages (60TB) in ARC file format http://aws.amazon.com/datasets/41740
- Divided into three major subsets:
- Current Crawl - s3://aws-publicdatasets/common-crawl/parse-output/ - crawl data from 2012
- Archived Crawl #1 - s3://aws-publicdatasets/common-crawl/crawl-001/ - crawl data from 2008/2010
- Archived Crawl #2 - s3://aws-publicdatasets/common-crawl/crawl-002/ - crawl data from 2009/2010