Open Data Collectors

A set of Pushpop jobs that collect data from various APIs over time and push data into Keen IO.

There are 2 ways to use this project:

• Use the Keen IO Project ID and Read Key specified in each collector's documentation to use the data.
• Capture the data yourself by cloning and deploying this repository pointing to your own project ID.

Note: Install keen-cli to run example CLI queries.

Collectors

Bitcoin Prices

Uses the BitcoinAverage API to capture Bitcoin price trends over time.

• Code – jobs/bitcoin_prices.rb
• Interval - Every minute
• Contact Info - @dzello
• Runs On – Heroku
• Example Event - Extract One
• Keen IO Data Set Information
  • Project ID: 53f3a8687d8cb95095000001
  • Read Key: 99a06e48fd7fb1279bc40995160eb0b61a9e0efaab8b651b029f0d895f77c0a804ba089282eff28bf8ad07f337422441d0542b7feaac9fea1e92fc153ee7efc51afad3276bda8d7754a338b349d540bfb402cba0dfdc82498c217054efd8abd0f47a0c0bc963bbdf0dc938c91b17d9f2
  • Collection Name: bitcoin-prices

Example CLI Queries

Note: Make sure to set environment variables in .env or specify --project and --read-key parameters.

# price last 30 minutes piped to spark
$ keen average -c bitcoin-prices -y "USD.averages.last" -t last_30_minutes -i minutely --spark | spark
▄███▄▄▄▄▄▁▄▄▄▄▄▄▄▁▁▁▄▄▄▄██▄

# highest USD price today
$ keen maximum -c bitcoin-prices -y "USD.averages.last"
481.77

# total USD volume last 24 hours
$ keen extraction -c bitcoin-prices --property-names "USD.averages.total_vol" --latest 1
[
  {
    "USD": {
      "averages": {
        "total_vol": 38276.39
      }
    }
  }
]

Contributing

More jobs! More open data sets!

If you add a job please fill out all the details (use Bitcoin Prices as a guide). Keen IO project and credentials are not required if you expect someone to capture the data on their own and not use a shared project.