Skip to content

Latest commit

 

History

History
159 lines (122 loc) · 7.96 KB

README.md

File metadata and controls

159 lines (122 loc) · 7.96 KB
Raw

This code is related to the paper A tool for personal data extraction and is part of the YourDigitalSelf project.

The code is outdated! Last updated in 2017.

The instructions below were tested on Linux only! Please, be aware that things are constantly changing and some installation commands may stop working. However, the package requirements are still the same.

Installation
APIs
  1. download rauth (pip install rauth)
  2. find rauth installation on local machine — /usr/local/lib/python2.7/dist-packages/rauth — and replace files service.py and session.py by the ones provided in neemi/packages/rauth/
  1. Google APIs Client Library for Python
    https://developers.google.com/api-client-library/python/start/installation
    pip install —upgrade google-api-python-client
  2. Google Data API
    pip install gdata
Configuration

  • Django settings Edit file DigitalSelf/webapps/settings.py specifying name of database to be created to store all user data: Example - connect(‘digital_self’)

  • keys.py To be able to access an API the user has to register an application with their services. During registrations, the API’s will provide at least a pair of id/secrets for each application registered and each application can serve multiple users. A skeleton of the keys.py file can be found in DigitalSelf/webapp/keys.py

    1. Registering a Dropbox application: Go to https://www.dropbox.com/developers/apps and follow the steps to create a new application. At the end, add the app key and secret to the keys.py file. Use the following arguments while creating the Dropbox application:
    • Status: Development
    • Permission type: Full Dropbox
    • OAuth redirect URIs: (you can leave it empty)
    • Drop-ins domains: (you can leave it empty)
    1. Registering a Twitter application: Go to https://dev.twitter.com/apps and follow the steps to create a new application. At the end, add the consumer key and secret to the file keys.py. Use the following arguments while creating the Twitter application:
    • Access level: Read, write, and direct messages
    • Callback URL: (not relevant)
    1. Registering a LinkedIn application Go to http://developer.linkedin.com/documents/authentication. Step 1 explain how to create a new application. Once you've registered your LinkedIn app, you will be provided with an API key and secret key that should be added to the keys.py file. Use the following arguments while creating the Linkedin application:
    • In OAuth User Agreement, check options: r_network, rw_groups, r_fullprofile, r_contactinfo, rw_nus. There is no need to set any of the Redirect URLs.
    1. Registering a Foursquare application Go to https://foursquare.com/developers/apps and follow the steps to create a new application. At the end, add the client id and secret to the file keys.py. Use the following arguments while creating the Foursquare application:

    1. Registering a Facebook application Go to https://developers.facebook.com/apps and follow the steps to create a new application. Client id and secret should be added to the file keys.py.

    2. Registering a Google application (The instructions below are following the old console interface, that can still be used. A new console is available and the instructions will be a little bit different. For the new console you will find the credentials and services under the option APIs & auth.) Go to https://code.google.com/apis/console and register a new application. In the option “API Access” you can find your client id and secret. Besides that, you will have to enable each service that you are planning to use. It can be done under “Services”. Use the following arguments while creating the Google application:

Usage

  • Start mongoDB http://docs.mongodb.org/manual/tutorial/manage-mongodb-processes/

  • Start server and call app interface

  • The DigitalSelf interface

    • The first page asks for a user and password. A new user does not need to be registered a priori to have access to the application. The registration is done as soon as the user enters with the desired username and password.
    • In the main page the user can choose between the following options:
      • Register for Services: show all APIs implemented allowing users to register with a variety of different services.
      • Get Data from Services: user can get as much data as allowed by the APIs. This step requires the user to be authenticated with the service (Register for Services) before any data can be requested.
      • Search your Data (Not working on this version)
      • Delete Account and Data: clean all user data from database

  • The data collected One data collection is created for each service implemented: dropbox_data, facebook_data, foursquare_data, gcal_data, gcontacts_data, gmail_data, gplus_data, linked_in_data and twitter_data. In mongoDB, collections are composed by documents. Each document represents an unique data retrieved from a service. Most documents have the following fields:

    • data: contains the raw data retrieved
    • data_type: specifies the type of data collected (e.g. Facebook feed, Facebook photo)
    • {service}_user: user authenticated to access the specific service (API)
    • neemi_user: DigitalSelf user
    • time: time the data was retrieved or time the data was originally created

    Some observations:

    • Dropbox: Metadata and content are retrieved. Metadata is stored in the field “data” and a file content is stored in the field “file_content”.
    • Gmail Metadata and content are retrieved, both are stored in the field “data”
Possible problems