Merge pull request #294 from jpgill86/gdrive

Add capability to download files from Google Drive

README.rst

@@ -55,9 +55,9 @@ To use the app, first organize your datasets in a *metadata file* like this
 
 Open your metadata file in *neurotic* and choose a dataset. If the data and
 video files aren't already on your local computer, the app can download them
-for you, even from a password-protected server. Finally, click launch and the
-app will use a standard viewer layout to display your data to you using
-ephyviewer_.
+for you, even from a password-protected server or from Google Drive. Finally,
+click launch and the app will use a standard viewer layout to display your data
+to you using ephyviewer_.
 
 |Example screenshot|
 
@@ -87,12 +87,12 @@ Electrophysiologists will find this tool useful even if they don't need the
 video synchronization feature!
 
 **Portability is easy with neurotic!** Use relative paths in your metadata file
-along with a remotely accessible data store such as GIN_ to make your metadata
-file fully portable. The same metadata file can be copied to a different
-computer, and downloaded files will automatically be saved to the right place.
-Data stores can be password protected and *neurotic* will prompt you for a user
-name and password. This makes it easy to share the *neurotic* experience with
-your colleagues! 🤪
+along with a remotely accessible data store such as GIN_ or a Shared Drive on
+Google Drive to make your metadata file fully portable. The same metadata file
+can be copied to a different computer, and downloaded files will automatically
+be saved to the right place. Data stores can be password protected and
+*neurotic* will prompt you for a user name and password. This makes it easy to
+share the *neurotic* experience with your colleagues! 🤪
 
 Installing *neurotic*
 ---------------------
@@ -319,7 +319,7 @@ Documentation
 -------------
 
 For detailed information on configuring metadata, working examples, the API
-reference guide, and release notes, see the Documentation_.
+reference guide, release notes, and more, see the Documentation_.
 
 
 .. |neurotic logo| image:: https://raw.githubusercontent.com/jpgill86/neurotic/master/neurotic/gui/icons/img/neurotic-logo-30.png

docs/api.rst

@@ -38,6 +38,7 @@ as ``neurotic.MetadataSelector``.
    api/data
    api/download
    api/ftpauth
+   api/gdrive
    api/metadata
 
 

docs/api/gdrive.rst

@@ -0,0 +1,6 @@
+.. _api-gdrive:
+
+``neurotic.datasets.gdrive``
+=============================
+
+.. automodule:: neurotic.datasets.gdrive

docs/gdrive.rst

@@ -0,0 +1,162 @@
+.. _gdrive:
+
+Configuring for Google Drive
+============================
+
+*neurotic* can download files from Google Drive (see :ref:`gdrive-urls` for
+details on how to specify remote URLs). **However, before this capability can
+be used, you must complete some manual configuration.**
+
+.. _gdrive-credentials:
+
+Generating a Client Credentials File
+------------------------------------
+
+Because of limitations stated in Google's Terms of Service on what
+client-identifying credentials may be included with open source desktop
+applications, it is not possible for *neurotic* to supply users with an
+essential file needed for Google Drive access [1]_. This file is called the
+client credentials file, and you will need to generate one yourself. The
+instructions in this section will walk you through this one-time process.
+
+The file produced by following these steps, ``credentials.json``, does not
+provide access to your Google account, so your account is not compromised if
+the file falls into the wrong hands. However, it does identify you to Google as
+the owner of your installation of *neurotic* when you use it to download files;
+for this reason, you are discouraged from distributing this file to others, as
+abuse of it could cause Google to invalidate it and sanction your account.
+
+The steps below take advantage of a tutorial. If you know what you are doing,
+you can generate your own client credentials file from the `Google API
+Console`_ instead of using the tutorial, but that procedure is much more
+complicated. The tutorial is comparatively easy.
+
+1. Click the following link to open a web page. The web page is a tutorial for
+   accessing Google Drive programmatically, as *neurotic* does. You will not
+   need to follow the steps of this tutorial, but it does provide a convenient
+   shortcut for generating the client credentials file.
+
+        https://developers.google.com/drive/api/v3/quickstart/python
+
+2. Click the button labeled "Enable the Drive API" in step 1 of the tutorial.
+   You may be prompted to log into Google, and then a series of dialog boxes
+   will take you through the configuration process. Follow these steps:
+
+    a. Enter new project name: "neurotic".
+    b. Accept terms of service, if necessary.
+    c. Click "Next".
+    d. Configure your OAuth client: Choose "Desktop app".
+    e. Click "Create".
+    f. Click "Download Client Configuration", which will download a file called
+       ``credentials.json``.
+    g. Click "Done".
+
+3. Launch *neurotic*. Under the Help menu, click "Open Google Drive credentials
+   directory". Move the ``credentials.json`` file into this folder.
+
+4. Finally, close and restart *neurotic*.
+
+.. _gdrive-authorization:
+
+Authorizing *neurotic* to Access Google Drive
+---------------------------------------------
+
+After you have created a client credentials file and saved it to the right
+location, you can attempt to download files from Google Drive using a properly
+configured metadata file (see :ref:`gdrive-urls` for details). When you do this
+for the first time, or when you use the "Request Google Drive authorization
+now" action under the Help menu, you will need to complete the following steps
+to authorize *neurotic* to access your Google Drive:
+
+1. Your web browser will open and ask you to sign into a Google account before
+   continuing to "Quickstart". This name comes from the tutorial used to make
+   the client credentials file.
+
+   .. image:: _static/gdrive-auth-1.png
+    :alt: Google Drive authorization flow, step 1
+    :width: 300
+
+2. You will be presented with an intimidating warning about the app being
+   unverified by Google. You will see your own email listed as the developer
+   because this warning is associated with the client credentials file you
+   created.
+
+   .. image:: _static/gdrive-auth-2.png
+    :alt: Google Drive authorization flow, step 2
+    :width: 400
+
+3. Due to the terms-of-service limitations mentioned earlier, it is not
+   possible for *neurotic* to distribute a client credentials file that would
+   avoid presenting you with this warning. This policy exists because open
+   source desktop applications can be modified by anyone with access to the
+   system at any time, making it impossible for Google to certify that the
+   mutable app has any particular identity and is safe. If you trust the
+   application and are willing to proceed without Google's stamp of approval,
+   click "Advanced" to reveal more text, and then click "Go to Quickstart
+   (unsafe)", which will take you past this warning.
+
+   .. image:: _static/gdrive-auth-3.png
+    :alt: Google Drive authorization flow, step 3
+    :width: 400
+
+4. To download files from your Google Drive, *neurotic* needs the privileges to
+   access and read those files. Click "Allow" to indicate that you want to
+   allow this. Note that again the app is referred to as "Quickstart" due to
+   the settings of the tutorial used to create the client credentials file.
+
+   .. image:: _static/gdrive-auth-4.png
+    :alt: Google Drive authorization flow, step 4
+    :width: 300
+
+5. Click "Allow" another time to confirm. Again, "Quickstart" refers to your
+   client credentials file, which *neurotic* will use.
+
+   .. image:: _static/gdrive-auth-5.png
+    :alt: Google Drive authorization flow, step 5
+    :width: 300
+
+6. When you see this message in your browser, you can close it: "The
+   authorization flow has completed. You may close this window."
+
+The authorization process should now be complete, and you can begin using
+*neurotic* to access and download Google Drive files.
+
+.. _gdrive-save-token:
+
+Making Google Drive Authorization Persistent
+--------------------------------------------
+
+By default, authorization persists only until *neurotic* is closed. Each time
+*neurotic* is restarted and you want to download from Google Drive again, you
+will need to repeat the authorization procedure described above. You can avoid
+this by configuring *neurotic* to retain the products of authorization (access
+and refresh tokens) indefinitely. See :ref:`global-config` for details on the
+global configuration file; by setting the ``save_token`` parameter under the
+``gdrive`` heading to ``true``, you can minimize the frequency of authorization
+requests.
+
+.. warning::
+
+    Enabling ``save_token`` is not recommended on systems used by others you do
+    not trust. These others will be able to download files from your Google
+    Drive using *neurotic* with the same level of ease you experience, and with
+    access to the token file they could use it outside of *neurotic* to gain
+    read-only access to your Google Drive and your Shared Drives.
+
+.. _gdrive-purge-token:
+
+Purging Google Drive Authorization
+----------------------------------
+
+If you need to use a different Google account than the one you previously
+authorized *neurotic* to use, or if you set ``save_token=true`` and now want to
+remove the persistent access and refresh tokens from your system, you can use
+the "Purge Google Drive authorization token" action from the Help menu. After
+using this, you will need to complete the authorization procedure again, and
+you will have the opportunity to select a different Google account.
+
+
+.. [1] See `this StackOverflow question
+       <https://stackoverflow.com/q/27585412>`_ for an informal discussion.
+
+.. _Google API Console: https://console.developers.google.com

docs/globalconfig.rst

@@ -3,16 +3,18 @@
 Changing Default Behavior
 =========================
 
-Default parameters used by the command line interface for launching the app,
-such as which metadata file to open initially, can be configured using global
-configuration settings located in ``.neurotic/neurotic-config.txt`` in your
-home directory:
+Default parameters used by the app and by the command line interface, such
+as which metadata file to open initially or whether Google Drive access tokens
+should be stored indefinitely, can be configured using global configuration
+settings located in ``.neurotic/neurotic-config.txt`` in your home directory:
 
  - Windows: ``C:\Users\<username>\.neurotic\neurotic-config.txt``
  - macOS: ``/Users/<username>/.neurotic/neurotic-config.txt``
  - Linux: ``/home/<username>/.neurotic/neurotic-config.txt``
 
 The file can be opened easily using the "View global config file" menu action.
+You may edit it to customize your settings. The next time you launch the app or
+use the command line interface, your changes should be in effect.
 
 If this file does not exist when *neurotic* is launched, the following template
 is created for you:

docs/index.rst

@@ -28,6 +28,7 @@ at the same datasets!
    citations
    metadata
    examples
+   gdrive
    globalconfig
    api
    releasenotes

docs/metadata.rst

@@ -88,8 +88,9 @@ Remote Data Available for Download
 
 Data files must be stored on the local computer for *neurotic* to load them and
 display their contents. If the files are available for download from a remote
-server, *neurotic* can be configured to download them for you to the local
-directory specified by ``data_dir`` if the files aren't there already.
+server (e.g., a web site, an FTP server, or Google Drive), *neurotic* can be
+configured to download them for you to the local directory specified by
+``data_dir`` if the files aren't there already.
 
 Specify the URL to the directory containing the data on the remote server using
 ``remote_data_dir``. *neurotic* expects the local ``data_dir`` and the
@@ -164,13 +165,56 @@ to the metadata file. In the example above, if the metadata file is located in
 .. note::
 
     **Portability is easy with neurotic!** Use relative paths in your metadata
-    file along with a remotely accessible data store such as GIN_ to make your
-    metadata file fully portable. The example above is a simple model of this
-    style. A metadata file like this can be copied to a different computer, and
-    downloaded files will automatically be saved to the right place. Data
-    stores can be password protected and *neurotic* will prompt you for a user
-    name and password. This makes it easy to share the *neurotic* experience
-    with your colleagues! 🤪
+    file along with a remotely accessible data store such as GIN_ or a Shared
+    Drive on Google Drive (see details below) to make your metadata file fully
+    portable. The example above is a simple model of this style. A metadata
+    file like this can be copied to a different computer, and downloaded files
+    will automatically be saved to the right place. Data stores can be password
+    protected and *neurotic* will prompt you for a user name and password. This
+    makes it easy to share the *neurotic* experience with your colleagues! 🤪
+
+.. _gdrive-urls:
+
+URLs to Use with Google Drive
+.............................
+
+After completing some essential manual setup (see :ref:`gdrive`), *neurotic*
+can retrieve remote files from Google Drive using URL-like paths of the
+following form::
+
+    gdrive://<drive name>/<folder 1>/<...>/<folder N>/<file name>
+
+The ``<drive name>`` may be "``My Drive``" for files located in a personal
+Google Drive, or it may be the name of a Shared Drive that the user has
+permission to access.
+
+Note that these URL-like paths are not equivalent to ordinary URLs
+associated with Google Drive files, such as shareable links, which are
+composed of pseudorandom file IDs and do not reveal anything about the name
+of the file or the folders containing it. Instead, these URL-like paths allow
+you to structure your metadata with the file tree hierarchy in mind, so that
+relative paths can be used.
+
+For example, with datasets stored in subdirectories "datasets/A", "datasets/B",
+etc., of a Shared Drive titled "Lab Project Data", you
+could use this metadata to mirror the files locally:
+
+.. code-block:: yaml
+
+    neurotic_config:
+        remote_data_root:   gdrive://Lab Project Data/datasets
+
+    Dataset A:
+        data_dir:           A
+        remote_data_dir:    A
+        data_file:          data.axgx
+        video_file:         video.mp4
+
+    Dataset B:
+        data_dir:           B
+        remote_data_dir:    B
+        data_file:          data.axgx
+        video_file:         video.mp4
 
 .. _gin-urls:
 

docs/overview.rst

@@ -44,9 +44,9 @@ To use *neurotic*, first organize your datasets in a *metadata file* like this
 
 Open your metadata file in *neurotic* and choose a dataset. If the data and
 video files aren't already on your local computer, the app can download them
-for you, even from a password-protected server. Finally, click launch and the
-app will use a standard viewer layout to display your data to you using
-ephyviewer_.
+for you, even from a password-protected server or from Google Drive. Finally,
+click launch and the app will use a standard viewer layout to display your data
+to you using ephyviewer_.
 
 |Example screenshot|
 
@@ -76,12 +76,12 @@ Electrophysiologists will find this tool useful even if they don't need the
 video synchronization feature!
 
 **Portability is easy with neurotic!** Use relative paths in your metadata file
-along with a remotely accessible data store such as GIN_ to make your metadata
-file fully portable. The same metadata file can be copied to a different
-computer, and downloaded files will automatically be saved to the right place.
-Data stores can be password protected and *neurotic* will prompt you for a user
-name and password. This makes it easy to share the *neurotic* experience with
-your colleagues! 🤪
+along with a remotely accessible data store such as GIN_ or a Shared Drive on
+Google Drive to make your metadata file fully portable. The same metadata file
+can be copied to a different computer, and downloaded files will automatically
+be saved to the right place. Data stores can be password protected and
+*neurotic* will prompt you for a user name and password. This makes it easy to
+share the *neurotic* experience with your colleagues! 🤪
 
 
 .. |Example screenshot| image:: _static/example-screenshot.png

neurotic/__init__.py

@@ -89,6 +89,12 @@ def format(self, record):
         'ui_scale': 'medium',
         'theme': 'light',
     },
+    'gdrive': {
+        # parameters for Google Drive access
+        'credentials_file': os.path.join(neurotic_dir, 'gdrive-creds', 'credentials.json'),
+        'token_file': os.path.join(neurotic_dir, 'gdrive-creds', 'gdrive-token.pickle'),
+        'save_token': False,
+    },
 }
 
 # keep a copy of the original config before it is modified
@@ -148,6 +154,14 @@ def update_global_config_from_file(file=global_config_file):
     logger.error(f'Ignoring global config file due to parsing error ({global_config_file}): {e}')
 
 
+# create directories for storing Google Drive credentials and tokens if
+# necessary
+for file in [global_config['gdrive']['credentials_file'],
+             global_config['gdrive']['token_file']]:
+    if file and not os.path.exists(os.path.dirname(file)):
+        os.mkdir(os.path.dirname(file))
+
+
 from .datasets import *
 from .gui import *
 from .scripts import *

neurotic/datasets/__init__.py

@@ -4,6 +4,7 @@
 """
 
 from ..datasets.ftpauth import *
+from ..datasets.gdrive import *
 from ..datasets.download import *
 from ..datasets.metadata import *
 from ..datasets.data import *