Adding support for datastore emulator in system test.

CONTRIBUTING.rst

@@ -6,23 +6,23 @@ Contributing
 #. Make sure that your commit messages clearly describe the changes.
 #. Send a pull request.
 
-Here are some guidelines for hacking on gcloud-python.
+Here are some guidelines for hacking on ``gcloud-python``.
 
 Using a Development Checkout
 ----------------------------
 
-You'll have to create a development environment to hack on gcloud-python,
+You'll have to create a development environment to hack on ``gcloud-python``,
 using a Git checkout:
 
-- While logged into your GitHub account, navigate to the gcloud-python repo on
-  GitHub.
+- While logged into your GitHub account, navigate to the ``gcloud-python`` repo
+  on GitHub.
 
   https://github.com/GoogleCloudPlatform/gcloud-python
 
-- Fork and clone the gcloud-python repository to your GitHub account by
+- Fork and clone the ``gcloud-python`` repository to your GitHub account by
   clicking the "Fork" button.
 
-- Clone your fork of gcloud-python from your GitHub account to your local
+- Clone your fork of ``gcloud-python`` from your GitHub account to your local
   computer, substituting your account username and specifying the destination
   as "hack-on-gcloud".  E.g.::
 
@@ -39,25 +39,25 @@ using a Git checkout:
 Now your local repo is set up such that you will push changes to your GitHub
 repo, from which you can submit a pull request.
 
-- Create a virtualenv in which to install gcloud-python::
+- Create a virtualenv in which to install ``gcloud-python``::
 
    $ cd ~/hack-on-gcloud
-   $ virtualenv -ppython2.7 env
+   $ virtualenv --python python2.7 env
 
   Note that very old versions of virtualenv (virtualenv versions below, say,
   1.10 or thereabouts) require you to pass a ``--no-site-packages`` flag to
   get a completely isolated environment.
 
-  You can choose which Python version you want to use by passing a ``-p``
-  flag to ``virtualenv``.  For example, ``virtualenv -ppython2.7``
+  You can choose which Python version you want to use by passing a ``--python``
+  flag to ``virtualenv``.  For example, ``virtualenv --python python2.7``
   chooses the Python 2.7 interpreter to be installed.
 
   From here on in within these instructions, the ``~/hack-on-gcloud/env``
   virtual environment you created above will be referred to as ``$VENV``.
   To use the instructions in the steps that follow literally, use the
   ``export VENV=~/hack-on-gcloud/env`` command.
 
-- Install gcloud-python from the checkout into the virtualenv using
+- Install ``gcloud-python`` from the checkout into the virtualenv using
   ``setup.py develop``.  Running ``setup.py develop`` *must* be done while
   the current working directory is the ``gcloud-python`` checkout directory::
 
@@ -83,7 +83,7 @@ and then ``pip install`` the dependencies again::
 Adding Features
 ---------------
 
-In order to add a feature to gcloud-python:
+In order to add a feature to ``gcloud-python``:
 
 - The feature must be documented in both the API and narrative
   documentation (in ``docs/``).
@@ -124,18 +124,18 @@ Exceptions to PEP8:
 Running Tests
 --------------
 
-- To run all tests for gcloud-python on a single Python version, run
+- To run all tests for ``gcloud-python`` on a single Python version, run
   ``nosetests`` from your development virtualenv (See
   *Using a Development Checkout* above).
 
-- To run the full set of gcloud-python tests on all platforms, install ``tox``
-  (https://testrun.org/tox/) into a system Python.  The ``tox`` console
+- To run the full set of ``gcloud-python`` tests on all platforms, install
+  ``tox`` (https://testrun.org/tox/) into a system Python.  The ``tox`` console
   script will be installed into the scripts location for that Python.  While
-  ``cd``'ed to the gcloud-python checkout root directory (it contains ``tox.ini``),
-  invoke the ``tox`` console script.  This will read the ``tox.ini`` file and
-  execute the tests on multiple Python versions and platforms; while it runs,
-  it creates a virtualenv for each version/platform combination.  For
-  example::
+  ``cd``'ed to the ``gcloud-python`` checkout root directory (it contains
+  ``tox.ini``), invoke the ``tox`` console script.  This will read the
+  ``tox.ini`` file and execute the tests on multiple Python versions and
+  platforms; while it runs, it creates a virtualenv for each version/platform
+  combination.  For example::
 
    $ sudo /usr/bin/pip install tox
    $ cd ~/hack-on-gcloud/
@@ -211,6 +211,36 @@ Running System Tests
 
    $ python system_tests/clear_datastore.py
 
+- System tests can also be run against local `emulators`_ that mock
+  the production services. For example, to run the system tests
+  with the ``datastore`` emulator, first start the emulator and
+  take note of the process ID::
+
+   $ gcloud beta emulators datastore start &
+   [1] 33333
+
+  then determine the environment variables needed to interact with
+  the emulator::
+
+   $ gcloud beta emulators datastore env-init
+   export DATASTORE_LOCAL_HOST=localhost:8417
+   export DATASTORE_HOST=http://localhost:8417
+   export DATASTORE_DATASET=gcloud-settings-app-id
+   export DATASTORE_PROJECT_ID=gcloud-settings-app-id
+
+  using these environment variables run the emulator::
+
+   $ DATASTORE_HOST=http://localhost:8471 \
+   >   DATASTORE_DATASET=gcloud-settings-app-id \
+   >   GCLOUD_NO_PRINT=true \
+   >   python system_tests/run_system_test.py --package=datastore
+
+  and after completion stop the emulator::
+
+   $ kill 33333
+
+.. _emulators: https://cloud.google.com/sdk/gcloud/reference/beta/emulators/
+
 Test Coverage
 -------------
 
@@ -230,22 +260,22 @@ changed to reflect the bug fix, ideally in the same commit that fixes the bug
 or adds the feature.
 
 To build and review docs (where ``$VENV`` refers to the virtualenv you're
-using to develop gcloud-python):
+using to develop ``gcloud-python``):
 
 1. After following the steps above in "Using a Development Checkout", install
    Sphinx and all development requirements in your virtualenv::
 
      $ cd ~/hack-on-gcloud
      $ $VENV/bin/pip install Sphinx
 
-2. Change into the ``docs`` directory within your gcloud-python checkout and
+2. Change into the ``docs`` directory within your ``gcloud-python`` checkout and
    execute the ``make`` command with some flags::
 
      $ cd ~/hack-on-gcloud/gcloud-python/docs
      $ make clean html SPHINXBUILD=$VENV/bin/sphinx-build
 
    The ``SPHINXBUILD=...`` argument tells Sphinx to use the virtualenv Python,
-   which will have both Sphinx and gcloud-python (for API documentation
+   which will have both Sphinx and ``gcloud-python`` (for API documentation
    generation) installed.
 
 3. Open the ``docs/_build/html/index.html`` file to see the resulting HTML

scripts/pep8_on_repo.py

@@ -21,6 +21,7 @@
 
 import os
 import subprocess
+import sys
 
 
 def main():
@@ -32,7 +33,8 @@ def main():
     python_files = python_files.strip().split()
 
     pep8_command = ['pep8'] + python_files
-    subprocess.call(pep8_command)
+    status_code = subprocess.call(pep8_command)
+    sys.exit(status_code)
 
 
 if __name__ == '__main__':

system_tests/clear_datastore.py

@@ -14,17 +14,16 @@
 
 """Script to populate datastore with system test data."""
 
+from __future__ import print_function
+
+import os
+
 from six.moves import input
 
 from gcloud import datastore
-from gcloud.datastore import client
 from gcloud.environment_vars import TESTS_DATASET
 
 
-client.DATASET = TESTS_DATASET
-CLIENT = datastore.Client()
-
-
 FETCH_MAX = 20
 ALL_KINDS = [
     'Character',
@@ -36,9 +35,14 @@
 TRANSACTION_MAX_GROUPS = 5
 
 
-def fetch_keys(kind, fetch_max=FETCH_MAX, query=None, cursor=None):
+def print_func(message):
+    if os.getenv('GCLOUD_NO_PRINT') != 'true':
+        print(message)
+
+
+def fetch_keys(kind, client, fetch_max=FETCH_MAX, query=None, cursor=None):
     if query is None:
-        query = CLIENT.query(kind=kind, projection=['__key__'])
+        query = client.query(kind=kind, projection=['__key__'])
 
     iterator = query.fetch(limit=fetch_max, start_cursor=cursor)
 
@@ -53,46 +57,49 @@ def get_ancestors(entities):
     return list(set(key_roots))
 
 
-def remove_kind(kind):
+def remove_kind(kind, client):
     results = []
 
-    query, curr_results, cursor = fetch_keys(kind)
+    query, curr_results, cursor = fetch_keys(kind, client)
     results.extend(curr_results)
     while curr_results:
-        query, curr_results, cursor = fetch_keys(kind, query=query,
-                                                 cursor=cursor)
+        query, curr_results, cursor = fetch_keys(
+            kind, client, query=query, cursor=cursor)
         results.extend(curr_results)
 
     if not results:
         return
 
     delete_outside_transaction = False
-    with CLIENT.transaction():
+    with client.transaction():
         # Now that we have all results, we seek to delete.
-        print('Deleting keys:')
-        print(results)
+        print_func('Deleting keys:')
+        print_func(results)
 
         ancestors = get_ancestors(results)
         if len(ancestors) > TRANSACTION_MAX_GROUPS:
             delete_outside_transaction = True
         else:
-            CLIENT.delete_multi([result.key for result in results])
+            client.delete_multi([result.key for result in results])
 
     if delete_outside_transaction:
-        CLIENT.delete_multi([result.key for result in results])
+        client.delete_multi([result.key for result in results])
 
 
-def remove_all_entities():
-    print('This command will remove all entities for the following kinds:')
-    print('\n'.join(['- ' + val for val in ALL_KINDS]))
-    response = input('Is this OK [y/n]? ')
-    if response.lower() != 'y':
-        print('Doing nothing.')
-        return
-
+def remove_all_entities(client=None):
+    if client is None:
+        # Get a client that uses the test dataset.
+        client = datastore.Client(dataset_id=TESTS_DATASET)
     for kind in ALL_KINDS:
-        remove_kind(kind)
+        remove_kind(kind, client)
 
 
 if __name__ == '__main__':
-    remove_all_entities()
+    print_func('This command will remove all entities for '
+               'the following kinds:')
+    print_func('\n'.join(['- ' + val for val in ALL_KINDS]))
+    response = input('Is this OK [y/n]? ')
+    if response.lower() == 'y':
+        remove_all_entities()
+    else:
+        print_func('Doing nothing.')

system_tests/datastore.py

@@ -13,18 +13,25 @@
 # limitations under the License.
 
 import datetime
+import os
+
 import unittest2
 
 from gcloud._helpers import UTC
 from gcloud import datastore
 from gcloud.datastore import client
+from gcloud.environment_vars import GCD_DATASET
 from gcloud.environment_vars import TESTS_DATASET
 from gcloud.exceptions import Conflict
 # This assumes the command is being run via tox hence the
 # repository root is the current directory.
+from system_tests import clear_datastore
 from system_tests import populate_datastore
 
 
+EMULATOR_DATASET = os.getenv(GCD_DATASET)
+
+
 class Config(object):
     """Run-time configuration to be modified at set-up.
 
@@ -35,8 +42,17 @@ class Config(object):
 
 
 def setUpModule():
-    client.DATASET = TESTS_DATASET
-    Config.CLIENT = datastore.Client()
+    if EMULATOR_DATASET is None:
+        client.DATASET = TESTS_DATASET
+        Config.CLIENT = datastore.Client()
+    else:
+        Config.CLIENT = datastore.Client(dataset_id=EMULATOR_DATASET)
+        populate_datastore.add_characters(client=Config.CLIENT)
+
+
+def tearDownModule():
+    if EMULATOR_DATASET is not None:
+        clear_datastore.remove_all_entities(client=Config.CLIENT)
 
 
 class TestDatastore(unittest2.TestCase):
@@ -271,15 +287,20 @@ def test_projection_query(self):
         self.assertEqual(catelyn_stark_dict,
                          {'name': 'Catelyn', 'family': 'Stark'})
 
-        catelyn_tully_entity = entities[3]
+        if EMULATOR_DATASET is None:
+            catelyn_tully_entity = entities[3]
+            sansa_entity = entities[8]
+        else:
+            catelyn_tully_entity = entities[8]
+            sansa_entity = entities[7]
+
         catelyn_tully_dict = dict(catelyn_tully_entity)
         self.assertEqual(catelyn_tully_dict,
                          {'name': 'Catelyn', 'family': 'Tully'})
 
         # Check both Catelyn keys are the same.
         self.assertEqual(catelyn_stark_entity.key, catelyn_tully_entity.key)
 
-        sansa_entity = entities[8]
         sansa_dict = dict(sansa_entity)
         self.assertEqual(sansa_dict, {'name': 'Sansa', 'family': 'Stark'})