Use / recommend use of data-dir for datastore emulator

[dhermes]

@pcostell moving discussion over here. I wanted to discuss two things:

1. How can I incorporate the indexes into runs of the emulator?
2. Do you think we should add the emulator to our automated testing?

[dhermes]

When I tried to run it as-is with the current location of index.yaml I got an error:

$ gcloud beta emulators datastore start --data-dir=gcloud-python/system_tests/data
...
[datastore] Invalid project directory: .../gcloud-python/system_tests/data/.
    Did you create it using 'create'?
...

when doing the Java specific thing you suggested

$ mkdir gcloud-python/system_tests/data/WEB-INF
$ cp gcloud-python/system_tests/data/index.yaml \
>   gcloud-python/system_tests/data/WEB-INF

I ended up with the same error

[pcostell]

1. For testing, I would just have your static yaml of indexes that you use in production and in your emulator tests and keep it checked in to github. Within your <data-dir>/WEB-INF there is an index.yaml file where you can put the indexes.
2. Yes I would include it assuming it isn't too much work.

Hmm, it should do that for you automatically:

~ % gcloud beta emulators datastore start --data-dir=~/tmp/emdir

Results in:

Executing: /bin/bash /Users/pcostello/.local/google-cloud-sdk/platform/gcd/gcd.sh create -project_id=pcostello-cloud /tmp/emdir
[datastore] Created new Cloud Datastore project in '/Users/pcostello//tmp/emdir' with project ID 'pcostello-cloud'.
Executing: /bin/bash /Users/pcostello/.local/google-cloud-sdk/platform/gcd/gcd.sh start --host=localhost --port=8863 --store_on_disk=True --consistency=0.9 --allow_remote_shutdown ~/tmp/emdir
...

[dhermes]

So you're saying I should run the emulator with an empty temp directory and then manually copy the index file into <data-dir>/WEB-INF?

[pcostell]

ya probably. We should probably make it easier to get this setup specifically for indexes -- maybe a different flag?

[dhermes]

Yes I agree. It's especially painful since start just launches in the foreground.

[pcostell]

@eddavisson thoughts here? One of the reasons to require the directory is so that we can then upload the index.yaml file. But if we include index.yaml as a specific flag, we could change update/vacuum indexes to also just accept the index.yaml file.

Since we've moved away from xml, there is now only one file to deal with.

[eddavisson]

The local_db.bin file also ends up in the data directory.

When we were developing the Cloud SDK CLI, we tried to target two use cases:

1. Unit testing. Neither the data nor the required indexes are persisted on disk.
2. Interactive testing. Data and required indexes are persisted on disk. Resulting index.yaml file is suitable for update/vacuum commands.

Does 1) capture your use case (or are you tracking index usage across tests)?

[pcostell]

This isn't quite handled for unit testing in the case you want to do emulator + prod integration tests. In this case not persisting the indexes caused issues because prod used one index and the emulator generated a different index.

[dhermes]

@pcostell The above is mostly about how to run with our own indexes, which seems like a work in progress?

What about my second question? Do you think we should run the emulator tests on a regular basis or is hitting the live API on every merge to master sufficient?

[pcostell]

I don't think you need to run them as part of the merge -- the live API is probably better. However, integrating them with your normal testing infrastructure can make normal development iteration a lot easier.

[dhermes]

Good call. I'm doing to add an optional tox environment (e.g. we run tox -e docs-rtd if we want a preview of http://gcloud-python.readthedocs.org/ which is different than our "main" docs http://googlecloudplatform.github.io/gcloud-python/stable/ which runs as tox -e docs)

[dhermes]

@pcostell I sent a PR (#1393) to make running the emulator easier on our users.

• Is there a place to track the progress of updates to the tool?
• Is there a reason that gcloud beta emulators datastore start launches in the foreground only and prints lots of things to stdout? It seems it'd make sense to have an equivalent datastore stop command or at least a flag to allow logging to be done to a file rather than stdout. Though maybe you considering this and deemed redirection to a file to cover the case?

[lukesneeringer]

Hello,
One of the challenges of maintaining a large open source project is that sometimes, you can bite off more than you can chew. As the lead maintainer of google-cloud-python, I can definitely say that I have let the issues here pile up.

As part of trying to get things under control (as well as to empower us to provide better customer service in the future), I am declaring a "bankruptcy" of sorts on many of the old issues, especially those likely to have been addressed or made obsolete by more recent updates.

My goal is to close stale issues whose relevance or solution is no longer immediately evident, and which appear to be of lower importance. I believe in good faith that this is one of those issues, but I am scanning quickly and may occasionally be wrong. If this is an issue of high importance, please comment here and we will reconsider. If this is an issue whose solution is trivial, please consider providing a pull request.

Thank you!