Batch commit throws ServiceUnavailable: 503 Backend Error

[charlesbclifford]

See below:

OPERATING SYSTEM

$ uname -mrs
Linux 3.19.5-100.fc20.x86_64 x86_64

ENVIRONMENT VARIABLES

PYTHON 2.7

export PYTHONHOME=/usr/bin/python2.7

export PYTHONPATH=/usr/lib/python2.7/site-packages

PYTHON modules for GCP

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/bin
export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib
export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib/googlecloudsdk
export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib/googlecloudsdk/gcloud
export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib/googlecloudapis

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/bq

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gcutil

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gcutil/lib

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gsutil

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gsutil/gslib

export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gsutil/third_party

export PATH=$PATH:$PYTHONHOME

export PATH=$PATH:$PYTHONPATH

Google AppEngine SDK

export PATH=$PATH:/home/charles/google_appengine/

Google Cloud SDK

export CLOUDSDK_CONFIG=/home/charles/.config/gcloud
export CLOUDSDK_PYTHON=/usr/bin/python2.7
export CLOUDSDK_PYTHON_SITEPACKAGES=1

BOTO

export BOTO_CONFIG=/home/charles/.boto

export NVM_DIR="/home/charles/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm

The next line updates PATH for the Google Cloud SDK.

source '/home/charles/google-cloud-sdk/path.bash.inc'

The next line enables bash completion for gcloud.

source '/home/charles/google-cloud-sdk/completion.bash.inc'

FORM OF AUTHENTICATION

The credentialed account being used is the owner of the Google project, for which Cloud Datastore API is enabled.

PYTHON CODE

from gcloud import storage
storage.set_default_bucket('')
storage.set_default_project(_PROJECT_NAME)
from gcloud import datastore
datastore.set_default_dataset_id(_DATA_SET_ID)

datastore.set_defaults(namespace=_NAMESPACE)

from gcloud.datastore.batch import Batch

import traceback

def Load_Datastore():
try:
Get_Bucket_Metadata()
#set configuration variables
storage_conn = storage.get_connection()
storage.set_default_connection(storage_conn)
datastore_conn = datastore.get_connection()
datastore.set_default_connection(datastore_conn)

    for _BUCKET_NAME in _BUCKET_LIST:
        print "Bucket Name: %s" % (_BUCKET_NAME)
    .
    .
    .

                #an empty list is false
                if not _RECORDS:
                    print "_RECORDS is empty"
                else:
                    print "there are %s records to write into Datastore"%(str(len(_RECORDS)))
                    with datastore.batch.Batch(connection=datastore_conn) as batch:
                        _ENTITIES=[]                                           
                        for _RECORD in _RECORDS:
                            _PROPERTIES=[]

                            if _RECORD:
                                for _KEY_VALUES in _RECORD:
                                    _KEY_VALUE_PAIR=_KEY_VALUES.split('|')
                                    if str(_KEY_VALUE_PAIR[0])=='ROW_KEY':
                                        _ENTITY_KEY=str(_KEY_VALUE_PAIR[1])
                                    else:
                                        _PROPERTIES.append("%s|%s"%(str(_KEY_VALUE_PAIR[0]),str(_KEY_VALUE_PAIR[1])))

                            if not _PROPERTIES:   
                                print "_PROPERTIES is empty"
                            else:                                              
                                try:
                                    key = datastore.Key(_SOURCE_TABLE_NAME.upper(),_ENTITY_KEY,namespace=_NAMESPACE,dataset_id=_DATA_SET_ID)

                                    try:
                                        DS_ENTITY = datastore.Entity(key)
                                        for _PROPERTY in _PROPERTIES:
                                            _NAME_VALUE=_PROPERTY.split('|')
                                            _COLUMN_NAME=str(_NAME_VALUE[0])
                                            _COLUMN_VALUE=str(_NAME_VALUE[1])
                                            DS_ENTITY.update({_COLUMN_NAME:_COLUMN_VALUE})
                                        # Now let's save it to our datastore:
                                        _ENTITIES.append(DS_ENTITY)                                 
                                    except Exception, inst:
                                        print "datastore.Entity(%s) Unexpected error: %s"%(str(_ENTITY_KEY),str(sys.exc_info()[0]))
                                        if isinstance(inst,ValueError):
                                            print "ValueError: %s"%(inst.message)
                                        if isinstance(inst,AttributeError):
                                            print "AttributeError: %s"%(inst.message)
                                except Exception, inst:
                                    print "datastore.Key(_SOURCE_TABLE_NAME,_ENTITY_KEY,_NAMESPACE,_DATASET_ID Unexpected error: %s"%(str(sys.exc_info()[0]))
                                    if isinstance(inst,ValueError):
                                        print "ValueError: %s"%(inst.message)
                                    if isinstance(inst,AttributeError):
                                        print "AttributeError: %s"%(inst.message)

                        for _ENTITY in _ENTITIES:
                            batch.put(_ENTITY)
                            print 'batch.put(_ENTITY)'


except Exception, inst:
    print '\nLoad_Datastore() Unexpected error: ' + str(sys.exc_info()[0])
    if isinstance(inst,ValueError):
        print "ValueError: %s\n"%(inst.message)
    if isinstance(inst,AttributeError):
        print "AttributeError: %s\n"%(inst.message)
    if isinstance(inst,TypeError):
        print "TypeError: %s\n"%(inst.message)
    print '-'*60
    traceback.print_exc(file=sys.stdout)
    print '-'*60                            
    raise

STACK TRACE

Load_Datastore() Unexpected error: <class 'gcloud.exceptions.ServiceUnavailable'>

Traceback (most recent call last):
File "/home/charles/workspace_Python/HBaseSchemaFromXML/pkg1/datastore_loader.py", line 326, in Load_Datastore
print 'batch.put(_ENTITY)'
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/batch.py", line 232, in exit
self.commit()
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/batch.py", line 207, in commit
response = self.connection.commit(self._dataset_id, self.mutation)
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 324, in commit
datastore_pb.CommitResponse)
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 108, in _rpc
data=request_pb.SerializeToString())
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 85, in _request
raise make_exception(headers, content, use_json=False)

ServiceUnavailable: 503 Backend Error

Unexpected error: <class 'gcloud.exceptions.ServiceUnavailable'>
Exception instance type: <class 'gcloud.exceptions.ServiceUnavailable'>
done.

---

Ran 0 tests in 0.000s

NOTES

The script executes all of the batch.put() statements, and then throws the above error after the 'when' clause finishes.

Any suggestions on how to fix this bug are appreciated

[dhermes]

@charlesbclifford Sorry I should have been more specific in #865.

I don't need your entire environment, just was curious if you were on Windows or a *nix system.

I was also curious how you were doing authentication. Are you setting the GOOGLE_APPLICATION_CREDENTIALS environment variable? Are you relying on gcloud login to pick up those credentials?

When you call

storage_conn = storage.get_connection()
storage.set_default_connection(storage_conn)

datastore_conn = datastore.get_connection()
datastore.set_default_connection(datastore_conn)

this is what is getting invoked.

[charlesbclifford]

I have switched to using a service account. I've defined the GOOGLE_APPLICATION_CREDENTIALS env variable in my .bashrc file.

And I have set that env var in the Python script as well
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = '/home/charles/arch-csu-pov-19658eff86e1.json'

I've also used gcloud to set the service account to now be the 'active' credentials

Traceback (most recent call last):
  File "/home/charles/workspace_Python/HBaseSchemaFromXML/pkg1/datastore_loader.py", line 6, in <module>
    from googleapiclient import errors
  File "/usr/lib/python2.7/site-packages/googleapiclient/errors.py", line 26, in <module>
    from oauth2client import util
  File "/home/charles/google-cloud-sdk/lib/oauth2client/util.py", line 38, in <module>
    logger = logging.getLogger(__name__)
AttributeError: 'module' object has no attribute 'getLogger'

[dhermes]

OK it looks like you have a library installed called googleapiclient. That is not our library. Also, the error is now one caused by what is likely a conflicting logging module somewhere on your path (rather than the one that ships with Python).

Can we just focus on a small snipper rather than running your monolithic sample? Just run

from gcloud import datastore

# Make sure proper env. vars. are set for config
q = datastore.Query('Foo')
print list(q.fetch())

and make sure that you have set

export GOOGLE_APPLICATION_CREDENTIALS="/home/charles/arch-csu-pov-19658eff86e1.json"

before running the code.

[charlesbclifford]

The 503 is still returned.

On Friday, May 8, 2015, Danny Hermes notifications@github.com wrote:

OK it looks like you have a library installed called googleapiclient.
That is not our library. Also, the error is now one caused by what is
likely a conflicting logging module somewhere on your path (rather than
the one that ships with Python).

Can we just focus on a small snipper rather than running your monolithic
sample? Just run

from gcloud import datastore

Make sure proper env. vars. are set for config

q = datastore.Query('Foo')print list(q.fetch())

and make sure that you have set

export GOOGLE_APPLICATION_CREDENTIALS="/home/charles/arch-csu-pov-19658eff86e1.json"

before running the code.

—
Reply to this email directly or view it on GitHub
#866 (comment)
.

Charles B. Clifford
(720) 842-7251

[dhermes]

Can you provide a stacktrace? Have you made sure that the JSON credentials are associated with the dataset ID you are using?

[charlesbclifford]

STACKTRACE

Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 421, in iter
self.next_page()
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 388, in next_page
transaction_id=transaction and transaction.id,
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 257, in run_query
datastore_pb.RunQueryResponse)
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 108, in _rpc
data=request_pb.SerializeToString())
File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 85, in _request
raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error

QUESTION
How is a dataset_id associated with the JSON credentials?

TRIVIA
pip list these installed 'Google-like' packages:
gcloud (0.5.0)
gclouddatastore (0.1.2)
gcs-oauth2-boto-plugin (1.9)
google-api-python-client (1.4.0)
google-apitools (0.4.4)
oauth2client (1.4.7)

Are you suggesting that one or more of the above packages need to be un-installed?

[jgeewax]

That's definitely weird. This SO question (http://stackoverflow.com/questions/25855510/google-datastore-returning-503-backend-error) says to try toggling the API on and off for Datastore.

Just to make sure we all are in agreement:

• you're running the code snippet @dhermes sent
• you have created a project inside cloud.google.com/console
• you are able to view something on this page : https://console.developers.google.com/project/_/datastore/query (select your project)
• You have created a service account for that project (https://console.developers.google.com/project/_/apiui/credential)
• You are using the .json keyfile in the environment variable
• The credentials you created are in the same project as the project Id you're providing

Is that all true?

[dhermes]

Also, in my snippet are you setting the environment variable:

export GCLOUD_DATASET_ID="your-project-dataset-id"

---

RE: "Are you suggesting that one or more of the above packages need to be un-installed?" No I'm not suggesting you uninstall anything. You were simply reporting an error from a library outside of the one we are debugging here.

[charlesbclifford]

Yes, we are in agreement on points you raised.

On Friday, May 8, 2015, JJ Geewax notifications@github.com wrote:

That's definitely weird. This SO question (
http://stackoverflow.com/questions/25855510/google-datastore-returning-503-backend-error)
says to try toggling the API on and off for Datastore.

Just to make sure we all are in agreement:

• you're running the code snippet @dhermes
  https://github.com/dhermes sent
• you have created a project inside cloud.google.com/console
• you are able to view something on this page :
  https://console.developers.google.com/project/_/datastore/query
  (select your project)
• You have created a service account for that project (
  https://console.developers.google.com/project/_/apiui/credential)
• You are using the .json keyfile in the environment variable
• The credentials you created are in the same project as the project
  Id you're providing

Is that all true?

—
Reply to this email directly or view it on GitHub
#866 (comment)
.

Charles B. Clifford
(720) 842-7251

[charlesbclifford]

Here is the query, the stack trace and the settings of the env vars:

>>> q=datastore.Query('foo')
>>> print list(q.fetch())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 421, in __iter__
    self.next_page()
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 388, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 257, in run_query
    datastore_pb.RunQueryResponse)
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 108, in _rpc
    data=request_pb.SerializeToString())
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 85, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error
>>> os.environ['GCLOUD_DATASET_ID']
'csu_pov'
>>> os.environ['GOOGLE_APPLICATION_CREDENTIALS']
'/home/charles/arch-csu-pov-19658eff86e1.json'
>>> 

[jgeewax]

Just out of curiosity, can you attach a screenshot of what you see when you go to https://appengine.google.com/datastore/explorer?&app_id=s~csu_pov ? (I'm plugging in your dataset ID as "csu_pov")

[charlesbclifford]

I have dis-abled and re-enabled the Cloud Datastore API.

I've re-run my script and here's the stack trace:

Traceback (most recent call last):
  File "/home/charles/workspace_Python/HBaseSchemaFromXML/pkg1/datastore_loader.py", line 6, in <module>
    from googleapiclient import errors
  File "/usr/lib/python2.7/site-packages/googleapiclient/errors.py", line 26, in <module>
    from oauth2client import util
  File "/home/charles/google-cloud-sdk/lib/oauth2client/util.py", line 38, in <module>
    logger = logging.getLogger(__name__)
AttributeError: 'module' object has no attribute 'getLogger'

And, when I next re-run the simple query, here's the stack trace:

>>> q=datastore.Query('foo')
>>> print list(q.fetch())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 421, in __iter__
    self.next_page()
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 388, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 257, in run_query
    datastore_pb.RunQueryResponse)
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 108, in _rpc
    data=request_pb.SerializeToString())
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 85, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error
>>>

And, when I go to the above link(...&app_id=s~csu_pov?), the following is remitted:

Client Error
The request is invalid for an unspecified reason.

Perhaps I should point out that I am running my script on a laptop at home, and not on a GCE or other GCP infrastructure. And, there is no GAE web app whose Datastore data set the script is trying to write to. The script is attempting to create an entirely new name space, new data set, and new kind, and write a few 100 of those entities using the Datastore API (to Bigtable).

[dhermes]

Let's walk before we can run. It's not worth trying to debug your script when we can't even get a three-line script to run.

The issue almost certainly is relating to auth. When you run it make sure your defaults are set beforehand:

from gcloud import datastore

# Make sure proper env. vars. are set for config
print 'Default Cnxn:', datastore.get_default_connection()
print 'Default ID:', datastore.get_default_dataset_id()
q = datastore.Query('Foo')
print list(q.fetch())

[charlesbclifford]

Good point. Here's the results of the experiment:

>>> print 'Default Cnxn:', datastore.get_default_connection()
Default Cnxn: <gcloud.datastore.connection.Connection object at 0x256c490>
>>> print 'Default ID:', datastore.get_default_dataset_id()
Default ID: csu_pov
>>> q=datastore.Query('foo')
>>> print list(q.fetch())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 421, in __iter__
    self.next_page()
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/query.py", line 388, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 257, in run_query
    datastore_pb.RunQueryResponse)
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 108, in _rpc
    data=request_pb.SerializeToString())
  File "/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py", line 85, in _request
    raise make_exception(headers, content, use_json=False)
gcloud.exceptions.ServiceUnavailable: 503 Backend Error
>>> 

[dhermes]

Can you hop on GitHub and format that? It's real tough to parse from your email client

[charlesbclifford]

Interesting that the datastore default connection object did not copy over. But, trust me it was printed to my console.

[charlesbclifford]

I am using the GitHub web site to respond.

[dhermes]

Yeah the triple chevron (>>>) made it look like email. My bad. I added three backticks to format them and they look great.

It seems like you must have a mismatch of your credentials or that you don't have the datastore API turned on.

Do you see anything in your logs for the csu_pov application?

[jgeewax]

Hi @charlesbclifford : Can you share any information on how you resolved the issue? It might be helpful to others who end up with the same problem...

[charlesbclifford]

I have a document I have written (with key help from you and dhermes) that
builds GCP SDK on an new Fedora 22 installation, successfully. Let me know
where to post it.

On Sunday, May 10, 2015, JJ Geewax notifications@github.com wrote:

Hi @charlesbclifford https://github.com/charlesbclifford : Can you
share any information on how you resolved the issue? It might be helpful to
others who end up with the same problem...

—
Reply to this email directly or view it on GitHub
#866 (comment)
.

Charles B. Clifford
(720) 842-7251

[jgeewax]

feel free to share the doc here and we'll send it along :)

[charlesbclifford]

I will, just as soon as I figure out how to build out my Linux laptop so
that it can run Python scripts which interact with GCS and with Google
BigQuery.

I now have a script which reads from GCS buckets and parses and loads 1000s
of CSV files into Google Cloud Datastore. The parser is governed by a
metadata rules engine/method that re-creates the source schema within
Datastore.

I still have to figure out, by 9am tomorrow morning, how to read 1000s of
those same CSVs and load them into BigQuery, and have the BigQuery schema
likewise conform to the source schema.

Another fun weekend.

On Sun, May 10, 2015 at 11:24 AM, JJ Geewax notifications@github.com
wrote:

feel free to share the doc here and we'll send it along :)

—
Reply to this email directly or view it on GitHub
#866 (comment)
.

Charles B. Clifford
(720) 842-7251