pubsub client relies on psutil which fails on appengine standard

[ericbuehl]

Using:

Version: 0.28.2
Summary: Python Client for Google Cloud Pub/Sub
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: /<proj>/.env/lib/python2.7/site-packages
Requires: googleapis-common-protos, grpc-google-iam-v1, grpcio, google-gax, psutil, google-cloud-core

I get the following stack trace when deployed to an app engine standard module:

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/base/data/home/runtimes/python27_experiment/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/base/data/home/apps/s~<proj>/admin:20170828t162057.403716850892108987/main.py", line 10, in <module>
    from google.cloud import pubsub
  File "/base/data/home/apps/s~<proj>/admin:20170828t162057.403716850892108987/lib/google/cloud/pubsub.py", line 17, in <module>
    from google.cloud.pubsub_v1 import PublisherClient
  File "/base/data/home/apps/s~<proj>/admin:20170828t162057.403716850892108987/lib/google/cloud/pubsub_v1/__init__.py", line 17, in <module>
    from google.cloud.pubsub_v1 import types
  File "/base/data/home/apps/s~<proj>/admin:20170828t162057.403716850892108987/lib/google/cloud/pubsub_v1/types.py", line 20, in <module>
    import psutil
  File "/base/data/home/apps/s~<proj>/admin:20170828t162057.403716850892108987/lib/psutil/__init__.py", line 80, in <module>
    from . import _pslinux as _psplatform
  File "/base/data/home/apps/s~<proj>/admin:20170828t162057.403716850892108987/lib/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: cannot import name _psutil_linux

[theacodes]

The short answer is that due to limitations in the App Engine standard environment, this library is not supported there.

The long answer(s) are over at #1893. I'm going to go ahead and close this as a duplicate, but by all means please continue to comment- the feedback is useful in communicating issues with the App Engine team. :)

[ericbuehl]

For what it is worth, old versions did not have this problem (eg pubsub-v1==0.15.4)

[theacodes]

Yep, that's true for a few libraries here. Mostly that was just accidental and there's a lot of edge cases where those would fail. We simply can not offer the same level of support in standard as we do elsewhere at this point in time, but we're working really hard to do so.

In the meantime I'd recommend using the API client as mentioned in #1893 or pinning to an older version.

[Chez]

I have recently run into an import error with psutils in the types.py called from google.cloud.pubsub_v1 on a GAE standard project inside Pycharm. I messed around for a day trying to find away around it, but no go. psutil imports in python run from the terminal and the console and from a vanilla python project but not from a GAE project. To double up on this I made a basic test of the two as seen below:

Standard Vanilla Python Project:

main.py contains

import psutil

def x():
    print "hello world"

if __name__ == "__main__":
    x()

Google App Engine Project -
main.py contains

import webapp2
import psutil

class MainHandler(webapp2.RequestHandler):
    def get(self):
        self.response.write('Hello world!')

app = webapp2.WSGIApplication([
    ('/', MainHandler)
], debug=True)

The import in the vanilla python project works fine, but the import in the GAE app returns the traceback:

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/Users/chez 1/google-app-engine-projects/untitled/main.py", line 18, in <module>
    import psutil
ImportError: No module named psutil

[tseaver]

pubsub uses psutil only for computing a maximum amount of RAM to be used: ISTM we could fall back to some GAE-standard-appropriate default if import psutil raises.

[Chez]

It does raise on a mac El Capitan / Pycharm in GAE Standard env, even when installed as a third party to a libs directory. So now we have to write a cloud function to access the pubsub, not ideal...

Traceback (most recent call last):

  File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "/usr/local/google_appengine/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "/Users/chez 1/google-app-engine-projects/pitpit-gae/main.py", line 56, in <module>
    import supplier.api.v1.routes as supplierroutes
  File "/Users/chez 1/google-app-engine-projects/pitpit-gae/supplier/api/v1/routes.py", line 8, in <module>
    from supplier.api.v1.handlers import *
  File "/Users/chez 1/google-app-engine-projects/pitpit-gae/supplier/api/v1/handlers.py", line 9, in <module>
    import psutil
  File "/Users/chez 1/google-app-engine-projects/pitpit-gae/external/psutil/__init__.py", line 149, in <module>
    from . import _psosx as _psplatform
  File "/Users/chez 1/google-app-engine-projects/pitpit-gae/external/psutil/_psosx.py", line 16, in <module>
    from . import _psutil_osx as cext
  File "/usr/local/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 965, in load_module
    raise ImportError('No module named %s' % fullname)

[davidbernat]

What was the solution to this? I am having the same issue with _psutil_osx.

[dhermes]

@davidbernat I would recommend putting a custom psutil in your lib/ that just returns a mocked result.

[davidbernat]

@dhermes Thank you for your suggestion. Can you point me to a reference which will walk me through which methods to mock in psutil? Being unfamiliar with the library I do not know to what extent is safe to mock without eliminating other useful or essential features of the library. Thank you again.

[dhermes]

I would just grep for usage of psutil in your application. It's a third-party app (i.e. from PyPI) so you only need to check your application code.