Importing apitools.base.py takes half a second

[dhermes]

This makes using the library a very jarring experience for application start-up times.

[dhermes]

As mentioned in googleapis/google-cloud-python#755 I did some timing and here is what I found (with a fresh environment for each import)

==================
base_api: 0.0385411
---- Imports: ----
    # from apitools.base.py import credentials_lib
    from apitools.base.py import encoding
    from apitools.base.py import exceptions
    from apitools.base.py import http_wrapper
    from apitools.base.py import util
==================
batch: 0.05901
---- Imports: ----
    from apitools.base.py import exceptions
    from apitools.base.py import http_wrapper
==================
credentials_lib: 0.515645
---- Imports: ----
    from apitools.base.py import exceptions
    from apitools.base.py import util
==================
encoding: 0.029279
---- Imports: ----
    from apitools.base.py import exceptions
==================
exceptions: 0.00117803
---- Imports: ----
==================
extra_types: 0.032212
---- Imports: ----
    from apitools.base.py import encoding
    from apitools.base.py import exceptions
    from apitools.base.py import util
==================
http_wrapper: 0.0387058
---- Imports: ----
    from apitools.base.py import exceptions
    from apitools.base.py import util
==================
list_pager: 0.00224304
---- Imports: ----
==================
transfer: 0.0412002
---- Imports: ----
    from apitools.base.py import buffered_stream
    from apitools.base.py import exceptions
    from apitools.base.py import http_wrapper
    from apitools.base.py import stream_slice
    from apitools.base.py import util
==================
util: 0.028183
---- Imports: ----
    from apitools.base.py import encoding
    from apitools.base.py import exceptions
==================

[dhermes]

From the above, it seems like exceptions is the culprit, but list_pager also takes very long without ever importing any other apitools modules.

They were all similarly timed because every import by necessity picks up apitools.base.py.__init__. So I turned off all the imports there to get the running times.

After focusing in on base_api and credentials_lib, I commented out credentials_lib before in base_api. This makes it clear the issue is fully in credentials_lib.

Now to analyze the code.

[dhermes]

After analysis, it comes from oauth2client in credentials_lib. My guess is that it is caused by the GCE check, but I'll need to verify.

Initial hunch was wrong. I narrowed it down to from oauth2client import crypt. The import that takes the long time is OpenSSL:

$ time python -c 'from OpenSSL import crypto'

real    0m0.485s
user    0m0.459s
sys     0m0.024s

[dhermes]

Relevant issue: pyca/pyopenssl#137

---

We could use something like pkgutil in oauth2client:

python -c "import pkgutil; print pkgutil.get_loader('OpenSSL').filename"

This way the things that depend on OpenSSL.crypto can be defined but the import can be put off until the first use.

[craigcitro]

yucks. thanks for tracking this down.

i think the sanest move is to just wait for this to get fixed in oauth2client. (or depend on an old version of PyOpenSSL.)

on the flipside, PyOpenSSL isn't a required dependency -- installing without it works fine, until you try to use the old p12 file for a service account.

[dhermes]

Yeah I was able to address it with a 3 line change to oauth2client (referenced just above your comment, probably at the same moment you were typing).

[dhermes]

OK so at this point this is just release blocked on being fixed

[craigcitro]

done!

or do you also need an oauth2client release?

[dhermes]

It isn't urgent. 0.5 seconds sucks, but it's just a startup cost.

However, closing this out shouldn't happen until oauth2client>1.4.7 (idea for a GitHub add-on, a PyPI trigger)

[craigcitro]

should be good now, thanks to your oauth2client cleanups.

[dhermes]

Yup. And the hack will be able to be removed once cryptography==1.0 comes out (source).