Split out APIs into separately-installable distributions?

[tseaver]

• gcloud on PyPI would include "core" modules.
• gcloud would (maybe) become a namespace package
• Each API would have a separate PyPI "project"/releases

Cons (mostly for maintainers):

• separate release cycles
• More complicated CI
• Might need to split repos?

Pros (mostly for users):

• "Hard" dependencies for each API (gRPC, I'm looking at you)
• Separately updateable distributions
• "Pay only for what you eat"

[dhermes]

Comments on cons:

• separate release cycles: I think this is a good thing, but a tiny bit problematic if updates to core break the service-specific packages
• More complicated CI: Yup, we'd need to kick off a run of Travis every time a downstream dependency is updated
• Might need to split repos?: I see this is a positive (outside of the CI stuff)

Another potential con:

• Building unified docs becomes harder

[jgeewax]

The one I'm concerned about is unified docs -- that's something that could actually be a show-stopper we'd have to sort out first...

[dhermes]

Yeah this could be a lot of work. The flipside though is that the library is becoming too big and its a dis-service to developers to make them buy the whole house when they only want the kitchen (or a single lightswitch).

[jgeewax]

its a dis-service to developers to make them buy the whole house when they only want the kitchen

This is actually tricky. The majority of people in UX studies want the whole house. The more advanced users eventually run into a problem and decide they only want the kitchen -- but only after trying out lots of things and using gcloud for a while...

So, we have a bit of an issue: do we optimize for the advanced folks? If we do, we might alienate and frustrate users, effectively reducing the number of people entering the funnel who could eventually become those advanced folks...

My thought here is still that we need to make both possible, but make the common case the simplest: pip install gcloud gives me the house. If I want just the kitchen, or just the light switch, this should be possible but not necessarily something we optimize for. Does that make sense?

[dhermes]

Yeah that'd have to be supported by also having a gcloud-core PyPI package and then using each of the gcloud-SERVICENAME packages.

[dhermes]

So we'd need (as of right now) the following PyPI packages:

• gcloud
• gcloud-core
• gcloud-bigquery
• gcloud-bigtable
• gcloud-datastore
• gcloud-dns
• gcloud-pubsub
• gcloud-resource-manager
• gcloud-search
• gcloud-storage

[jgeewax]

I have no objection to this layout. Sounds to me like...

• gcloud depends on gcloud-* ("The whole house")
• gcloud-* (except gcloud-core) all depend on gcloud-core ("The kitchen and the foundation")

This means that if I install any of the individual services, I get the service I want, and the core library. If I install gcloud I just get everything.

The trouble (just a wild guess here) will be in the namespace issues... That is...

$ pip install gcloud-pubsub
$ python
>>> from gcloud import pubsub  # Can we make this work?
$ pip install gcloud
$ python
>>> from gcloud import pubsub  # This should (still) work.

[tseaver]

Yes, that pattern will work. We will need to:

• Move the code which is now directly in gcloud/*.py down into a gcloud/core/*.py, updating all the affected imports.
• Declare gcloud as a namespace package in all the distributions (in their calls to setup() in setup.py),
• Make gcloud/__init__.py contain only the following::

__import__('pkg_resources').declare_namespace(__name__)

• If we don't split the repository, then we will need to update setup.py to call setup() once for gcloud-core, once for each API, and once for the omnibus metapackage.

[tbetbetbe]

👍 modularization

FYI googleapis folks: @Anthm, @vsubramani, @rok987