Start project to publish pb2 generated modules for services we depend on.

[dhermes]

See #1317 for context

It's also possible that the owners of googleapis-common-protobufs will publish this

/cc @tbetbetbe

[tbetbetbe]

We're going to start doing this from next week. The capability is already there now, but there are breaking changes in the gRPC client surface in v0.12, so we'll start next week once gRPC 0.12 packages are published

[dhermes]

We'd like to rely on them with gRPC as an optional dependency, e.g. for datastore.

Is this a possibility or at least putting the proto message definitions in separate modules than the gRPC service definitions?

[dhermes]

Also is there some public code or issue tracker or repo we can follow along with?

[dhermes]

@tbetbetbe Any updates here?

[tbetbetbe]

FYI: @nathanielmanistaatgoogle, there is a grpc issue tracking part of this grpc/grpc#3866, will you open another to track the actual split ?

[dhermes]

Thanks for the update!

[tseaver]

Logging and Pubsub already have releases of GAX and grpc wrappers.

[dhermes]

and grpc wrappers

Are these different than GAX? Do similar ones exist for Datastore and Bigtable?

[tseaver]

The grpc-google-pubsub-v1 and grpc-google-logging-v2 wrappers are generated straight from the .proto files with the grpc plugin enabled. The GAX wrappers (gax-google-pubsub-v1 and gax-google-logging-v2) depend on them, and provide a somewhat smoother surface.

[bjwatson]

What's the delta between what this issue requests and what https://github.com/googleapis/googleapis provides?

[dhermes]

@bjwatson I updated the title to make it more explicit.

We want pip-installable packages for each service.

[daspecster]

I could have just missed this, but I can't find repos for the generated versions of...
gax-google-cloud-vision
grpc-google-cloud-vision
...etc.

It would be nice to have README's for them but that might be asking a bit much for generated code. It would be nice just for code browsing and such though.

[bjwatson]

Thanks @dhermes. Makes sense. What else is needed beyond what googleapis-common-protos already provides? Could the delta just be added to that package to resolve this issue?

[bjwatson]

@daspecster Are you asking for:

https://pypi.python.org/pypi/gax-google-cloud-vision-v1
https://pypi.python.org/pypi/grpc-google-cloud-vision-v1

...or did you want something else?

They both have READMEs in them, although I'm open to improvements you think they need.

[dhermes]

@bjwatson We just want it for everything we depend on, like Bigtable and Datastore. i.e.

https://pypi.python.org/pypi/gax-google-{FOO}-v{BAR}
https://pypi.python.org/pypi/grpc-google-{FOO}-v{BAR}

[bjwatson]

So basically an index of where you can find the packages for all our auto-gen code?

[dhermes]

That would also be great. But this was originally filed because we don't have all the packages we depend on, so we were going to publish those packages ourselves if it came down to it.

Are you saying Datastore v1 and Bigtable v2 actually exist somewhere? The "expected" links are dead:

https://pypi.python.org/pypi/grpc-google-datastore-v1
https://pypi.python.org/pypi/gax-google-datastore-v1
https://pypi.python.org/pypi/grpc-google-bigtable-v2
https://pypi.python.org/pypi/gax-google-bigtable-v2

though the previous version of datastore does have a working link

https://pypi.python.org/pypi/grpc-google-datastore-v1beta3

(but not a working GAX link)

[bjwatson]

No, we have never generated BigTable, and apparently only did a rather old build of gRPC for datastore. An index might help, since it communicates what we have done and what we are maintaining.

[dhermes]

Sounds good to me. We have Makefile.datastore and Makefile.bigtable_v2 and a bunch of generated code and custom scripts for pulling out gRPC. We also have the pain of rewriting imports.

We'd love to delete 100% of that code and just add grpc-google-datastore-v1 and grpc-google-bigtable-v2 to our setup.py dependencies.

PS BigQuery uses a capital in the middle but Bigtable does not. Thanks Obama!

[daspecster]

@bjwatson, yeah I but having the generated code available on github is what I was thinking.

[dhermes]

@daspecster Not sure why that is necessary. It's not meant to be read by a human so checking it into a repo is somewhat silly. pip install and be on your way. The protos themselves should be the source of documentation. We can chat offline for more details?

[bjwatson]

@dhermes @daspecster LMK if you have reasons for source repo(s) on GH. I've thought about it myself, but I don't have strong enough reasons for why it's worth the maintenance overhead for ourselves.

I'll do the Datastore and Bigtable (thanks for spelling update) packages later today or early next week, depending on how things go with our remote build server.

[bjwatson]

I've created this index in #2144. PTAL

It includes the APIs we plan to auto-generated in the next few weeks. PLMK if there's anything missing that you expected.

[daspecster]

At some level they do end up getting used by humans, so that's why I thought it would be good to have a bit more documentation than there currently is.

But I agree it may not be worth the maintenance.

[bjwatson]

The index I just created includes links to docs on http://pythonhosted.org/. We currently only have Pub/Sub and Logging there, but plan to automate doc generation so we can easily do it for all APIs.

[daspecster]

The docs on pythonhosed.org look nice! I could use the Vision ones any time 👍.
The docstrings and README for gax-google-cloud-vision could use some deeper examples. Like actually building a request. Although this could probably go in grpc-google-cloud-vision. Either place really I suppose but it would be nice to have it all in one spot.

[bjwatson]

Thanks! Sphinx does a lot of the heavy lifting to help the readability. @geigerj can you push docs to pythonhosted for Vision as a one-off?

@daspecster Do you think there's something that could be done in auto-gen to improve the documentation, or is your feedback more for the Vision team to improve the documentation in their protos?

[geigerj]

Vision docs pushed to here: https://pythonhosted.org/gax-google-cloud-vision-v1

Note that right now, the docs require some manual effort per-upload because the code generator does not always format the docstrings in a way that Sphinx understands; we're tracking those issues (at googleapis/gapic-generator#401, googleapis/gapic-generator#402) so that we can automate doc generation in the future.

[daspecster]

@bjwatson @geigerj I think the pythonhosted.org stuff will work well enough for me if a full example was in them.

Either here http://pythonhosted.org/gax-google-cloud-vision-v1/google.cloud.vision.v1.image_annotator_api.html or here http://pythonhosted.org/gax-google-cloud-vision-v1/starting.html#examples

[bjwatson]

@daspecster I created an issue to improve our auto-gen sample for that method: https://github.com/googleapis/googleapis/issues/99

For your purpose, however, it would probably be faster if you looked at existing samples like https://github.com/GoogleCloudPlatform/cloud-vision/blob/master/python/face_detection/faces.py, and adapt this to the GAX surface. There's not much we were able to do in the GAX surface, due to the structure of this API (single method with very rich argument structure), so it should be a straightforward mapping. Please let me know if you have any questions.

[bjwatson]

What's left to be done to close this issue? Filling in all the TBDs in https://github.com/googleapis/googleapis/blob/master/PACKAGES.md with package links? Anything else?

[dhermes]

It's fine to close since there is a dedicated team for this very purpose.

Where do we file issues for new services? (e.g. #2128 is still something we want / need)

[bjwatson]

You can file such issues here and assign them to me. This seems more appropriate than any of the googleapis repos.

[dhermes]

Great thanks!