Convert gRPC APIs to use GAPIC

[stephenplusplus]

• Bigtable
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works
• Datastore
  • Generated files are in repo
  • Handwritten API is using generated API (will be complete when https://github.com/googleapis/nodejs-datastore is live)
  • Passing through gaxOptions is available on all requests (will be complete when https://github.com/googleapis/nodejs-datastore is live)
  • Automatic projectId insertion works
• Language
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works
• Logging
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works
• Pub/Sub
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works
• Spanner
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works
• Speech
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works (Not applicable)
• Vision
  • Generated files are in repo
  • Handwritten API is using generated API
  • Passing through gaxOptions is available on all requests
  • Automatic projectId insertion works (Not applicable)

Cursoring Options

options.autoPaginate

Since the generated API accepts this option, I don't believe we'll need common/paginator. We will simply decorate whatever our callback to gax executes with.

options.maxApiCalls

All API calls come with a cost, and autoPaginate: true leaves that out of the user's control. We added this option to give that control back. Is this supported in the generated layer?

options.maxResults

We support this currently, though I don't know if this is supported in the generated layer. It will simply cut off making further requests after this amount has been reached. So a user can effectively say "autopaginate up until 58 results".

Common Methods

Since each resource will still have some or all of a shared set of methods:

• delete
• getMetadata
• setMetadata

and we can still wrap them with helpers like:

• create
• exists
• get

I believe it will require only a small bit of re-factoring to support, but GrpcService & GrpcServiceObject will likely continue to have a use. Alternatively, it might be possible to use the generic Service and ServiceObject if we can find an acceptable amount of convention/configuration.

common/paginator & common/streamify

Will we need these?

common/util.promisify

Will we need this?

Others

Is there anything else I could be missing?

I came up with this list by trying to migrate the Logging handwritten API to use the generated one. I quickly ran into these bumps, and figured we should resolve these and others up front. Ideally, our repo will remain clean of any artifacts of past implementations, so this will involve a thorough review of the utilities we have in common. @callmehiphop feel free to try a similar migration (pick an API) and see if you uncover any other issues.

// cc @callmehiphop @jmuk @bjwatson

[stephenplusplus]

/bump after the big week last week :) Some things are @callmehiphop questions, others are @jmuk questions. Please let me know if you need any clarity.

[callmehiphop]

AFAIK we will not need any of the common stuff you mentioned if a library uses GAX, but code changes will be required to return whatever the GAPIC layer returns to the user from the handwritten.

[stephenplusplus]

@jmuk is it possible to generate the Bigtable and Datastore GAX layer?

[jmuk]

Datastore will be ready soon. The bigtable API -- it seems we don't have its admin API support yet. You need both normal bigtable and its admin APIs, is that right?

Also cc @lukesneeringer

[stephenplusplus]

Correct, both are needed. Thanks for getting Datastore ready!

[stephenplusplus]

This issue didn't have enough labels, so I marked it as blocked to reflect the changes we need from google-gax that were uncovered in the Logging conversion PR.

[bjwatson]

I propose we remove the API labels for the APIs that have already been implemented fully. We don't need this issue coming up in Vision, etc. queries, since it's already done for them.

[stephenplusplus]

I'm actually fine with it (in this case!) because it would help us map back to this change (and its goals and discussion), which is helpful in the case we someday find a bug and can track it to this event.

[bjwatson]

SGTM

[stephenplusplus]

I've updated this issue with the task of supporting automatic project ID insertion. This functionality was inadvertently lost when we switched to GAX, and each API that was considered complete will need re-working to support it.

[bjwatson]

@lukesneeringer @swcloud Friendly reminder to prioritize GAPIC porting work.

[bjwatson]

@swcloud Ping. I'd like to keep making progress in Q2 on porting manual layers onto GAPIC code, as long as it doesn't interfere with our API deliverables. Stephen and Dave need priorities for that.

[stephenplusplus]

To do a re-cap on what this issue is about, each API in the opening post has 4 tasks (defined below):

Generated files are in repo
Is the gapic layer in the repo?

Handwritten API is using generated API
Once the gapic layer made it into the repo, are we using it from the handwritten layer?

Passing through gaxOptions is available on all requests
GAX accepts a group of options that can be things like "autoPaginate" or very fine-grained request settings (deadlines, retry configuration, etc). Are we allowing these settings to be controlled from the user?

Automatic projectId insertion works -- ⚠️BLOCKED⚠️
This is what we're stuck on. The goal is to support automatic project ID detection and insertion into any required request parameters, i.e., creating a resource, like "createTopic({ resource: '{resourceId}' })", where resourceId is a full path "projects/{project}/topics/{topicName}". GAX already has support for detecting {project} when possible (ADC, GCP environments), but what it lacks is a way to automatically insert it into request parameters like these.

The issue for this subject is here: googleapis/gax-nodejs#134

[bjwatson]

Thanks for explaining the current context for this issue @stephenplusplus.

IIUC, the automatic projectId insertion is bound up with this issue raised by @ofrobots: #1891. Correct?

FYI @lukesneeringer

[ofrobots]

This is what we're stuck on. The goal is to support automatic project ID detection and insertion into any required request parameters, i.e., creating a resource, like "createTopic({ resource: '{resourceId}' })", where resourceId is a full path "projects/{project}/topics/{topicName}"

I am worried about automatic text insertion as it could do data corruption if the user data happens to contain strings that happen to match the pattern.

Could we use promises to deal with async initialization, something like:

function promiseOfTopic(topicName) {
  return promiseOfProjectId
      .then(projectId => `projects/${projectId}/topics/${topicName}`);
}

[stephenplusplus]

Yes, we can do that, however two things to consider:

• A solution without any changes to the string replacement pattern was already discussed in Automatic project ID insertion gax-nodejs#134, without any additional voices of concern raised. I think randomizing the string further ({{projectId}}${uuid}, or something similar) would make the GCN codebase simpler, and push more menial tasks to the generated layer.
• A bug in GAX is blocking us either way: we can't instantiate a GAX client without projectId, else it throws multiple errors. (avoid unhandled promise rejection on wrongly initialized stub gax-nodejs#121).

[ofrobots]

I am still fairly strongly opposed to string replacement patterns - even with more randomization. We ought to be able to guarantee that we will not clobber user data under any conditions.

[stephenplusplus]

Two APIs remain, Bigtable and Datastore.

• Bigtable is in its own repo, but does not have a GAPIC API.
• Datastore is in its own repo, but it is not live. After it goes public and gets a release, the conversion to GAPIC is complete.

[stephenplusplus]

Datastore has made the transition, leaving only Bigtable. Issue opened on the Bigtable tracker: googleapis/nodejs-bigtable#23