-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding helpers to parse Bigtable create cluster operation. #1264
Conversation
@tseaver Sorry this is kind of a doozy. I wrote a long description to try to give you some context and quick windows into what the actual data returned from the API is. This PR is doing something very simple: parse the information from the |
seconds=timestamp.seconds, | ||
microseconds=(timestamp.nanos / 1000.0), | ||
) | ||
) |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
A CreateCluster request response doesn't actual indicate success or failure. Rather it returns a cluster object with the validated request parts inside and a `current_operation` attached. We implement `_process_operation` so that we can determine the ID of that long-running operation (so it can be checked for completion / success, if desired by the user). In addition we seek to notify the user when the request began. From the [service definition][1] we know that the `current_operation` is a [long-running operation][2] and that: > The embedded operation's "metadata" field type is `CreateClusterMetadata`, > The embedded operation's "response" field type is `Cluster`, if successful. The [`Operation` metadata][3] is of type [`Any`][4] (which uses a `type_url` and raw bytes to provide **any** protobuf message type in a single field, but still allow it to be parsed into it's true type after the fact). So we expect `CreateCluster` responses to have long-running operations with a type URL matching [`CreateClusterMetadata`][5]. As a result, we introduce a utility (`_parse_pb_any_to_native`) for parsing an `Any` field into the native protobuf type specified by the type URL. Since we know we need to handle `CreateClusterMetadata` values, we add a default mapping (`_TYPE_URL_MAP`) from the corresponding type url for that message type to the native Python type. The `CreateClusterMetadata` type has `request_time` and `finish_time` fields of type [`Timestamp`][6] so we also add the `_pb_timestamp_to_datetime` helper for converting protobuf messages into native Python `datetime.datetime` objects. [1]: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/8e363d72eb39d921dfdf5daf4a36032aa9d003e2/bigtable-protos/src/main/proto/google/bigtable/admin/cluster/v1/bigtable_cluster_service.proto#L64 [2]: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/8e363d72eb39d921dfdf5daf4a36032aa9d003e2/bigtable-protos/src/main/proto/google/bigtable/admin/cluster/v1/bigtable_cluster_data.proto#L74 [3]: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/8e363d72eb39d921dfdf5daf4a36032aa9d003e2/bigtable-protos/src/main/proto/google/longrunning/operations.proto#L82 [4]: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/8e363d72eb39d921dfdf5daf4a36032aa9d003e2/bigtable-protos/src/main/proto/google/protobuf/any.proto#L58 [5]: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/8e363d72eb39d921dfdf5daf4a36032aa9d003e2/bigtable-protos/src/main/proto/google/bigtable/admin/cluster/v1/bigtable_cluster_service_messages.proto#L83-L92 [6]: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/8e363d72eb39d921dfdf5daf4a36032aa9d003e2/bigtable-protos/src/main/proto/google/protobuf/timestamp.proto#L78
9bcb777
to
388bf37
Compare
LGTM |
Adding helpers to parse Bigtable create cluster operation.
A
CreateCluster
request response doesn't actually indicatesuccess or failure. Rather it returns a
Cluster
message withthe validated request parts inside and a
current_operation
attached.
We implement
_process_operation
so that we can determine theID of that long-running operation (so it can be checked for
completion / success, if desired by the user). In addition
we seek to notify the user when the request began.
From the service definition we know that the
current_operation
is a long-running operation and that:
The
Operation
metadata is of typeAny
(which uses atype_url
and raw bytes to provide any protobuf message type in a single
field, but still allow it to be parsed into it's true type after
the fact). So we expect
CreateCluster
responses to have long-runningoperations with a type URL matching
CreateClusterMetadata
.As a result, we introduce a utility (
_parse_pb_any_to_native
) forparsing an
Any
field into the native protobuf type specified by thetype URL. Since we know we need to handle
CreateClusterMetadata
values,we add a default mapping (
_TYPE_URL_MAP
) from the corresponding type urlfor that message type to the native Python type.
The
CreateClusterMetadata
type hasrequest_time
andfinish_time
fields of typeTimestamp
so we alsoadd the
_pb_timestamp_to_datetime
helper for converting protobufmessages into native Python
datetime.datetime
objects.