Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add samples for Cloud Tasks #1068

Merged
merged 7 commits into from
Sep 14, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 31 additions & 33 deletions appengine/flexible/tasks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ On Compute Engine and Container Engine, authentication credentials will be
automatically detected, but the instances must have been created with the
necessary scopes.

In any other environment, for example Compute Engine instance without the
In any other environment, for example a Compute Engine instance without the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above

necessary scopes, you should set `GOOGLE_APPLICATION_CREDENTIALS` environment
variable to a JSON key file for a service account.

Expand All @@ -42,17 +42,18 @@ for more information.

## Creating a queue

To create a queue using the Cloud SDK, use the provided queue.yaml:
To create a queue using the Cloud SDK, use the following gcloud command:

gcloud app deploy queue.yaml
gcloud alpha tasks queues create-app-engine-queue "my-appengine-queue"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to put the queue ID in quotes

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


## Deploying the App Engine app

First, vendor the dependencies into the project:
Note: A newly created queue will route to the default App Engine service and
version unless configured to do otherwise. Read the online help for the
`create-app-engine-queue` or the `update-app-engine-queue` commands to learn
about routing overrides for App Engine queues.

pip install -r requirements.txt
## Deploying the App Engine app

Next, deploy the App Engine app
Deploy the App Engine app with gcloud:

gcloud app deploy

Expand All @@ -61,47 +62,44 @@ Verify the index page is serving:
gcloud app browse

The App Engine app serves as a target for the push requests. It has an
endpoint `/set_payload` that that stores the payload from the HTTP POST data in
Cloud Datastore. The payload can be accessed in your browser at the
`/get_payload` endpoint with a GET request.
endpoint `/log_payload` that reads the payload (i.e., the request body) of the
HTTP POST request and logs it. The log output can be viewed with:

gcloud app logs read

## Running the Samples

The project ID must be specified either as a command line argument using
`--project-id`, or by editing `DEFAULT_PROJECT_ID` within `task_snippets.py`.

Set the environment variables:
Set environment variables:

First, your project ID:

export PROJECT_ID=my-project-id
export LOCATION_ID=us-central1
export QUEUE_ID=my-appengine-queue # From queue.yaml

View all queues:
Then the queue ID, as specified at queue creation time. Queue IDs already
created can be listed with `gcloud alpha tasks queue list`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the command is queues plural

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


python app_engine_queue_snippets.py --api_key=$API_KEY list-queues --project_id=$PROJECT_ID --location_id=$LOCATION_ID
export QUEUE_ID=my-appengine-queue

Set the queue name as an environment variable:
And finally the location ID, which can be discovered with
`gcloud alpha tasks queue describe $QUEUE_ID`, with the location embedded in the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

queues plural

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

"name" value (for instance, if the name is
"projects/my-project/locations/us-central1/queues/my-appengine-queue", then the
location is "us-central1").

export QUEUE_NAME=projects/$PROJECT_ID/locations/$LOCATION_ID/queues/$QUEUE_ID
export LOCATION_ID=us-central1

Create a task, targeted at the `set_payload` endpoint with a payload specified:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be log_payload?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


python app_engine_queue_snippets.py --api_key=$API_KEY create-task --queue_name=$QUEUE_NAME --payload=hello

Now view that the payload was received and verify the count and payload:

http://your-app-id.appspot.com/get_payload

Create a task that will be scheduled for a few seconds in the future using
the `--in_seconds` flag:

python app_engine_queue_snippets.py --api_key=$API_KEY create-task --queue_name=$QUEUE_NAME --payload=hello --in_seconds=30
python create_app_engine_queue_task.py --project=$PROJECT_ID --queue=QUEUE_ID --location=LOCATION_ID --payload=hello
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ before variable names

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why the queue_name parameter was removed in favor of breaking it out into three components? While I think the naming conventions of "queue name" vs. "queue" vs. "queue ID" are hard to follow, they're apparently in line with OnePlatform. Further, when we want to add more params like headers, App Engine routing, relative URL, and response view, the command will get pretty long.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed queue_name because gcloud queues list doesn't show the queue name. There's no real way to discover it without hitting the API separately. The queue name construct is still visible in the code itself, which seems appropriate for a sample.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the describe command does, which is what you're using to get the location anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's right. I think using the "queue_name" is an awkward abstraction, but since "describe" shows the queue_name I could add it back in. I'll reconsider.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discussed with Jon and he told me that our other samples that have resource paths under the hoods use a similar "pass in the components on the command line, and assemble the full path in the code itself" system. So, I think I'll keep it like this for now. I think it's awkward to have the user use "describe" to look at the resource path just to get the location out of it, but I've been assured that the resource path will be included in "list" output soon at which point that will no longer be necessary.


Since `--in_seconds` was set to 30, it will take 30 seconds for the new
payload to be pushed to the `/get_payload` endpoint, which can then be viewed at:
Now view that the payload was received and verify the payload:

http://your-app-id.appspot.com/get_payload
gcloud app logs read

It might also be helpful to view the request logs of your App Engine app:
Create a task that will be scheduled for a time in the future using the
`--in_seconds` flag:

https://console.cloud.google.com/logs
python create_app_engine_queue_task.py --project=$PROJECT_ID --queue=QUEUE_ID --location=LOCATION_ID --payload=hello --in_seconds=30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$ before variable names

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

147 changes: 0 additions & 147 deletions appengine/flexible/tasks/app_engine_queue_snippets.py

This file was deleted.

112 changes: 112 additions & 0 deletions appengine/flexible/tasks/create_app_engine_queue_task.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# Copyright 2017 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from __future__ import print_function

import argparse
import base64
import datetime
import pprint

from googleapiclient import discovery


def seconds_from_now_to_rfc3339_datetime(seconds):
"""Return an RFC 3339 datetime string for a number of seconds from now."""
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=seconds)
return d.isoformat('T') + 'Z'


def create_task(project, queue, location, payload=None, in_seconds=None):
"""Create a task for a given queue with an arbitrary payload."""
client = get_client()

url = '/log_payload'
body = {
'task': {
'app_engine_task_target': {
'http_method': 'POST',
'relative_url': url
}
}
}

if payload is not None:
# Payload is a string (unicode), so
body['task']['app_engine_task_target']['payload'] = base64.b64encode(
payload.encode()).decode()

if in_seconds is not None:
scheduled_time = seconds_from_now_to_rfc3339_datetime(in_seconds)
body['task']['schedule_time'] = scheduled_time

queue_name = 'projects/{}/locations/{}/queues/{}'.format(
project, location, queue)

print('Sending task {}'.format(pprint.pformat(body)))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might it be even better to print this as JSON?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


response = client.projects().locations().queues().tasks().create(
parent=queue_name, body=body).execute()

# By default CreateTaskRequest.responseView is BASIC, so not all
# information is retrieved by default because some data, such as payloads,
# might be desirable to return only when needed because of its large size
# or because of the sensitivity of data that it contains.
print('Created task {}'.format(response['name']))
return response


def get_client():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer that you just put this inline in the function above.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

"""Build an http client."""
DISCOVERY_URL = 'https://cloudtasks.googleapis.com/$discovery/rest?version=v2beta2'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not a constant, so it doesn't need to be capital.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a constant? It's not modified or replaced anywhere, it's just a string literal with a name, right?

client = discovery.build('cloudtasks', 'v2beta2',
discoveryServiceUrl=DISCOVERY_URL)
return client


if __name__ == '__main__':
parser = argparse.ArgumentParser(
description=create_task.__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)

parser.add_argument(
'--project',
help='Project of the queue to add the task to.'
)

parser.add_argument(
'--queue',
help='ID (short name) of the queue to add the task to.'
)

parser.add_argument(
'--location',
help='Location of the queue to add the task to.'
)

parser.add_argument(
'--payload',
help='Optional payload to attach to the push queue.'
)

parser.add_argument(
'--in_seconds',
help='The number of seconds from now to schedule task attempt.'
)

args = parser.parse_args()

create_task(
args.project, args.queue, args.location,
args.payload, args.in_seconds)
Loading