-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add samples for Cloud Tasks #1068
Changes from 1 commit
87c721f
507a54b
1a43939
ee920fb
fa4270c
4d1f228
5e7a379
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -33,7 +33,7 @@ On Compute Engine and Container Engine, authentication credentials will be | |
automatically detected, but the instances must have been created with the | ||
necessary scopes. | ||
|
||
In any other environment, for example Compute Engine instance without the | ||
In any other environment, for example a Compute Engine instance without the | ||
necessary scopes, you should set `GOOGLE_APPLICATION_CREDENTIALS` environment | ||
variable to a JSON key file for a service account. | ||
|
||
|
@@ -42,17 +42,18 @@ for more information. | |
|
||
## Creating a queue | ||
|
||
To create a queue using the Cloud SDK, use the provided queue.yaml: | ||
To create a queue using the Cloud SDK, use the following gcloud command: | ||
|
||
gcloud app deploy queue.yaml | ||
gcloud alpha tasks queues create-app-engine-queue "my-appengine-queue" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. you don't need to put the queue ID in quotes There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
## Deploying the App Engine app | ||
|
||
First, vendor the dependencies into the project: | ||
Note: A newly created queue will route to the default App Engine service and | ||
version unless configured to do otherwise. Read the online help for the | ||
`create-app-engine-queue` or the `update-app-engine-queue` commands to learn | ||
about routing overrides for App Engine queues. | ||
|
||
pip install -r requirements.txt | ||
## Deploying the App Engine app | ||
|
||
Next, deploy the App Engine app | ||
Deploy the App Engine app with gcloud: | ||
|
||
gcloud app deploy | ||
|
||
|
@@ -61,47 +62,44 @@ Verify the index page is serving: | |
gcloud app browse | ||
|
||
The App Engine app serves as a target for the push requests. It has an | ||
endpoint `/set_payload` that that stores the payload from the HTTP POST data in | ||
Cloud Datastore. The payload can be accessed in your browser at the | ||
`/get_payload` endpoint with a GET request. | ||
endpoint `/log_payload` that reads the payload (i.e., the request body) of the | ||
HTTP POST request and logs it. The log output can be viewed with: | ||
|
||
gcloud app logs read | ||
|
||
## Running the Samples | ||
|
||
The project ID must be specified either as a command line argument using | ||
`--project-id`, or by editing `DEFAULT_PROJECT_ID` within `task_snippets.py`. | ||
|
||
Set the environment variables: | ||
Set environment variables: | ||
|
||
First, your project ID: | ||
|
||
export PROJECT_ID=my-project-id | ||
export LOCATION_ID=us-central1 | ||
export QUEUE_ID=my-appengine-queue # From queue.yaml | ||
|
||
View all queues: | ||
Then the queue ID, as specified at queue creation time. Queue IDs already | ||
created can be listed with `gcloud alpha tasks queue list`. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the command is There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
python app_engine_queue_snippets.py --api_key=$API_KEY list-queues --project_id=$PROJECT_ID --location_id=$LOCATION_ID | ||
export QUEUE_ID=my-appengine-queue | ||
|
||
Set the queue name as an environment variable: | ||
And finally the location ID, which can be discovered with | ||
`gcloud alpha tasks queue describe $QUEUE_ID`, with the location embedded in the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
"name" value (for instance, if the name is | ||
"projects/my-project/locations/us-central1/queues/my-appengine-queue", then the | ||
location is "us-central1"). | ||
|
||
export QUEUE_NAME=projects/$PROJECT_ID/locations/$LOCATION_ID/queues/$QUEUE_ID | ||
export LOCATION_ID=us-central1 | ||
|
||
Create a task, targeted at the `set_payload` endpoint with a payload specified: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Shouldn't this be log_payload? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
python app_engine_queue_snippets.py --api_key=$API_KEY create-task --queue_name=$QUEUE_NAME --payload=hello | ||
|
||
Now view that the payload was received and verify the count and payload: | ||
|
||
http://your-app-id.appspot.com/get_payload | ||
|
||
Create a task that will be scheduled for a few seconds in the future using | ||
the `--in_seconds` flag: | ||
|
||
python app_engine_queue_snippets.py --api_key=$API_KEY create-task --queue_name=$QUEUE_NAME --payload=hello --in_seconds=30 | ||
python create_app_engine_queue_task.py --project=$PROJECT_ID --queue=QUEUE_ID --location=LOCATION_ID --payload=hello | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. $ before variable names There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. curious why the queue_name parameter was removed in favor of breaking it out into three components? While I think the naming conventions of "queue name" vs. "queue" vs. "queue ID" are hard to follow, they're apparently in line with OnePlatform. Further, when we want to add more params like headers, App Engine routing, relative URL, and response view, the command will get pretty long. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I removed queue_name because gcloud queues list doesn't show the queue name. There's no real way to discover it without hitting the API separately. The queue name construct is still visible in the code itself, which seems appropriate for a sample. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. But the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, that's right. I think using the "queue_name" is an awkward abstraction, but since "describe" shows the queue_name I could add it back in. I'll reconsider. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I discussed with Jon and he told me that our other samples that have resource paths under the hoods use a similar "pass in the components on the command line, and assemble the full path in the code itself" system. So, I think I'll keep it like this for now. I think it's awkward to have the user use "describe" to look at the resource path just to get the location out of it, but I've been assured that the resource path will be included in "list" output soon at which point that will no longer be necessary. |
||
|
||
Since `--in_seconds` was set to 30, it will take 30 seconds for the new | ||
payload to be pushed to the `/get_payload` endpoint, which can then be viewed at: | ||
Now view that the payload was received and verify the payload: | ||
|
||
http://your-app-id.appspot.com/get_payload | ||
gcloud app logs read | ||
|
||
It might also be helpful to view the request logs of your App Engine app: | ||
Create a task that will be scheduled for a time in the future using the | ||
`--in_seconds` flag: | ||
|
||
https://console.cloud.google.com/logs | ||
python create_app_engine_queue_task.py --project=$PROJECT_ID --queue=QUEUE_ID --location=LOCATION_ID --payload=hello --in_seconds=30 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. $ before variable names There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
# Copyright 2017 Google Inc. All Rights Reserved. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
from __future__ import print_function | ||
|
||
import argparse | ||
import base64 | ||
import datetime | ||
import pprint | ||
|
||
from googleapiclient import discovery | ||
|
||
|
||
def seconds_from_now_to_rfc3339_datetime(seconds): | ||
"""Return an RFC 3339 datetime string for a number of seconds from now.""" | ||
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=seconds) | ||
return d.isoformat('T') + 'Z' | ||
|
||
|
||
def create_task(project, queue, location, payload=None, in_seconds=None): | ||
"""Create a task for a given queue with an arbitrary payload.""" | ||
client = get_client() | ||
|
||
url = '/log_payload' | ||
body = { | ||
'task': { | ||
'app_engine_task_target': { | ||
'http_method': 'POST', | ||
'relative_url': url | ||
} | ||
} | ||
} | ||
|
||
if payload is not None: | ||
# Payload is a string (unicode), so | ||
body['task']['app_engine_task_target']['payload'] = base64.b64encode( | ||
payload.encode()).decode() | ||
|
||
if in_seconds is not None: | ||
scheduled_time = seconds_from_now_to_rfc3339_datetime(in_seconds) | ||
body['task']['schedule_time'] = scheduled_time | ||
|
||
queue_name = 'projects/{}/locations/{}/queues/{}'.format( | ||
project, location, queue) | ||
|
||
print('Sending task {}'.format(pprint.pformat(body))) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Might it be even better to print this as JSON? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
response = client.projects().locations().queues().tasks().create( | ||
parent=queue_name, body=body).execute() | ||
|
||
# By default CreateTaskRequest.responseView is BASIC, so not all | ||
# information is retrieved by default because some data, such as payloads, | ||
# might be desirable to return only when needed because of its large size | ||
# or because of the sensitivity of data that it contains. | ||
print('Created task {}'.format(response['name'])) | ||
return response | ||
|
||
|
||
def get_client(): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. prefer that you just put this inline in the function above. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
"""Build an http client.""" | ||
DISCOVERY_URL = 'https://cloudtasks.googleapis.com/$discovery/rest?version=v2beta2' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it's not a constant, so it doesn't need to be capital. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's not a constant? It's not modified or replaced anywhere, it's just a string literal with a name, right? |
||
client = discovery.build('cloudtasks', 'v2beta2', | ||
discoveryServiceUrl=DISCOVERY_URL) | ||
return client | ||
|
||
|
||
if __name__ == '__main__': | ||
parser = argparse.ArgumentParser( | ||
description=create_task.__doc__, | ||
formatter_class=argparse.RawDescriptionHelpFormatter) | ||
|
||
parser.add_argument( | ||
'--project', | ||
help='Project of the queue to add the task to.' | ||
) | ||
|
||
parser.add_argument( | ||
'--queue', | ||
help='ID (short name) of the queue to add the task to.' | ||
) | ||
|
||
parser.add_argument( | ||
'--location', | ||
help='Location of the queue to add the task to.' | ||
) | ||
|
||
parser.add_argument( | ||
'--payload', | ||
help='Optional payload to attach to the push queue.' | ||
) | ||
|
||
parser.add_argument( | ||
'--in_seconds', | ||
help='The number of seconds from now to schedule task attempt.' | ||
) | ||
|
||
args = parser.parse_args() | ||
|
||
create_task( | ||
args.project, args.queue, args.location, | ||
args.payload, args.in_seconds) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above