Skip to content
This repository has been archived by the owner on Aug 10, 2023. It is now read-only.

Updated App Engine Sidekiq + Rails tutorial to work with split health checks #1544

Merged
merged 9 commits into from
Dec 16, 2020
Merged

Updated App Engine Sidekiq + Rails tutorial to work with split health checks #1544

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
926011a
updated sidekiq appengine tutorial to work with split health checks
Dec 16, 2020
0f77027
Merge branch 'master' into fix-appengine-sidekiq-tutorial
Dec 16, 2020
3232e6f
Update appengine-ruby-rails-activejob-sidekiq.md
ToddKopriva Dec 16, 2020
1472160
fixed language, addered service attribute back to worker.yam
Dec 16, 2020
4431302
grammer tweak
Dec 16, 2020
6c0ed81
spcing fix
Dec 16, 2020
f01f862
Merge branch 'fix-appengine-sidekiq-tutorial' of https://github.com/M…
Dec 16, 2020
ebf3ce9
used correct terminology for worker.yaml description
Dec 16, 2020
89d498c
edit pass
ToddKopriva Dec 16, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 82 additions & 54 deletions tutorials/appengine-ruby-rails-activejob-sidekiq.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: Ruby on Rails background processing on App Engine with ActiveJob and Sidekiq
description: Learn how to run background jobs using Ruby on Rails ActiveJob.
author: chingor13
author: chingor13,mohayat
tags: App Engine, Ruby, Ruby on Rails, ActiveJob, Sidekiq
date_published: 2017-06-08
---
Expand All @@ -11,27 +11,28 @@ Jeff Ching | Software Engineer | Google
<p style="background-color:#CAFACA;"><i>Contributed by Google employees.</i></p>

This tutorial shows how to create and configure a [Ruby on Rails](http://rubyonrails.org/) application to run
background processing jobs on App Engine flexible environment using
background processing jobs on the App Engine flexible environment using
[ActiveJob](http://guides.rubyonrails.org/active_job_basics.html) and [Sidekiq](http://sidekiq.org/).

## Objectives

* Create a background processing job
* Deploy your application to Google App Engine flexible environment
* Verify background jobs are running
* Create a background processing job.
* Deploy your application to the App Engine flexible environment.
* Verify that background jobs are running.

## Before you begin

You'll need the following:

* A Google Cloud project. You can use an existing project or click the button to create a new project
* [Ruby 2.2.2+ installed](https://www.ruby-lang.org/en/documentation/installation/)
* A Google Cloud project. You can use an existing project or create a new project.
* [Ruby 2.2.2+ installed](https://www.ruby-lang.org/en/documentation/installation/).
* A Rails 4.2+ application. Follow the
[official "Getting Started with Rails" guide](http://guides.rubyonrails.org/getting_started.html) to get started.
* [Cloud SDK installed](https://cloud.google.com/sdk/downloads)
["Getting started with Rails" guide](http://guides.rubyonrails.org/getting_started.html) to get started.
* [Cloud SDK installed](https://cloud.google.com/sdk/downloads).
* A Redis instance running in your project. Follow [this guide](https://cloud.google.com/community/tutorials/setting-up-redis)
to set up Redis on Compute Engine. This tutorial assumes the Redis instance is running in the *default*
network so that the App Engine services can access it without restriction.
network so that the App Engine services can access it without restriction. Save or copy the password that you set for the Redis instance,
which is used later in this tutorial.

## Costs

Expand All @@ -50,20 +51,20 @@ and execution logic independently of queue and job runner implementations.
Ruby on Rails provides command-line tools for generating templated skeletons for things such as database migrations,
controllers, and even background jobs.

You will create a job named `HelloJob` that will accept a `name` argument and print "Hello #{name}" to standard output.
You create a job named `HelloJob` that accepts a `name` argument and prints `"Hello #{name}"` to standard output.

1. Use the Rails generator feature to create `HelloJob`:
1. Use the Rails generator feature to create the `HelloJob` job:

bin/rails generate job Hello

Rails creates stub files from templates:
Rails creates stub files from templates:

invoke test_unit
create test/jobs/hello_job_test.rb
create app/jobs/hello_job.rb
create app/jobs/application_job.rb

1. Edit your `app/jobs/hello_job.rb` with the following:
1. Edit `app/jobs/hello_job.rb` with the following:

class HelloJob < ApplicationJob
queue_as :default
Expand All @@ -76,8 +77,8 @@ You will create a job named `HelloJob` that will accept a `name` argument and pr

## Create a test URL to queue the job

You will create a controller named `HelloController` that will provide an action called `say` which will queue
our `HelloJob` to execute in the background.
You create a controller named `HelloController` that provides an action called `say` that queues
the `HelloJob` job to run in the background.

1. Use the Rails generator feature to create `HelloController`:

Expand All @@ -99,7 +100,7 @@ our `HelloJob` to execute in the background.
invoke scss
create app/assets/stylesheets/hello.scss

1. Add a `say` action to `HelloController`. Edit your `app/controllers/hello_controller.rb` with the following:
1. Add a `say` action to `HelloController` by adding the following to your `app/controllers/hello_controller.rb` file:

class HelloController < ApplicationController
def say
Expand All @@ -110,16 +111,16 @@ our `HelloJob` to execute in the background.

This action will queue our `HelloJob` with the provided request parameter `name`.

1. Create a route to this action. In `config/routes.rb`, add:
1. Create a route to this action by adding the following to your `config/routes.rb` file:

get '/hello/:name', to: 'hello#say'

When you make an HTTP GET request to `/hello/Jeff`, the `HelloController` will handle the request using the `say`
action with parameter `:name` as "Jeff"
action with parameter `:name` as `"Jeff"`.

## Configuring your background worker to use Sidekiq

ActiveJob can be configured with various different background job runners. This tutorial will cover Sidekiq which
ActiveJob can be configured with various different background job runners. This tutorial uses Sidekiq, which
requires a Redis instance to manage the job queue.

1. Add `sidekiq` gem to your `Gemfile`:
Expand All @@ -135,16 +136,16 @@ requires a Redis instance to manage the job queue.

## Deploying to App Engine flexible environment

For Sidekiq, the Redis connection configuration can be provided as an environment variable at runtime. You will
need to obtain the internal address of your redis instance. In the Cloud Console, go to the
**[VM Instances](https://console.cloud.google.com/compute/instances)** page and find the internal IP address of
your Compute Engine instance with Redis installed. This IP address will be provided via environment variables
at deploy time to configure Sidekiq.
For Sidekiq, the Redis connection configuration can be provided as an environment variable at run time. You
need to obtain the internal address and password of your Redis instance. In the Cloud Console, go to the
**[VM instances](https://console.cloud.google.com/compute/instances)** page and find the internal IP address of
your Compute Engine instance with Redis installed. This IP address and the password that you saved are be provided through environment variables
at deployment time to configure Sidekiq.

### Option A: Shared worker and web application

For this option, the App Engine service will run both the web server and a worker process via a process manager called
[foreman](https://ddollar.github.io/foreman/). If you choose this method, App Engine will scale your web and worker
For this option, the App Engine service runs both the web server and a worker process through a process manager called
[foreman](https://ddollar.github.io/foreman/). If you choose this method, App Engine scales your web and worker
instances together.

1. Add `foreman` gem to your `Gemfile`:
Expand All @@ -166,22 +167,45 @@ instances together.
env_variables:
REDIS_PROVIDER: REDIS_URL
REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
REDIS_PASSWORD: [PASSWORD]
SECRET_KEY_BASE: [SECRET_KEY]

Be sure to replace the `[REDIS_IP_ADDRESS]` with the internal IP address of your Redis instance. Also be sure to
replace the `[SECRET_KEY]` with a secret key for Rails sessions.
Replace `[REDIS_IP_ADDRESS]` and `[PASSWORD]` with the internal IP address of your Redis instance and its required password that you gave it,
respectively. Replace `[SECRET_KEY]` with a secret key for Rails sessions.

1. Deploy to App Engine

gcloud app deploy app.yaml

### Option B: Separate worker and web application

For this option, you are creating 2 App Engine services - one runs the web server and one runs worker processes. Both
For this option, you create two App Engine services: one runs the web server and one runs worker processes. Both
services use the same application code. This configuration allows you to scale background worker instances independently
of your web instances at the cost of potentially using more resources.
of your web instances at the cost of potentially using more resources. To pass the App Engine health checks and keep your background worker instance
alive, you use the [sidekiq_alive](https://github.com/arturictus/sidekiq_alive) gem to enable the Sidekiq server to respond to each liveness and readiness
request with a `200` HTTP status code.

1. Add `sidekiq_alive` to your `Gemfile`:

bundle add sidekiq_alive

1. Create a `sidekiq_alive.rb` initializer. In `config/initializers`:

SidekiqAlive.setup do |config|
# ==> Server port
# Port to bind the server.
config.port = 8080

1. Create an `app.yaml` for deploying the web service to Google App Engine:
# ==> Server path
# HTTP path to respond to.
config.path = '/health_check'

# ==> Rack server
# Web server used to serve an HTTP response.
config.server = 'puma'
end

1. Create an `app.yaml` file for deploying the web service to App Engine:

runtime: ruby
env: flex
Expand All @@ -191,12 +215,13 @@ of your web instances at the cost of potentially using more resources.
env_variables:
REDIS_PROVIDER: REDIS_URL
REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
REDIS_PASSWORD: [PASSWORD]
SECRET_KEY_BASE: [SECRET_KEY]

Be sure to replace the `[REDIS_IP_ADDRESS]` with the internal IP address of your Redis instance. Also be sure to
replace the `[SECRET_KEY]` with a secret key for Rails sessions.
Replace `[REDIS_IP_ADDRESS]` and `[PASSWORD]` with the internal IP address of your Redis instance and its required password that you gave it,
respectively. Replace `[SECRET_KEY]` with a secret key for Rails sessions.

1. Create a `worker.yaml` for deploying the worker service to Google App Engine:
1. Create a `worker.yaml` file for deploying the worker service to App Engine:

runtime: ruby
env: flex
Expand All @@ -207,54 +232,59 @@ of your web instances at the cost of potentially using more resources.
env_variables:
REDIS_PROVIDER: REDIS_URL
REDIS_URL: redis://[REDIS_IP_ADDRESS]:6379
REDIS_PASSWORD: [PASSWORD]
SECRET_KEY_BASE: [SECRET_KEY]

health_check:
enable_health_check: False
liveness_check:
path: '/health_check'

readiness_check:
path: '/health_check'

# Optional scaling configuration
manual_scaling:
instances: 1

Be sure to replace the `[REDIS_IP_ADDRESS]` with the internal IP address of your Redis instance. Also be sure to
replace the `[SECRET_KEY]` with a secret key for Rails sessions.
Replace `[REDIS_IP_ADDRESS]` and `[PASSWORD]` with the internal IP address of your Redis instance and its required password that you gave it, respectively.
Replace the `[SECRET_KEY]` with a secret key for Rails sessions.

Note that the health check is disabled here because the worker service is not running a web server and cannot
respond to the health check ping.
The `path` attribute for both the `liveness_check` and `readiness_check` sections has been set to the value of `config.path` in your `sidekiq_alive.rb`
initializer.

As mentioned above, you can configure scaling for the worker service independently of the default (web) service.
In the `manual_scaling` section, you have configured the worker service to start with 1 worker instance. For
more information on scaling options, see [scaling configuration options in app.yaml](https://cloud.google.com/appengine/docs/flexible/ruby/configuring-your-app-with-app-yaml#services).
As mentioned above, you can configure scaling for the worker service independent of the default (web) service.
In the `manual_scaling` section, you have configured the worker service to start with one worker instance. For
more information on scaling options, see
[scaling configuration options in app.yaml](https://cloud.google.com/appengine/docs/flexible/ruby/configuring-your-app-with-app-yaml#services).
If you choose an `automatic_scaling` option, be aware that scaling for the background processing is based off
of CPU utilization, not queue size.

1. Deploy both services to App Engine

gcloud app deploy app.yaml worker.yaml

## Verify your background queuing works
## Verify that background queuing works

1. In the Cloud Console, go to the
**[App Engine Services](https://console.cloud.google.com/appengine/services)** page. Locate the service that is
**[App Engine services](https://console.cloud.google.com/appengine/services)** page. Locate the service that is
running your background workers (if option A, it should be the *default* service, if option B, it should be
the *worker* service). Click Tools -> Logs for that service.
the *worker* service). Click **Tools** > **Logs** for that service.

1. In a separate window, navigate to your deployed Rails application at:
1. In a separate window, navigate to your deployed Rails application:

https://[YOUR_PROJECT_ID].appspot.com/hello/Jeff

Be sure to replace `[YOUR_PROJECT_ID]` with your Google Cloud project ID.
Replace `[YOUR_PROJECT_ID]` with your Google Cloud project ID.

1. Navigate back to the Logs dashboard. In the **Filter by label or text search** field, add `"Hello, Jeff"`
and you should see a logging statement like the following if using foreman:

13:13:52.000 worker.1 | Hello, Jeff

or if using a second service:
Or, if using a second service:

13:13:52.000 Hello, Jeff

Congratulations! You have successfully set up background job processing on Google App Engine with Sidekiq.
Congratulations! You have successfully set up background job processing on App Engine with Sidekiq.

## Cleaning up

Expand All @@ -266,8 +296,6 @@ resources.

The easiest way to eliminate billing is to delete the project you created for the tutorial.

To delete the project:

1. In the Cloud Console, go to the **[Projects](https://console.cloud.google.com/iam-admin/projects)** page.
1. Click the trash can icon to the right of the project name.

Expand All @@ -284,7 +312,7 @@ To delete an App Engine service:
1. Click the checkbox next to the service you wish to delete.
1. Click **Delete** at the top of the page to delete the service.

If you are trying to delete the *default* service, you cannot. Instead:
If you are trying to delete the *default* service, you cannot. Instead, do the following:

1. Click on the number of versions which will navigate you to App Engine Versions page.
1. Select all the versions you wish to disable and click **Stop** at the top of the page. This will free
Expand Down