rfc044-in_parallel.md

RFC	Title	Author	Status	Type
44	Parallelization in Chef Recipes	John Keiser <jkeiser@chef.io>	On Hold	Standards Track

Parallelization in Chef Recipes

Chef presently runs the resources in a recipe serially, one after the next. In this proposal, groups of resources can be created which will run in parallel.

MVP (Minimum Viable Product) Features

resource_group

To run a group of resources in parallel, you write it this way:

resource_group :parallel do
  remote_file '/tmp/bigfile.txt' do
    source 'https://a.com/bigfile.txt'
  end
  remote_file '/tmp/bigfile2.txt' do
    source 'https://a.com/bigfile2.txt'
  end
end

resource_group groups a set of resources together to apply a behavior to all of them. It can take a options, including :serial, :parallel, and :concurrency => n (the maximum number of things to do simultaneously).

By default, a resource_group is :serial, meaning resources are executed in order, one by one.

Declaring Parallel Safety

In order to run inside a :parallel group, a resource must declare that it is parallel safe. To do this, it should override the parallel_safe? method and return true. If the parallel_safe? method is missing or returns false, the recipe will fail to compile.

Notifications

If a resource in a parallel group notifies another resource, behavior will depend on whether the notified resource is parallel-safe. If it is parallel-safe, the notification will be queued to run as part of the existing parallel group. If it is not parallel-safe, the notification will be queued to run after the entire parallel group completes.

Failure Handling

If a resource action in a parallel group fails, the parallel group will run all other actions to completion (or failure) before exiting the recipe.

Group notifications

When the entire group finishes, it is sometimes desirable to send a single notification. The subscribes and notifies primitives work inside a resource_group (whether it is serial or parallel) and if any resource is changed, the group will send the notification.

resource_groups can also be given names if a string is passed as the first parameter:

resource_group 'hi' do
  ...
end
file '/x.txt'
  action :nothing
  subscribes 'resource_group[hi]', :create
end

Output formatting: groups

Groups are a form of nested resource (whose output would be similar to that of use_inline_resources). Nested resources currently print something like this:

Recipe: @recipe_files::/Users/jkeiser/oc/code/opscode/chef-rfc/blah.rb
  * compound_resource[a] action create  * file[/Users/jkeiser/x.txt] action create (up to date)
  * file[/Users/jkeiser/y.txt] action create (up to date)
 (up to date)

With this proposal, a resource_group or other compound resource would print like this:

Recipe: @recipe_files::/Users/jkeiser/oc/code/opscode/chef-rfc/blah.rb
 * compound_resource[a] action create
   * file[/Users/jkeiser/x.txt] action create (up to date)
   * file[/Users/jkeiser/y.txt] action create (up to date)
   compound_resource[a] action create (up to date)

Output formatting: parallelism and batch

There are two major ways of formatting output when many things run in parallel: batched output or streamed output. In the batch case, we queue output for each resource and print it when the resource completes. In the streamed case, we print each piece of output as the resource goes. By default, output is batched. Streamed output is a future enhancement.

Future Features

This is intended to give a flavor of the future--anything underspecified here will require a new RFC.

Thread pool configuration

The Chef::Config.concurrency parameter, and --concurrency argument to chef-client, limits the number of concurrent parallel resources globally. To limit them specifically, you add parameters to the in_parallel directive like so:

resource_group :parallel, :concurrency => 10 do
  ...
end

in_serial

To run a group of actions serially, but inside a parallel grouping, you write this:

resource_group :parallel do
  resource_group do
    directory '/dir1'
    file '/dir1/blah.txt' do
      content 'hi'
    end
  end
  resource_group do
    directory '/dir2'
    file '/dir2/blah.txt' do
      content 'hi'
    end
  end
end

Custom parallelization

Some resource types (such as packages) handle parallelization internally. We will create a directive allowing multiple resources to collaborate and run a single parallel thing to handle multiple actions. For example, if you wrote 10 package directives inside resource_group :parallel, they would cooperate and run a single package installation command, passing all the packages to it.

Output formatting: parallelism and streaming

We will support streamed output from parallel groups, so that you can see when things start and when they end. How this happens and what the output looks like will be specified later.

Copyright

This work is in the public domain. In jurisdictions that do not allow for this, this work is available under CC0. To the extent possible under law, the person who associated CC0 with this work has waived all copyright and related or neighboring rights to this work.