Skip to content

ratelimit/dynflow

BranchesTags
 
 

Repository files navigation

Dynflow

Build Issues Gem version License

Note: There is a project page and documentation being build in url http://dynflow.github.io/. It's still work in progress but you may find useful information there. It'll eventually replace this README.

Dynflow [DYN(amic work)FLOW] is a workflow engine written in Ruby that allows to:

  • keep track of the progress of running process
  • run the code asynchronously
  • resume the process when something goes wrong, skip some steps when needed
  • detect independent parts and run them concurrently
  • compose simple actions into more complex scenarios
  • extend the workflows from third-party libraries
  • keep consistency between local transactional database and external services
  • suspend the long-running steps, not blocking the thread pool
  • cancel steps when possible
  • extend the actions behavior with middlewares
  • define the input/output interface between the building blocks (planned)
  • define rollback for the workflow (planned)
  • have multiple workers for distributing the load (planned)

Dynflow doesn't try to choose the best tool for the jobs, as the right tool depends on the context. Instead, it provides interfaces for persistence, transaction layer or executor implementation, giving you the last word in choosing the right one (providing default implementations as well).

Screenshot

Current status

Dynflow has been under heavy development for several months to be able to support the services orchestration in the Katello and Foreman projects, getting to production-ready state in couple of weeks.

How it works

In traditional workflow engines, you specify a static workflow and then run it with various inputs. Dynflow takes different approach. You specify the inputs and the workflow is generated on the fly. You can either specify the steps explicitly or subscribe one action to another. This is suitable for plugin architecture, where you can't write the whole process on one place.

Dynflow doesn't differentiate between workflow and action. Instead, every action can populate another actions. This allows composing more simpler workflows into a big one.

The whole execution is done in three phases:

  1. Plan phase

Construct the execution plan for the workflow. Two mechanisms are used to get the set of actions to be executed:

a. explicit calls of `plan_action` methods in the `plan` method

b. implicit associations: an action A subscribes to an action B,
which means that the action A is executed whenever the action B
occurs.

The output of this phase is a set of actions and their inputs.

  1. Run phase

The plan is being executed step by step, calling the run method of an action with corresponding input. The results of every action are written into output attribute.

The run method should be stateless, with all the needed information included in the input from planning phase. This allows us to control the workflow execution: the state of every action can be serialized therefore the workflow itself can be persisted. This makes it easy to recover from failed actions by rerunning it.

  1. Finalize phase

Take the results from the execution phase and perform some additional tasks. This is suitable for example for recording the results into database.

Every action can participate in every phase.

Examples

The examples directory contains simple ruby scripts different features in action. You can just run the example files and see the Dynflow in action.

  • orchestrate.rb - example worlflow of getting some infrastructure up and running, with ability to rescue from some error states.

  • orchestrate_evented.rb - the same workflow using the ability to suspend/wakeup actions while waiting for some external event. It also demonstrates the ability to cancel actions that support it.

  • remote_executor.rb - example of executing the flows in external process

The Anatomy of Action Class

# every action needs to inherit from Dynflow::Action
class Action < Dynflow::Action

  # OPTIONAL: the input format for the execution phase of this action
  # (https://github.com/iNecas/apipie-params for more details.
  # Validations can be performed against this description (turned off
  # for now)
  input_format do
    param :id, Integer
    param :name, String
  end

  # OPTIONAL: every action can produce an output in the execution
  # phase. This allows to describe the output.
  output_format do
    param :uuid, String
  end

  # OPTIONAL: this specifies that this action should be performed when
  # AnotherAction is triggered.
  def self.subscribe
    AnotherAction
  end

  # OPTIONAL: executed during the planning phase. It's possible to
  # specify explicitly the workflow here. By default it schedules just
  # this action.
  def plan(object_1, object_2)
    # +plan_action+ schedules the SubAction to be part of this
    # workflow
    # the +object_1+ is passed to the +SubAction#plan+ method.
    plan_action SubAction, object_1
    # we can specify, where in the workflow this action should be
    # placed, as well as prepare the input.
    plan_self { id: object_2.id, name: object_2.name}
  end

  # OPTIONAL: run the execution part of this action. Transform the
  # data from +input+ to +