Create per-pipeline ETL locks

[smgallo]

ETLv2 allows multiple ETL processes to be run concurrently but concurrent executions should not include the same action. To support this, lock files are not solely pid-based but also take into account the names of the actions that are being executed. This allows multiple ETL pipelines to be executed concurrently as long as the actions being performed do not overlap.

Description

A new ETL\LockFile class supports a lock file that contains a PID as well as a unique list of actions to be executed by that process. Prior to obtaining a lock, all ETLv2 lock files are checked to make sure that no running process is executing an action that overlaps the new process. Orphaned files are also cleaned up at this point. Note that a prefix (default of etlv2_) is required to identify ETLv2 lock files since multiple processes can run concurrently.

The ETL overseer script now supports --lock-dir and --lock-file-prefix options. In addition, a new etl section in the portal_settings.ini allows any option available as a command-line argument to be set (command-line options override portal_settings.ini values). For example:

[etl]
lock_dir = '/var/lock/xdmod'
lock_file_prefix = 'etlv2_'

Also implemented ETL\Maintenance\TestAction to facilitate testing. This action allows one of the following operations to be specified: sleep a number of seconds, throw an exception (helps test exception handling including the configurable behavior of continuing or stopping on an exception), echo a string. Additional test actions can be added as needed.

Motivation and Context

Allow parallel ETLv2 pipelines but facilitate identification of possible conflicts, especially when automated.

Tests performed

Tested various configurations:

1. Single pipeline
2. Multiple pipelines with no intersecting actions (both pipelines run)
3. Multiple pipelines with intersection actions (2nd pipeline aborts)
4. Orphaned lock files left by old processes (lock files are cleaned up)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project as found in the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.