Skip to content

Commit

Permalink
docs: archiveGC and archiveTTL parameters argoproj#10904 (argoproj#11050
Browse files Browse the repository at this point in the history
)

Signed-off-by: vitalyrychkov <57966425+vitalyrychkov@users.noreply.github.com>
  • Loading branch information
vitalyrychkov authored May 9, 2023
1 parent bd89a77 commit fb890a1
Showing 1 changed file with 77 additions and 7 deletions.
84 changes: 77 additions & 7 deletions docs/workflow-archive.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,94 @@

> v2.5 and after
For many uses, you may wish to keep workflows for a long time. Argo can save completed workflows to an SQL database.
If you want to keep completed workflows for a long time, you can use the workflow archive to save them in a Postgres or MySQL (>= 5.7.8) database.
The workflow archive stores the status of the workflow, which pods have been executed, what was the result etc.
The job logs of the workflow pods will not be archived.
If you need to save the logs of the pods, you must setup an [artifact repository](artifact-repository-ref.md) according to [this doc](configure-artifact-repository.md).

To enable this feature, configure a Postgres or MySQL (>= 5.7.8) database under `persistence` in [your configuration](workflow-controller-configmap.yaml) and set `archive: true`.
The quick-start deployment includes a Postgres database server.
In this case the workflow archive is already enabled.
Such a deployment is convenient for test environments, but in a production environment you must use a production quality database service.

Be aware that this feature will only archive the statuses of the workflows (which pods have been executed, what was the result, ...)
## Enabling Workflow Archive

However, the logs of each pod will NOT be archived. If you need to access the logs of the pods, you need to setup [an artifact repository](artifact-repository-ref.md) thanks to [this doc](configure-artifact-repository.md).
To enable archiving of the workflows, you must configure database parameters in the `persistence` section of [your configuration](workflow-controller-configmap.yaml) and set `archive:` to `true`.

In addition to the table specified in the config map above, the following tables are created when enabling archiving:
Example:

persistence:
archive: true
postgresql:
host: localhost
port: 5432
database: postgres
tableName: argo_workflows
userNameSecret:
name: argo-postgres-config
key: username
passwordSecret:
name: argo-postgres-config
key: password

You must also create the secret with database user and password in the namespace of the workflow controller.

Example:

kubectl create secret generic argo-postgres-config -n argo --from-literal=password=mypassword --from-literal=username=argodbuser

The following tables will be created in the database when you start the workflow controller with enabled archive:

* `argo_workflows`
* `argo_archived_workflows`
* `argo_archived_workflows_labels`
* `schema_history`

The database migration will only occur successfully if none of the tables exist. If a partial set of the tables exist, the database migration may fail and the Argo workflow-controller pod may fail to start. If this occurs delete all of the tables and try restarting the deployment.
## Automatic Database Migration

Every time the Argo workflow-controller starts with persistence enabled, it tries to migrate the database to the correct version.
If the database migration fails, the workflow-controller will also fail to start.
In this case you can delete all the above tables and restart the workflow-controller.

If you know what are you doing you also have an option to skip migration:

persistence:
skipMigration: true

## Required database permissions

### Postgres

The database user/role needs to have `CREATE` and `USAGE` permissions on the `public` schema of the database so that the necessary table can be generated during the migration.
The database user/role must have `CREATE` and `USAGE` permissions on the `public` schema of the database so that the tables can be created during the migration.

## Archive TTL

You can configure the time period to keep archived workflows before they will be deleted by the archived workflow garbage collection function.
The default is forever.

Example:

persistence:
archiveTTL: 10d

The `ARCHIVED_WORKFLOW_GC_PERIOD` variable defines the periodicity of running the garbage collection function.
The default value is documented [here](environment-variables.md).
When the workflow controller starts, it sets the ticker to run every `ARCHIVED_WORKFLOW_GC_PERIOD`.
It does not run the garbage collection function immediately and the first garbage collection happens only after the period defined in the `ARCHIVED_WORKFLOW_GC_PERIOD` variable.

## Cluster Name

Optionally you can set a unique name of your Kubernetes cluster. This name will populate the `clustername` field in the `argo_archived_workflows` table.

Example:

persistence:
clusterName: dev-cluster

## Disabling Workflow Archive

To disable archiving of the workflows, set `archive:` to `false` in the `persistence` section of [your configuration](workflow-controller-configmap.yaml).

Example:

persistence:
archive: false

0 comments on commit fb890a1

Please sign in to comment.