Skip to content

Commit

Permalink
Update Kanister docs and add new func
Browse files Browse the repository at this point in the history
  • Loading branch information
@SupriyaKasten
SupriyaKasten committed Aug 16, 2019
1 parent 77cbb4b commit 14fe565
Show file tree
Hide file tree
Showing 4 changed files with 222 additions and 44 deletions.
39 changes: 20 additions & 19 deletions docs/architecture.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -90,15 +90,18 @@ The definition of a `BlueprintAction` is:
// BlueprintPhase is a an individual unit of execution.
type BlueprintPhase struct {
Func string `json:"func"`
Name string `json:"name"`
Args map[string]interface{} `json:"args"`
Func string `json:"func"`
Name string `json:"name"`
ObjectRefs map[string]ObjectReference `json:"objects"`
Args map[string]interface{} `json:"args"`
}
- `Func` is required as the name of a registered Kanister function.
See :ref:`functions` for the list of functions supported by the controller.
- `Name` is mostly cosmetic. It is useful in quickly identifying which
phases the controller has finished executing.
- `Object` is a map of references to the Kubernetes objects on which
the action will be performed.
- `Args` is a map of named arguments that the controller will pass to
the Kanister function.
String argument values can be templates that the controller will
Expand Down Expand Up @@ -149,6 +152,7 @@ as follows:
ConfigMaps map[string]ObjectReference `json:"configMaps"`
Secrets map[string]ObjectReference `json:"secrets"`
Profile *ObjectReference `json:"profile"`
Options map[string]string `json:"options"`
}
- `Name` is required and specifies the action in the Blueprint.
Expand All @@ -163,6 +167,7 @@ as follows:
specified in the Blueprint referencing the Kubernetes object to be used.
- `Profile` is a reference to a :ref:`Profile<profiles>` Kubernetes
CustomResource that will be made available to the Blueprint.
- `Options` is used to specify additional values to be used in the Blueprint

As a reference, below is an example of a ActionSpec.

Expand Down Expand Up @@ -263,25 +268,22 @@ The definition of a `Profile` is:
type LocationType string
const (
LocationTypeGCS LocationType = "gcs"
LocationTypeS3Compliant LocationType = "s3Compliant"
LocationTypeAzure LocationType = "azure"
)
// Location
type Location struct {
Type LocationType `json:"type"`
S3Compliant *S3CompliantLocation `json:"s3Compliant"`
}
// S3Compliant
type S3CompliantLocation struct {
Bucket string `json:"bucket"`
Endpoint string `json:"endpoint"`
Prefix string `json:"prefix"`
Region string `json:"region"`
Type LocationType `json:"type"`
Bucket string `json:"bucket"`
Endpoint string `json:"endpoint"`
Prefix string `json:"prefix"`
Region string `json:"region"`
}
- `Credential` is required and used to specify the credentials associated with
the `Location`. Currently, only key pair s3 location credentials are
the `Location`. Currently, only key pair s3, gcs and azure location credentials are
supported.

The definition of `Credential` is as follows:
Expand Down Expand Up @@ -326,11 +328,10 @@ As a reference, below is an example of a Profile and the corresponding secret.
namespace: example-namespace
location:
type: s3Compliant
s3Compliant:
bucket: example-bucket
endpoint: <endpoint URL>:<port>
prefix: ""
region: ""
bucket: example-bucket
endpoint: <endpoint URL>:<port>
prefix: ""
region: ""
credential:
type: keyPair
keyPair:
Expand Down
193 changes: 185 additions & 8 deletions docs/functions.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,9 @@ to stop a database process before restoring files.
:align: left
:widths: 5,5,5,15

`namespace`, Yes, `string`, namespace in which to execute
`kind`, Yes, `string`, `deployment` or `statefulset`
`namespace`, No, `string`, namespace in which to execute
`name`, No, `string`, name of the workload to scale
`kind`, No, `string`, `deployment` or `statefulset`
`replicas`, Yes, `int`, The desired number of replicas

Example of scaling down:
Expand Down Expand Up @@ -223,6 +224,7 @@ ScaleWorkload.
`image`, Yes, `string`, image to be used the command
`volumes`, No, `map[string]string`, Mapping of `pvcName` to `mountPath` under which the volume will be available.
`command`, Yes, `[]string`, command list to execute
`serviceaccount`, No, `string`, service account info

.. note::
The `volumes` argument does not support `subPath` mounts so the
Expand Down Expand Up @@ -262,7 +264,8 @@ Example:
BackupData
----------

This function backs up data from a container into an S3 compatible object store.
This function backs up data from a container into any object store
supported by Kanister.

.. note::
It is important that the application includes a `kanister-tools`
Expand All @@ -281,6 +284,7 @@ Arguments:
`container`, Yes, `string`, container in which to execute
`includePath`, Yes, `string`, path of the data to be backed up
`backupArtifactPrefix`, Yes, `string`, path to store the backup on the object store
`encryptionKey`, No, `string`, encryption key to be used for backups

Outputs:

Expand All @@ -290,6 +294,7 @@ Outputs:
:widths: 5,5,15

`backupTag`,`string`, unique tag added to the backup
`backupID`,`string`, unique snapshot id generated by restic during backup

Example:

Expand All @@ -313,6 +318,63 @@ Example:
includePath: /mnt/data
backupArtifactPrefix: s3-bucket/path/artifactPrefix
.. _backupdataall:

BackupDataAll
-------------

This function concurrently backs up data from more than one pods into an any object store
supported by Kanister.

.. note::
It is important that the application includes a `kanister-tools`
sidecar container. This sidecar is necessary to run the
tools that capture path on a volume and store it on the object store.

Arguments:

.. csv-table::
:header: "Argument", "Required", "Type", "Description"
:align: left
:widths: 5,5,5,15

`namespace`, Yes, `string`, namespace in which to execute
`pods`, No, `string`, pods in which to execute (by default, runs on all the pods)
`container`, Yes, `string`, container in which to execute
`includePath`, Yes, `string`, path of the data to be backed up
`backupArtifactPrefix`, Yes, `string`, path to store the backup on the object store, appended by pod name later
`encryptionKey`, No, `string`, encryption key to be used for backups

Outputs:

.. csv-table::
:header: "Output", "Type", "Description"
:align: left
:widths: 5,5,15

`BackupAllInfo`,`string`, info about backup tag and identifier required for restore

Example:

.. code-block:: yaml
:linenos:
actions:
backup:
type: Deployment
outputArtifacts:
params:
keyValue:
backupInfo: "{{ .Phases.backupToObjectStore.Output.BackupAllInfo }}"
phases:
- func: BackupDataAll
name: BackupToObjectStore
args:
namespace: "{{ .Deployment.Namespace }}"
container: kanister-tools
includePath: /mnt/data
backupArtifactPrefix: s3-bucket/path/artifactPrefix
.. _restoredata:

RestoreData
Expand All @@ -339,15 +401,17 @@ and restores data to the specified path.
`namespace`, Yes, `string`, namespace in which to execute
`image`, Yes, `string`, image to be used for running restore
`backupArtifactPrefix`, Yes, `string`, path to the backup on the object store
`backupTag`, Yes, `string`, unique tag added during the backup
`backupIdentifier`, No, `string`, (required if backupTag not provided), unique snapshot id generated by restic during backup
`backupTag`, No, `string`, (required if backupIdentifier not provided)unique tag added during the backup
`restorePath`, No, `string`, path where data is restored
`pod`, No, `string`, pod to which the volumes are attached
`volumes`, No, `map[string]string`, Mapping of `pvcName` to `mountPath` under which the volume will be available
`encryptionKey`, No, `string`, encryption key to be used during backups

.. note::
The `image` argument requires the use of `kanisterio/kanister-tools`
image since it includes the required tools to restore data from
the S3 compatible object store.
the object store.
Between the `pod` and `volumes` arguments, exactly one argument
must be specified.

Expand Down Expand Up @@ -384,6 +448,80 @@ For this phase, we will use the `backupInfo` Artifact provided by backup functio
kind: Deployment
replicas: 1
.. _restoredataall:

RestoreDataAll
--------------

This function concurrently restores data backed up by the :ref:`backupdataall`
function, on more than one pods.
It concurrenlty runs a job Pod for each workload Pod, that mounts the
respective PVCs of that Pod and restores data to the specified path.

.. note::
It is extremely important that, the PVCs are not be currently
in use by an active application container, as they are required
to be mounted to the new Pod (ensure by using
ScaleWorkload with replicas=0 first).
For advanced use cases, it is possible to have concurrent access but
the PV needs to have RWX mode enabled and the volume needs to use a
clustered file system that supports concurrent access.

.. csv-table::
:header: "Argument", "Required", "Type", "Description"
:align: left
:widths: 5,5,5,15

`namespace`, Yes, `string`, namespace in which to execute
`image`, Yes, `string`, image to be used for running restore
`backupArtifactPrefix`, Yes, `string`, path to the backup on the object store
`restorePath`, No, `string`, path where data is restored
`pods`, No, `string`, pods to which the volumes are attached
`encryptionKey`, No, `string`, encryption key to be used during backups
`backupInfo`, Yes, `string`, snapshot info generated as output in BackupDataAll function

.. note::
The `image` argument requires the use of `kanisterio/kanister-tools`
image since it includes the required tools to restore data from
the object store.
Between the `pod` and `volumes` arguments, exactly one argument
must be specified.

Example:

Consider a scenario where you wish to restore the data backed up by the
:ref:`backupdataall` function. We will first scale down the application,
restore the data and then scale it back up. We will not specify `pods`
arg, so this function will restore data on all pods concurrently.
For this phase, we will use the `params` Artifact provided by BackupDataAll function.

.. code-block:: yaml
:linenos:
- func: ScaleWorkload
name: ShutdownApplication
args:
namespace: "{{ .Deployment.Namespace }}"
name: "{{ .Deployment.Name }}"
kind: Deployment
replicas: 0
- func: RestoreDataAll
name: RestoreFromObjectStore
args:
namespace: "{{ .Deployment.Namespace }}"
image: kanisterio/kanister-tools:0.20.0
backupArtifactPrefix: s3-bucket/path/artifactPrefix
backupInfo: "{{ .ArtifactsIn.params.KeyValue.backupInfo }}"
- func: ScaleWorkload
name: StartupApplication
args:
namespace: "{{ .Deployment.Namespace }}"
name: "{{ .Deployment.Name }}"
kind: Deployment
replicas: 2
CopyVolumeData
--------------

Expand All @@ -408,6 +546,7 @@ Arguments:
`namespace`, Yes, `string`, namespace the source PVC is in
`volume`, Yes, `string`, name of the source PVC
`dataArtifactPrefix`, Yes, `string`, path on the object store to store the data in
`encryptionKey`, No, `string`, encryption key to be used during backups

Outputs:

Expand All @@ -416,6 +555,8 @@ Outputs:
:align: left
:widths: 5,5,15

`backupID`,`string`, unique snapshot id generated by restic where data was copied
`backupRoot`,`string`, parent directory location of the data copied from
`backupArtifactLocation`,`string`, location in objectstore where data was copied
`backupTag`,`string`, unique string to identify this data copy

Expand Down Expand Up @@ -445,7 +586,9 @@ This function deletes the snapshot data backed up by the BackupData function.

`namespace`, Yes, `string`, namespace in which to execute
`backupArtifactPrefix`, Yes, `string`, path to the backup on the object store
`backupTag`, Yes, `string`, unique tag added during the backup
`backupIdentifier`, No, `string`, (required if backupTag not provided), unique snapshot id generated by restic during backup
`backupTag`, No, `string`, (required if backupIdentifier not provided)unique tag added during the backup
`encryptionKey`, No, `string`, encryption key to be used during backups

Example:

Expand All @@ -463,11 +606,45 @@ For this phase, we will use the `backupInfo` Artifact provided by backup functio
backupArtifactPrefix: s3-bucket/path/artifactPrefix
backupTag: "{{ .ArtifactsIn.backupInfo.KeyValue.backupIdentifier }}"
DeleteDataAll
-------------

This function concurrently deletes the snapshot data backed up by the BackupDataAll function.


.. csv-table::
:header: "Argument", "Required", "Type", "Description"
:align: left
:widths: 5,5,5,15

`namespace`, Yes, `string`, namespace in which to execute
`backupArtifactPrefix`, Yes, `string`, path to the backup on the object store
`backupInfo`, Yes, `string`, snapshot info generated as output in BackupDataAll function
`encryptionKey`, No, `string`, encryption key to be used during backups
`reclaimSpace`, NO, `bool`, provides a way to specify if space should be reclaimed

Example:

Consider a scenario where you wish to delete all the data backed up by the
:ref:`backupdataall` function.
For this phase, we will use the `params` Artifact provided by backup function.

.. code-block:: yaml
:linenos:
- func: DeleteDataAll
name: DeleteFromObjectStore
args:
namespace: "{{ .Namespace.Name }}"
backupArtifactPrefix: s3-bucket/path/artifactPrefix
backupInfo: "{{ .ArtifactsIn.params.KeyValue.backupInfo }}"
reclaimSpace: true
LocationDelete
--------------

This function uses a new Pod to delete the specified artifact
from an S3 compatible object store.
from an object store.

.. csv-table::
:header: "Argument", "Required", "Type", "Description"
Expand All @@ -479,7 +656,7 @@ from an S3 compatible object store.
.. note::
The Kubernetes job uses the `kanisterio/kanister-tools` image,
since it includes all the tools required to delete the artifact
from an S3 compatible object store.
from an object store.

Example:

Expand Down

0 comments on commit 14fe565

Please sign in to comment.