From 17fea8371ed73d818f691666504b7d00e33a80b0 Mon Sep 17 00:00:00 2001 From: llewellyn-sl Date: Mon, 16 Sep 2024 21:24:11 +0200 Subject: [PATCH 01/15] 1st commit --- .../version-24.1/seqerakit/overview.mdx | 417 ++++++++++++++++++ .../seqerakit/templates/actions.yml | 24 + .../seqerakit/templates/compute-envs.yml | 36 ++ .../seqerakit/templates/credentials.yml | 44 ++ .../seqerakit/templates/datasets.yml | 8 + .../seqerakit/templates/labels.yml | 6 + .../seqerakit/templates/launch.yml | 22 + .../seqerakit/templates/members.yml | 5 + .../seqerakit/templates/organizations.yml | 8 + .../seqerakit/templates/participants.yml | 10 + .../seqerakit/templates/pipelines.yml | 24 + .../seqerakit/templates/secrets.yml | 6 + .../seqerakit/templates/teams.yml | 8 + .../seqerakit/templates/workspaces.yml | 8 + 14 files changed, 626 insertions(+) create mode 100644 platform_versioned_docs/version-24.1/seqerakit/overview.mdx create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/members.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml diff --git a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx new file mode 100644 index 00000000..7b4a9abc --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx @@ -0,0 +1,417 @@ +--- +title: "Overview" +description: "An overview of seqerakit" +date: "17 May 2024" +tags: [data, studios] +--- + +`seqerakit` is a Python wrapper for the [Platform CLI](../cli/overview.mdx). Use `seqerakit` to automate the creation of all Seqera Platform entities via a simple configuration file in YAML format. + +The key features are: + +- **Simple configuration**: All of the command-line options available in the Platform CLI can be defined in simple YAML format. +- **Infrastructure as code**: Enable users to manage and provision their infrastructure. +- **Automation**: End-to-end creation of Seqera Platform entities, from adding an organization to launching pipelines. + +## Installation + +`seqerakit` has three dependencies: + +1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli/releases) +2. [Python (`>=3.8`)](https://www.python.org/downloads/) +3. [PyYAML](https://pypi.org/project/PyYAML/) + +### Conda + +To install `seqerakit` and its dependencies via Conda, first configure the correct channels: + +```console +conda config --add channels bioconda +conda config --add channels conda-forge +conda config --set channel_priority strict +``` + +Then create a conda environment with `seqerakit` installed: + +``` +conda env create -n seqerakit seqerakit +conda activate seqerakit +``` + +### Pip + +If you already have [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): + +```console +pip install seqerakit +``` + +Overwrite an existing installation to use the latest version: + +```console +pip install --upgrade --force-reinstall seqerakit +``` + +### Local development installation + +Install the development branch of `seqerakit` on your local machine to test the latest features and updates: + +1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. +1. To install directly from pip: + ```bash + pip install git+https://github.com/seqeralabs/seqera-kit.git@dev + ``` +1. Alternatively, clone the repository locally and install manually: + ```bash + git clone https://github.com/seqeralabs/seqera-kit.git + cd seqera-kit + git checkout dev + pip install . + ``` +1. Verify your installation: + ```bash + pip show seqerakit + ``` + +## Configuration + +Create a Seqera Platform access token using the [Seqera Platform](https://seqera.io/) web interface via the **Your Tokens** page in your profile. + +`seqerakit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below: + +```bash +export TOWER_ACCESS_TOKEN= +``` + +For Enterprise installations of Seqera Platform, you will also need to configure the API endpoint that will be used to connect to the Platform. You can do so by exporting the following environment variable: +```bash +export TOWER_API_ENDPOINT= +``` +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud. + + +## Usage + +To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to the Platform is working as expected. This will run the `tw info` command under the hood: + +```bash +seqerakit --info +``` + +Use the `--help` or `-h` parameter to list the available commands and their associated options: +```bash +seqerakit --help +``` + +Use `--version` or `-v` to retrieve the current version of your seqerakit installation: +```bash +seqerakit --version +``` +### Input +`seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin). + +#### Using File Path + +```bash +seqerakit /path/to/file.yaml +``` +#### Using stdin + +```console +$ cat file.yaml | seqerakit - +``` +See the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s). + +### Dryrun + +To print the commands that would executed with `tw` when using a YAML file, you can run `seqerakit` with the `--dryrun` flag: + +```bash +seqerakit file.yaml --dryrun +``` + +### Recursively delete + +Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag: + +```bash +seqerakit file.yaml --delete +``` + +For example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `seqerakit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization. + +### Using `tw` specific CLI options + +`tw` specific CLI options can be specified with the `--cli=` flag: + +```bash +seqerakit file.yaml --cli="--arg1 --arg2" +``` + +You can find the full list of options by running `tw -h`. + +The Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag. + +To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes. + +For example: + +```bash +seqerakit file.yaml --cli="--insecure" +``` + +For Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `seqerakit` as you would to `tw`, preceded by `--cli=`. + +For example: + +```bash +seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts" +``` + +Note: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error. + +## Specify targets +When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names. + +For example, given a YAML file that defines the following resources: + +```yaml +workspaces: + - name: 'showcase' + organization: 'seqerakit_automation' +... +compute-envs: + - name: 'compute-env' + type: 'aws-batch forge' + workspace: 'seqerakit/test' +... +pipelines: + - name: "hello-world-test-seqerakit" + url: "https://github.com/nextflow-io/hello" + workspace: 'seqerakit/test' + compute-env: "compute-env" +... +``` + +You can target the creation of `pipelines` only by running: + +```bash +seqerakit test.yml --targets pipelines +``` +This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`. + +### Multiple Targets +You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run: + +```bash +seqerakit test.yml --targets workspaces,pipelines +``` + +## YAML Configuration Options + +There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. + +### 1. Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +Alternatively, to specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Optionally, you may provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. + +### 2. `overwrite` Functionality +For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. + +`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```console +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` +### 3. Specifying JSON configuration files with `file-path` +The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` + + +## Quick start + +You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. + +You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline. + +### Launch via YAML + +1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `` and `` entries as required: + + ```yaml # noqa + launch: + - name: 'hello-world' # Workflow name + workspace: '' # Workspace name + compute-env: '' # Compute environment + revision: 'master' # Pipeline revision + pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL + ``` + +2. Launch the pipeline with `seqerakit`: + + ```bash + seqerakit hello-world-config.yml + ``` + +3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! + +### Launch via a Python script + +You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python. + +1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `` and `` entries as required. + +2. Launch the pipeline with `seqerakit`: + +```bash + python launch_hello_world.py +``` + +3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! + +## Defining your YAML file using CLI options + +All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity. + +1. Retrieve CLI Options + +To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: + +```console +$ tw pipelines add -h + +Usage: tw pipelines add [OPTIONS] PIPELINE_URL + +Add a workspace pipeline. + +Parameters: +* PIPELINE_URL Nextflow pipeline URL. + +Options: +* -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... +``` +2. Define Key-Value Pairs in YAML + +Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: + +```yaml +pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' +``` + +In this example: + +- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. +- The corresponding values are user-defined + +### Best Practices: +- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. +- For complex configurations, refer to the [Templates](./templates/) provided in this repository. + + +## Templates + +We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: + +- [organizations.yml](./templates/organizations.yml) +- [teams.yml](./templates/teams.yml) +- [workspaces.yml](./templates/workspaces.yml) +- [participants.yml](./templates/participants.yml) +- [credentials.yml](./templates/credentials.yml) +- [secrets.yml](./templates/secrets.yml) +- [compute-envs.yml](./templates/compute-envs.yml) +- [actions.yml](./templates/actions.yml) +- [datasets.yml](./templates/datasets.yml) +- [labels.yml](./templates/labels.yml) +- [pipelines.yml](./templates/pipelines.yml) +- [launch.yml](./templates/launch.yml) + +## Real world example + +Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline. + +You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables: + +```console +$TOWER_GITHUB_PASSWORD +$DOCKERHUB_PASSWORD +$AWS_ACCESS_KEY_ID +$AWS_SECRET_ACCESS_KEY +$AWS_ASSUME_ROLE_ARN +$AZURE_BATCH_KEY +$AZURE_STORAGE_KEY +$GOOGLE_KEY +$SENTIEON_LICENSE_BASE64 +``` + +## Contributions and Support + +If you would like to contribute to `seqerakit`, please see the [contributing guidelines](./.github/CONTRIBUTING.md). + +For further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/seqera-kit/issues) in this repository. + +## Credits + +`seqerakit` was written by [Esha Joshi](https://github.com/ejseqera), [Adam Talbot](https://github.com/adamrtalbot) and [Harshil Patel](https://github.com/drpatelh) from the Scientific Development Team at [Seqera Labs](https://seqera.io/). diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml new file mode 100644 index 00000000..d243b9a8 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml @@ -0,0 +1,24 @@ +## To see the full list of options available run: "tw actions add" +actions: + - type: 'github' # required + name: 'my_github_action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'my_organization/my_workspace' # required + compute-env: 'my_aws_compute_environment' # required + work-dir: 's3://my_bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://my_bucket/my_results' + overwrite: True # optional + - type: 'tower' # required + name: 'my_tower_action' # required + pipeline: 'https://github.com/my_username/my_repo' # required + workspace: 'my_organization/my_workspace' # required + compute-env: 'my_aws_compute_environment' # required + work-dir: 's3://my_bucket' # required + revision: 'main' # required + profile: 'test' # optional + params: # optional + outdir: 's3://my_bucket/my_results' + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml new file mode 100644 index 00000000..b6bf1710 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml @@ -0,0 +1,36 @@ +## To see the full list of options available run: "tw compute-envs add" +## The options required to create compute environments can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from Seqera Platform with the "tw compute-envs export" command +compute-envs: +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True # optional + +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure + name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://my-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma separated list + allow-buckets: 's3://my-bucket,s3://my-other-bucket' # optional, comma separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml new file mode 100644 index 00000000..b7fef41d --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml @@ -0,0 +1,44 @@ +## To see the full list of options available run: "tw credentials add" +## To avoid exposing sensitive information about your credentials, +## use environment variables to supply passwords and secret keys +credentials: + - type: 'github' # required + name: 'my_github_credentials' # required + workspace: 'my_organization/my_workspace' # required + username: 'my_username' # required + password: '$SEQPLATFORM_GITHUB_PASSWORD' # required + overwrite: True # optional + - type: 'container-reg' # required + name: 'my_dockerhub_credentials' # required + workspace: 'my_organization/my_workspace' # required + username: 'my_username' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + - type: 'google' # required + name: 'my_google_credentials' # required + workspace: 'my_organization/my_workspace' # required + key: '$GOOGLE_KEY' # required + overwrite: True # optional + - type: 'aws' # required + name: 'my_aws_credentials' # required + workspace: 'my_organization/my_workspace' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + - type: 'azure' # required + name: 'my_azure_credentials' # required + workspace: 'my_organization/my_workspace' # required + batch-key: '$AZURE_BATCH_KEY' # required + batch-name: 'my_storage_name' # required + storage-key: '$AZURE_STORAGE_KEY' # required + storage-name: 'my_storage_name' # required + overwrite: True # optional + - type: 'codecommit' # required + name: 'codecommit_credentials' # required + workspace: 'my_organization/my_workspace' # required + access-key: '$CODECOMMIT_USER' # required + secret-key: '$CODECOMMIT_PASSWORD' # required + base-url: '$CODECOMMIT_BASEURL' # optional + overwrite: False # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml new file mode 100644 index 00000000..dce22bd8 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml @@ -0,0 +1,8 @@ +## To see the full list of options available run: "tw datasets add" +datasets: + - name: 'my_dataset' # required + description: 'My test dataset' # optional + header: true # optional + workspace: 'my_organization/my_workspace' # required + file-path: './datasets/my_dataset.csv' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml new file mode 100644 index 00000000..6a75adbd --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml @@ -0,0 +1,6 @@ +## To see the full list of options available run: "tw labels add" +labels: + - name: 'label_name' # required + value: 'label_value' # required + workspace: 'my_organization/my_workspace' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml new file mode 100644 index 00000000..c281c398 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml @@ -0,0 +1,22 @@ +## To see the full list of options available run: "tw launch" +## The options will vary if you are launching a pipeline: +## 1. Pre-configured in the Seqera Platform Launchpad where most options have already been specified +## 2. From source via a remote Git URL where you will need to explicitly specify all launch options +## Note: overwrite is not supported for "tw launch" + +launch: + - name: 'my_launchpad_launch' # required + workspace: 'my_organization/my_workspace' # required + pipeline: 'my_launchpad_pipeline' # required + params: # optional + outdir: 's3://my_bucket/my_results' + - name: 'my_remote_launch' # required + workspace: 'my_organization/my_workspace' # required + compute-env: 'my_aws_compute_environment' # required + pipeline: 'https://github.com/my_username/my_repo' # required + work-dir: 's3://my_bucket' # optional + profile: 'test' # optional + revision: 'main' # optional + params-file: './pipelines/my_params.yml' # optional + config: './pipelines/my_nextflow.config' # optional + pre-run: './pipelines/my_pre_run.txt' # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml new file mode 100644 index 00000000..d3f2d825 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml @@ -0,0 +1,5 @@ +## To see the full list of options available run: "tw members add" +participants: + - user: 'bob@myorg.io' # required + organization: 'myorg' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml new file mode 100644 index 00000000..acd4d18d --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml @@ -0,0 +1,8 @@ +## To see the full list of options available run: "tw organizations add" +organizations: + - name: 'my_organization' # required + full-name: 'my_organization' # required + description: 'My test organisation' # optional + location: 'Global' # optional + website: 'https://my_organization/' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml new file mode 100644 index 00000000..e0d69ccb --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml @@ -0,0 +1,10 @@ +## To see the full list of options available run: "tw participants add" +participants: + - name: 'my_team' # required + type: 'TEAM' # required + workspace: 'my_organization/my_workspace' # required + role: 'ADMIN' # required + - name: 'my_team_member@gmail.com' # required + type: 'MEMBER' # required + workspace: 'my_organization/my_workspace' # required + role: 'LAUNCH' # required \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml new file mode 100644 index 00000000..30882265 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml @@ -0,0 +1,24 @@ +## To see the full list of options available run: "tw pipelines add" +## The options required to create pipelines can be specified: +## 1. Explicitly in this file +## 2. Via a JSON file exported from SeqeraPlatform with the "tw pipelines export" command +pipelines: + - name: 'my_first_pipeline' # required + workspace: 'my_organization/my_workspace' # required + description: 'My test pipeline' # optional + compute-env: 'my_aws_compute_environment' # required + work-dir: 's3://my_bucket' # optional + profile: 'test' # optional + revision: 'main' # required + params: # optional + outdir: 's3://my-bucket/my_results' + config: './pipelines/my_nextflow.config' # optional + pre-run: './pipelines/my_pre_run.txt' # optional + url: 'https://github.com/my_username/my_repo' # required + overwrite: True # optional + - name: 'my_second_pipeline' # required + workspace: 'my_organization/my_workspace' # required + description: 'My test pipeline' # optional + compute-env: 'my_aws_compute_environment' # required + file-path: './pipelines/my_pipeline.json' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml new file mode 100644 index 00000000..b5cd79bb --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml @@ -0,0 +1,6 @@ +## To see the full list of options available run: "tw secrets add" +secrets: + - name: 'my_secret' # required + workspace: 'my_organization/my_workspace' # required + value: 'my_secret_value' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml new file mode 100644 index 00000000..e6f921b2 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml @@ -0,0 +1,8 @@ +## To see the full list of options available run: "tw teams add" +teams: + - name: 'my_team' # required + organization: 'my_organization' # required + description: 'My test team' # optional + members: # optional + - 'my_team_member@gmail.com' + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml new file mode 100644 index 00000000..c11cea33 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml @@ -0,0 +1,8 @@ +## To see the full list of options available run: "tw workspaces add" +workspaces: + - name: 'my_workspace' # required + full-name: 'my_workspace' # required + organization: 'my_organization' # required + description: 'My test workspace' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional \ No newline at end of file From e16db958ac528147a7ea2d2ae1376b6df56d482f Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Mon, 21 Oct 2024 19:11:29 +0200 Subject: [PATCH 02/15] Create overview, install, commands pages --- .../version-24.1/seqerakit/commands.mdx | 314 +++++++++++++ .../version-24.1/seqerakit/installation.mdx | 104 +++++ .../version-24.1/seqerakit/overview.mdx | 415 +----------------- .../version-24.1-sidebars.json | 19 +- 4 files changed, 442 insertions(+), 410 deletions(-) create mode 100644 platform_versioned_docs/version-24.1/seqerakit/commands.mdx create mode 100644 platform_versioned_docs/version-24.1/seqerakit/installation.mdx diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx new file mode 100644 index 00000000..2f61fccd --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -0,0 +1,314 @@ +--- +title: "Commands" +description: "seqerakit command options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, commands] +--- + +Use the `--help` or `-h` parameter to list the available commands and their associated options: + +```bash +seqerakit --help +``` + +### Input + +`seqerakit` supports input through paths to YAML configuration foiels or directly from standard input (stdin). + +- Using file path: + + ```bash + seqerakit /path/to/file.yaml + ``` + +- Using stdin: + + ```console + $ cat file.yaml | seqerakit - + ``` + +See [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. + +### Dryrun + +It is recommended to confirm that your configuration and command are correct before creating resources in your Platform account, particularly when automating end-to-end creation of multiple Platform entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your command with the `--dryrun` flag: + +```bash +seqerakit file.yaml --dryrun +``` + +### Delete resources + +Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag: + +```bash +seqerakit file.yaml --delete +``` + +For example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `seqerakit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization. + +### Using `tw` specific CLI options + +`tw` specific CLI options can be specified with the `--cli=` flag: + +```bash +seqerakit file.yaml --cli="--arg1 --arg2" +``` + +See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. + +The Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag. + +To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes. + +For example: + +```bash +seqerakit file.yaml --cli="--insecure" +``` + +For Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `seqerakit` as you would to `tw`, preceded by `--cli=`. + +For example: + +```bash +seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts" +``` + +Note: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error. + +## Specify targets +When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names. + +For example, given a YAML file that defines the following resources: + +```yaml +workspaces: + - name: 'showcase' + organization: 'seqerakit_automation' +... +compute-envs: + - name: 'compute-env' + type: 'aws-batch forge' + workspace: 'seqerakit/test' +... +pipelines: + - name: "hello-world-test-seqerakit" + url: "https://github.com/nextflow-io/hello" + workspace: 'seqerakit/test' + compute-env: "compute-env" +... +``` + +You can target the creation of `pipelines` only by running: + +```bash +seqerakit test.yml --targets pipelines +``` +This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`. + +### Multiple Targets +You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run: + +```bash +seqerakit test.yml --targets workspaces,pipelines +``` + +## YAML Configuration Options + +There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. + +### 1. Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +Alternatively, to specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Optionally, you may provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. + +### 2. `overwrite` Functionality +For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. + +`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```console +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` +### 3. Specifying JSON configuration files with `file-path` +The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` + + +## Quick start + +You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. + +You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline. + +### Launch via YAML + +1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `` and `` entries as required: + + ```yaml # noqa + launch: + - name: 'hello-world' # Workflow name + workspace: '' # Workspace name + compute-env: '' # Compute environment + revision: 'master' # Pipeline revision + pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL + ``` + +2. Launch the pipeline with `seqerakit`: + + ```bash + seqerakit hello-world-config.yml + ``` + +3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! + +### Launch via a Python script + +You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python. + +1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `` and `` entries as required. + +2. Launch the pipeline with `seqerakit`: + +```bash + python launch_hello_world.py +``` + +3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! + +## Defining your YAML file using CLI options + +All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity. + +1. Retrieve CLI Options + +To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: + +```console +$ tw pipelines add -h + +Usage: tw pipelines add [OPTIONS] PIPELINE_URL + +Add a workspace pipeline. + +Parameters: +* PIPELINE_URL Nextflow pipeline URL. + +Options: +* -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... +``` +2. Define Key-Value Pairs in YAML + +Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: + +```yaml +pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' +``` + +In this example: + +- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. +- The corresponding values are user-defined + +### Best Practices: +- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. +- For complex configurations, refer to the [Templates](./templates/) provided in this repository. + + +## Templates + +We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: + +- [organizations.yml](./templates/organizations.yml) +- [teams.yml](./templates/teams.yml) +- [workspaces.yml](./templates/workspaces.yml) +- [participants.yml](./templates/participants.yml) +- [credentials.yml](./templates/credentials.yml) +- [secrets.yml](./templates/secrets.yml) +- [compute-envs.yml](./templates/compute-envs.yml) +- [actions.yml](./templates/actions.yml) +- [datasets.yml](./templates/datasets.yml) +- [labels.yml](./templates/labels.yml) +- [pipelines.yml](./templates/pipelines.yml) +- [launch.yml](./templates/launch.yml) + +## Real world example + +Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline. + +You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables: + +```console +$TOWER_GITHUB_PASSWORD +$DOCKERHUB_PASSWORD +$AWS_ACCESS_KEY_ID +$AWS_SECRET_ACCESS_KEY +$AWS_ASSUME_ROLE_ARN +$AZURE_BATCH_KEY +$AZURE_STORAGE_KEY +$GOOGLE_KEY +$SENTIEON_LICENSE_BASE64 +``` \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx new file mode 100644 index 00000000..abafb366 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -0,0 +1,104 @@ +--- +title: "Installation" +description: "seqerakit installation options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, installation] +--- + +`seqerakit` has three dependencies: + +1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli/releases) +2. [Python (`>=3.8`)](https://www.python.org/downloads/) +3. [PyYAML](https://pypi.org/project/PyYAML/) + +### Pip + +If you already have [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): + +```console +pip install seqerakit +``` + +Overwrite an existing installation to use the latest version: + +```console +pip install --upgrade --force-reinstall seqerakit +``` + +### Conda + +To install `seqerakit` and its dependencies via Conda, first configure the correct channels: + +```console +conda config --add channels bioconda +conda config --add channels conda-forge +conda config --set channel_priority strict +``` + +Then create a conda environment with `seqerakit` installed: + +``` +conda env create -n seqerakit seqerakit +conda activate seqerakit +``` + +#### Local development installation + +Install the development branch of `seqerakit` on your local machine to test the latest features and updates: + +1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. +1. To install directly from pip: + ```bash + pip install git+https://github.com/seqeralabs/seqera-kit.git@dev + ``` +1. Alternatively, clone the repository locally and install manually: + ```bash + git clone https://github.com/seqeralabs/seqera-kit.git + cd seqera-kit + git checkout dev + pip install . + ``` +1. Verify your installation: + ```bash + pip show seqerakit + ``` + +### Configuration + +Create a Seqera Platform access token using the [Seqera Platform](https://cloud.seqera.io/) web interface via the **Your Tokens** page in your profile. + +`seqerakit` reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: + +```bash +export TOWER_ACCESS_TOKEN= +``` + +For Platform Enterprise installations, you must specify the custom API endpoint used to connect to the Platform. Export the API endpoint environment variable: + +```bash +export TOWER_API_ENDPOINT= +``` + +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud. + +### Usage + +To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to the Platform is working as expected. This will run the `tw info` command under the hood: + +```bash +seqerakit --info +``` + +Use `--version` or `-v` to retrieve the current version of your seqerakit installation: + +```bash +seqerakit --version +``` + +Use the `--help` or `-h` parameter to list the available commands and their associated options: + +```bash +seqerakit --help +``` + +See [Commands](./commands.mdx) for detailed instructions to use seqerakit. \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx index 7b4a9abc..ccbc08bf 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx @@ -1,417 +1,26 @@ --- title: "Overview" description: "An overview of seqerakit" -date: "17 May 2024" -tags: [data, studios] +date: "10 Oct 2024" +tags: [seqerakit, cli, automation] --- -`seqerakit` is a Python wrapper for the [Platform CLI](../cli/overview.mdx). Use `seqerakit` to automate the creation of all Seqera Platform entities via a simple configuration file in YAML format. +`seqerakit` is a Python wrapper that allows you to set [Platform CLI](../cli/overview.mdx) command options with YAML configuration files. Individual seqerakit commands and configuration files can be chained together to automate the creation of all Seqera Platform entities end-to-end. -The key features are: +### Key features + +As an extension of the Platform CLI, seqerakit enables: - **Simple configuration**: All of the command-line options available in the Platform CLI can be defined in simple YAML format. - **Infrastructure as code**: Enable users to manage and provision their infrastructure. - **Automation**: End-to-end creation of Seqera Platform entities, from adding an organization to launching pipelines. -## Installation - -`seqerakit` has three dependencies: - -1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli/releases) -2. [Python (`>=3.8`)](https://www.python.org/downloads/) -3. [PyYAML](https://pypi.org/project/PyYAML/) - -### Conda - -To install `seqerakit` and its dependencies via Conda, first configure the correct channels: - -```console -conda config --add channels bioconda -conda config --add channels conda-forge -conda config --set channel_priority strict -``` - -Then create a conda environment with `seqerakit` installed: - -``` -conda env create -n seqerakit seqerakit -conda activate seqerakit -``` - -### Pip - -If you already have [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): - -```console -pip install seqerakit -``` - -Overwrite an existing installation to use the latest version: - -```console -pip install --upgrade --force-reinstall seqerakit -``` - -### Local development installation - -Install the development branch of `seqerakit` on your local machine to test the latest features and updates: - -1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. -1. To install directly from pip: - ```bash - pip install git+https://github.com/seqeralabs/seqera-kit.git@dev - ``` -1. Alternatively, clone the repository locally and install manually: - ```bash - git clone https://github.com/seqeralabs/seqera-kit.git - cd seqera-kit - git checkout dev - pip install . - ``` -1. Verify your installation: - ```bash - pip show seqerakit - ``` - -## Configuration - -Create a Seqera Platform access token using the [Seqera Platform](https://seqera.io/) web interface via the **Your Tokens** page in your profile. - -`seqerakit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below: - -```bash -export TOWER_ACCESS_TOKEN= -``` - -For Enterprise installations of Seqera Platform, you will also need to configure the API endpoint that will be used to connect to the Platform. You can do so by exporting the following environment variable: -```bash -export TOWER_API_ENDPOINT= -``` -By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud. - - -## Usage - -To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to the Platform is working as expected. This will run the `tw info` command under the hood: - -```bash -seqerakit --info -``` - -Use the `--help` or `-h` parameter to list the available commands and their associated options: -```bash -seqerakit --help -``` - -Use `--version` or `-v` to retrieve the current version of your seqerakit installation: -```bash -seqerakit --version -``` -### Input -`seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin). - -#### Using File Path - -```bash -seqerakit /path/to/file.yaml -``` -#### Using stdin - -```console -$ cat file.yaml | seqerakit - -``` -See the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s). - -### Dryrun - -To print the commands that would executed with `tw` when using a YAML file, you can run `seqerakit` with the `--dryrun` flag: - -```bash -seqerakit file.yaml --dryrun -``` - -### Recursively delete - -Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag: - -```bash -seqerakit file.yaml --delete -``` - -For example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `seqerakit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization. - -### Using `tw` specific CLI options - -`tw` specific CLI options can be specified with the `--cli=` flag: - -```bash -seqerakit file.yaml --cli="--arg1 --arg2" -``` - -You can find the full list of options by running `tw -h`. - -The Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag. - -To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes. - -For example: - -```bash -seqerakit file.yaml --cli="--insecure" -``` - -For Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `seqerakit` as you would to `tw`, preceded by `--cli=`. - -For example: - -```bash -seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts" -``` - -Note: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error. - -## Specify targets -When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names. - -For example, given a YAML file that defines the following resources: - -```yaml -workspaces: - - name: 'showcase' - organization: 'seqerakit_automation' -... -compute-envs: - - name: 'compute-env' - type: 'aws-batch forge' - workspace: 'seqerakit/test' -... -pipelines: - - name: "hello-world-test-seqerakit" - url: "https://github.com/nextflow-io/hello" - workspace: 'seqerakit/test' - compute-env: "compute-env" -... -``` - -You can target the creation of `pipelines` only by running: - -```bash -seqerakit test.yml --targets pipelines -``` -This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`. - -### Multiple Targets -You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run: - -```bash -seqerakit test.yml --targets workspaces,pipelines -``` - -## YAML Configuration Options - -There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. - -### 1. Pipeline parameters using `params` and `params-file` - -To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file. - -For example, to specify pipeline parameters within your YAML: - -```yaml -params: - outdir: 's3://path/to/outdir' - fasta: 's3://path/to/reference.fasta' -``` - -Alternatively, to specify a file containing pipeline parameters: - -```yaml -params-file: '/path/to/my/parameters.yaml' -``` - -Optionally, you may provide both: - -```yaml -params-file: '/path/to/my/parameters.yaml' -params: - outdir: 's3://path/to/outdir' - fasta: 's3://path/to/reference.fasta' -``` - -**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. - -### 2. `overwrite` Functionality -For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. - -`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. - -```console -DEBUG:root: Overwrite is set to 'True' for organizations - -DEBUG:root: Running command: tw -o json organizations list -DEBUG:root: The attempted organizations resource already exists. Overwriting. - -DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME -DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' -``` -### 3. Specifying JSON configuration files with `file-path` -The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. - -An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: - -```yaml -compute-envs: - - name: 'my_aws_compute_environment' # required - workspace: 'my_organization/my_workspace' # required - credentials: 'my_aws_credentials' # required - wait: 'AVAILABLE' # optional - file-path: './compute-envs/my_aws_compute_environment.json' # required - overwrite: True -``` - - -## Quick start - -You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. - -You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline. - -### Launch via YAML - -1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `` and `` entries as required: - - ```yaml # noqa - launch: - - name: 'hello-world' # Workflow name - workspace: '' # Workspace name - compute-env: '' # Compute environment - revision: 'master' # Pipeline revision - pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL - ``` - -2. Launch the pipeline with `seqerakit`: - - ```bash - seqerakit hello-world-config.yml - ``` - -3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! - -### Launch via a Python script - -You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python. - -1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `` and `` entries as required. - -2. Launch the pipeline with `seqerakit`: - -```bash - python launch_hello_world.py -``` - -3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! - -## Defining your YAML file using CLI options - -All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity. - -1. Retrieve CLI Options - -To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: - -```console -$ tw pipelines add -h - -Usage: tw pipelines add [OPTIONS] PIPELINE_URL - -Add a workspace pipeline. - -Parameters: -* PIPELINE_URL Nextflow pipeline URL. - -Options: -* -n, --name= Pipeline name. - -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName - -d, --description= Pipeline description. - --labels=[,...] List of labels seperated by coma. - -c, --compute-env= Compute environment name. - --work-dir= Path where the pipeline scratch data is stored. - -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. - --params-file= Pipeline parameters in either JSON or YML format. - --revision= A valid repository commit Id, tag or branch name. - ... -``` -2. Define Key-Value Pairs in YAML - -Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: - -```yaml -pipelines: - - name: 'my_first_pipeline' - url: 'https://github.com/username/my_pipeline' - workspace: 'my_organization/my_workspace' - description: 'My test pipeline' - labels: 'yeast,test_data' - compute-env: 'my_compute_environment' - work-dir: 's3://my_bucket' - profile: 'test' - params-file: '/path/to/params.yaml' - revision: '1.0' -``` - -In this example: - -- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. -- The corresponding values are user-defined - -### Best Practices: -- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. -- Use quotes around strings that contain special characters or spaces. -- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. -- For complex configurations, refer to the [Templates](./templates/) provided in this repository. - - -## Templates - -We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: - -- [organizations.yml](./templates/organizations.yml) -- [teams.yml](./templates/teams.yml) -- [workspaces.yml](./templates/workspaces.yml) -- [participants.yml](./templates/participants.yml) -- [credentials.yml](./templates/credentials.yml) -- [secrets.yml](./templates/secrets.yml) -- [compute-envs.yml](./templates/compute-envs.yml) -- [actions.yml](./templates/actions.yml) -- [datasets.yml](./templates/datasets.yml) -- [labels.yml](./templates/labels.yml) -- [pipelines.yml](./templates/pipelines.yml) -- [launch.yml](./templates/launch.yml) - -## Real world example - -Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline. - -You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables: - -```console -$TOWER_GITHUB_PASSWORD -$DOCKERHUB_PASSWORD -$AWS_ACCESS_KEY_ID -$AWS_SECRET_ACCESS_KEY -$AWS_ASSUME_ROLE_ARN -$AZURE_BATCH_KEY -$AZURE_STORAGE_KEY -$GOOGLE_KEY -$SENTIEON_LICENSE_BASE64 -``` - -## Contributions and Support - -If you would like to contribute to `seqerakit`, please see the [contributing guidelines](./.github/CONTRIBUTING.md). +### Availability -For further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/seqera-kit/issues) in this repository. +seqerakit can be installed on macOS, Windows, and Linux. It is compatible with [Seqera Platform Cloud](https://cloud.seqera.io/) and Enterprise versions 21.08 and later. -## Credits +See [Installation](./installation.mdx) to get started. -`seqerakit` was written by [Esha Joshi](https://github.com/ejseqera), [Adam Talbot](https://github.com/adamrtalbot) and [Harshil Patel](https://github.com/drpatelh) from the Scientific Development Team at [Seqera Labs](https://seqera.io/). +:::info +You can also invoke and run `seqerakit` via a [Python script](https://github.com/seqeralabs/seqera-kit#launch-via-a-python-script). +::: diff --git a/platform_versioned_sidebars/version-24.1-sidebars.json b/platform_versioned_sidebars/version-24.1-sidebars.json index 8bf36e60..2d862dee 100644 --- a/platform_versioned_sidebars/version-24.1-sidebars.json +++ b/platform_versioned_sidebars/version-24.1-sidebars.json @@ -189,12 +189,7 @@ "type": "category", "label": "Developer tools", "collapsed": true, - "items": [ - { - "type": "doc", - "id": "getting-started/quickstart-demo/automation", - "label": "Overview" - }, + "items": [ "api/overview", { "type": "category", @@ -206,7 +201,17 @@ "cli/commands" ] }, - "pipeline-actions/overview" + { + "type": "category", + "label": "seqerakit", + "collapsed": true, + "items": [ + "seqerakit/overview", + "seqerakit/installation", + "seqerakit/commands" + ] + }, + "pipeline-actions/overview" ] }, { From 70d9dcfb17720320178091d6914e79833bc3a9fa Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Tue, 22 Oct 2024 13:27:20 +0200 Subject: [PATCH 03/15] Add quickstart and YML pages --- .../version-24.1/seqerakit/commands.mdx | 264 ++---------------- .../version-24.1/seqerakit/overview.mdx | 6 +- .../version-24.1/seqerakit/quickstart.mdx | 46 +++ .../seqerakit/yaml-configuration.mdx | 166 +++++++++++ 4 files changed, 246 insertions(+), 236 deletions(-) create mode 100644 platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx create mode 100644 platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index 2f61fccd..7c3c3280 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -5,7 +5,7 @@ date: "21 Oct 2024" tags: [seqerakit, cli, automation, commands] --- -Use the `--help` or `-h` parameter to list the available commands and their associated options: +Use the `--help` or `-h` parameter to list available commands and options: ```bash seqerakit --help @@ -13,7 +13,7 @@ seqerakit --help ### Input -`seqerakit` supports input through paths to YAML configuration foiels or directly from standard input (stdin). +`seqerakit` supports input through paths to YAML configuration files or directly from standard input (stdin). - Using file path: @@ -31,71 +31,32 @@ See [Defining your YAML file using CLI options](#defining-your-yaml-file-using-c ### Dryrun -It is recommended to confirm that your configuration and command are correct before creating resources in your Platform account, particularly when automating end-to-end creation of multiple Platform entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your command with the `--dryrun` flag: +It is recommended to confirm that your configuration and command are correct before creating resources in your Platform account, particularly when automating the end-to-end creation of multiple Platform entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your seqerakit command with the `--dryrun` flag: ```bash seqerakit file.yaml --dryrun ``` -### Delete resources - -Instead of adding or creating resources, you can recursively delete resources in your YAML file by specifying the `--delete` flag: - -```bash -seqerakit file.yaml --delete -``` - -For example, if you have a YAML file that defines an Organization -> Workspace -> Team -> Credentials -> Compute Environment that have already been created, with the `--delete` flag, `seqerakit` will recursively delete the Compute Environment -> Credentials -> Team -> Workspace -> Organization. - -### Using `tw` specific CLI options - -`tw` specific CLI options can be specified with the `--cli=` flag: - -```bash -seqerakit file.yaml --cli="--arg1 --arg2" -``` +### Specify targets -See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. - -The Seqera Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Seqera Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag. - -To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes. - -For example: +When using a YAML file as input that defines multiple resources, use the `--targets` flag to specify which resources to create. This flag accepts a comma-separated list of resource names. -```bash -seqerakit file.yaml --cli="--insecure" -``` - -For Seqera Platform Enterprise, to use an SSL certificate that is not accepted by the default Java certificate authorities and specify a custom `cacerts` store as accepted by the `tw` CLI, you can specify the `-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts` option enclosed in double quotes to `seqerakit` as you would to `tw`, preceded by `--cli=`. - -For example: - -```bash -seqerakit hello-world-config.yml --cli="-Djavax.net.ssl.trustStore=/absolute/path/to/cacerts" -``` - -Note: Use of `--verbose` option for the `tw` CLI is currently not supported by `seqerakit`. Supplying `--cli="--verbose"` will raise an error. - -## Specify targets -When using a YAML file as input that defines multiple resources, you can use the `--targets` flag to specify which resources to create. This flag takes a comma-separated list of resource names. - -For example, given a YAML file that defines the following resources: +For example, given a `test.yml` file that defines the following resources: ```yaml workspaces: - - name: 'showcase' - organization: 'seqerakit_automation' + - name: 'workspace-1' + organization: 'seqerakit' ... compute-envs: - name: 'compute-env' type: 'aws-batch forge' - workspace: 'seqerakit/test' + workspace: 'seqerakit/workspace-1' ... pipelines: - - name: "hello-world-test-seqerakit" + - name: "hello-world" url: "https://github.com/nextflow-io/hello" - workspace: 'seqerakit/test' + workspace: 'seqerakit/workspace-1' compute-env: "compute-env" ... ``` @@ -105,210 +66,47 @@ You can target the creation of `pipelines` only by running: ```bash seqerakit test.yml --targets pipelines ``` + This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`. -### Multiple Targets -You can also specify multiple resources to create by separating them with commas. For example, to create both workspaces and pipelines, run: +To create both workspaces and pipelines, run: ```bash seqerakit test.yml --targets workspaces,pipelines ``` -## YAML Configuration Options - -There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. - -### 1. Pipeline parameters using `params` and `params-file` - -To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file. - -For example, to specify pipeline parameters within your YAML: - -```yaml -params: - outdir: 's3://path/to/outdir' - fasta: 's3://path/to/reference.fasta' -``` - -Alternatively, to specify a file containing pipeline parameters: - -```yaml -params-file: '/path/to/my/parameters.yaml' -``` - -Optionally, you may provide both: - -```yaml -params-file: '/path/to/my/parameters.yaml' -params: - outdir: 's3://path/to/outdir' - fasta: 's3://path/to/reference.fasta' -``` - -**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. - -### 2. `overwrite` Functionality -For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. - -`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. - -```console -DEBUG:root: Overwrite is set to 'True' for organizations - -DEBUG:root: Running command: tw -o json organizations list -DEBUG:root: The attempted organizations resource already exists. Overwriting. - -DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME -DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' -``` -### 3. Specifying JSON configuration files with `file-path` -The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. +### Delete resources -An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: +Instead of adding or creating resources, recursively delete resources in your YAML file by specifying the `--delete` flag: -```yaml -compute-envs: - - name: 'my_aws_compute_environment' # required - workspace: 'my_organization/my_workspace' # required - credentials: 'my_aws_credentials' # required - wait: 'AVAILABLE' # optional - file-path: './compute-envs/my_aws_compute_environment.json' # required - overwrite: True +```bash +seqerakit file.yml --delete ``` +For example, if you have a `file.yml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yml --delete` to recursively delete the same resources. -## Quick start - -You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. - -You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline. +### Using `tw`-specific CLI options -### Launch via YAML - -1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `` and `` entries as required: - - ```yaml # noqa - launch: - - name: 'hello-world' # Workflow name - workspace: '' # Workspace name - compute-env: '' # Compute environment - revision: 'master' # Pipeline revision - pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL - ``` - -2. Launch the pipeline with `seqerakit`: - - ```bash - seqerakit hello-world-config.yml - ``` - -3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! - -### Launch via a Python script - -You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python. - -1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `` and `` entries as required. - -2. Launch the pipeline with `seqerakit`: +`tw`-specific CLI options can be specified with the `--cli=` flag: ```bash - python launch_hello_world.py +seqerakit file.yaml --cli="--arg1 --arg2" ``` -3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! - -## Defining your YAML file using CLI options - -All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity. - -1. Retrieve CLI Options - -To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: - -```console -$ tw pipelines add -h +See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. -Usage: tw pipelines add [OPTIONS] PIPELINE_URL +:::note +The `--verbose` option for `tw` CLI is currently not supported by `seqerakit`. +::: -Add a workspace pipeline. +#### Example: HTTP-only connections -Parameters: -* PIPELINE_URL Nextflow pipeline URL. +The Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag. -Options: -* -n, --name= Pipeline name. - -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName - -d, --description= Pipeline description. - --labels=[,...] List of labels seperated by coma. - -c, --compute-env= Compute environment name. - --work-dir= Path where the pipeline scratch data is stored. - -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. - --params-file= Pipeline parameters in either JSON or YML format. - --revision= A valid repository commit Id, tag or branch name. - ... -``` -2. Define Key-Value Pairs in YAML +To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes. -Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: +For example: -```yaml -pipelines: - - name: 'my_first_pipeline' - url: 'https://github.com/username/my_pipeline' - workspace: 'my_organization/my_workspace' - description: 'My test pipeline' - labels: 'yeast,test_data' - compute-env: 'my_compute_environment' - work-dir: 's3://my_bucket' - profile: 'test' - params-file: '/path/to/params.yaml' - revision: '1.0' +```bash +seqerakit file.yaml --cli="--insecure" ``` - -In this example: - -- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. -- The corresponding values are user-defined - -### Best Practices: -- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. -- Use quotes around strings that contain special characters or spaces. -- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. -- For complex configurations, refer to the [Templates](./templates/) provided in this repository. - - -## Templates - -We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: - -- [organizations.yml](./templates/organizations.yml) -- [teams.yml](./templates/teams.yml) -- [workspaces.yml](./templates/workspaces.yml) -- [participants.yml](./templates/participants.yml) -- [credentials.yml](./templates/credentials.yml) -- [secrets.yml](./templates/secrets.yml) -- [compute-envs.yml](./templates/compute-envs.yml) -- [actions.yml](./templates/actions.yml) -- [datasets.yml](./templates/datasets.yml) -- [labels.yml](./templates/labels.yml) -- [pipelines.yml](./templates/pipelines.yml) -- [launch.yml](./templates/launch.yml) - -## Real world example - -Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline. - -You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables: - -```console -$TOWER_GITHUB_PASSWORD -$DOCKERHUB_PASSWORD -$AWS_ACCESS_KEY_ID -$AWS_SECRET_ACCESS_KEY -$AWS_ASSUME_ROLE_ARN -$AZURE_BATCH_KEY -$AZURE_STORAGE_KEY -$GOOGLE_KEY -$SENTIEON_LICENSE_BASE64 -``` \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx index ccbc08bf..e5474126 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx @@ -5,14 +5,14 @@ date: "10 Oct 2024" tags: [seqerakit, cli, automation] --- -`seqerakit` is a Python wrapper that allows you to set [Platform CLI](../cli/overview.mdx) command options with YAML configuration files. Individual seqerakit commands and configuration files can be chained together to automate the creation of all Seqera Platform entities end-to-end. +`seqerakit` is a Python wrapper that allows you to set [Platform CLI](../cli/overview.mdx) command options with YAML configuration files. Individual seqerakit commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. ### Key features As an extension of the Platform CLI, seqerakit enables: -- **Simple configuration**: All of the command-line options available in the Platform CLI can be defined in simple YAML format. -- **Infrastructure as code**: Enable users to manage and provision their infrastructure. +- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. +- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. - **Automation**: End-to-end creation of Seqera Platform entities, from adding an organization to launching pipelines. ### Availability diff --git a/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx b/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx new file mode 100644 index 00000000..6fac88c0 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx @@ -0,0 +1,46 @@ +--- +title: "Quickstart" +description: "seqerakit Hello World pipeline example" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, commands, hello world] +--- + +You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. + +You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline. + +### Launch via YAML + +1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `` and `` entries as required: + + ```yaml + launch: + - name: 'hello-world' # Workflow name + workspace: '' # Workspace name + compute-env: '' # Compute environment + revision: 'master' # Pipeline revision + pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL + ``` + +2. Launch the pipeline with `seqerakit`: + + ```bash + seqerakit hello-world-config.yml + ``` + +3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! + +### Launch via a Python script + +You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python. + +1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `` and `` entries as required. + +2. Launch the pipeline with `seqerakit`: + +```bash + python launch_hello_world.py +``` + +3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! + diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx new file mode 100644 index 00000000..8fc96699 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -0,0 +1,166 @@ +--- +title: "YAML configuration" +description: "seqerakit YAML configuration file options" +date: "21 Oct 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +--- + +## Defining your YAML file using CLI options + +All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity. + +1. Retrieve CLI Options + +To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: + +```console +$ tw pipelines add -h + +Usage: tw pipelines add [OPTIONS] PIPELINE_URL + +Add a workspace pipeline. + +Parameters: +* PIPELINE_URL Nextflow pipeline URL. + +Options: +* -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... +``` +2. Define Key-Value Pairs in YAML + +Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: + +```yaml +pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' +``` + +In this example: + +- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. +- The corresponding values are user-defined + +### Best Practices: +- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. +- Use quotes around strings that contain special characters or spaces. +- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. +- For complex configurations, refer to the [Templates](./templates/) provided in this repository. + + +## Templates + +We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: + +- [organizations.yml](./templates/organizations.yml) +- [teams.yml](./templates/teams.yml) +- [workspaces.yml](./templates/workspaces.yml) +- [participants.yml](./templates/participants.yml) +- [credentials.yml](./templates/credentials.yml) +- [secrets.yml](./templates/secrets.yml) +- [compute-envs.yml](./templates/compute-envs.yml) +- [actions.yml](./templates/actions.yml) +- [datasets.yml](./templates/datasets.yml) +- [labels.yml](./templates/labels.yml) +- [pipelines.yml](./templates/pipelines.yml) +- [launch.yml](./templates/launch.yml) + +## Real world example + +Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline. + +You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables: + +```console +$TOWER_GITHUB_PASSWORD +$DOCKERHUB_PASSWORD +$AWS_ACCESS_KEY_ID +$AWS_SECRET_ACCESS_KEY +$AWS_ASSUME_ROLE_ARN +$AZURE_BATCH_KEY +$AZURE_STORAGE_KEY +$GOOGLE_KEY +$SENTIEON_LICENSE_BASE64 +``` + +## YAML Configuration Options + +There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. + +### 1. Pipeline parameters using `params` and `params-file` + +To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file. + +For example, to specify pipeline parameters within your YAML: + +```yaml +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +Alternatively, to specify a file containing pipeline parameters: + +```yaml +params-file: '/path/to/my/parameters.yaml' +``` + +Optionally, you may provide both: + +```yaml +params-file: '/path/to/my/parameters.yaml' +params: + outdir: 's3://path/to/outdir' + fasta: 's3://path/to/reference.fasta' +``` + +**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. + +### 2. `overwrite` Functionality + +For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. + +`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. + +```console +DEBUG:root: Overwrite is set to 'True' for organizations + +DEBUG:root: Running command: tw -o json organizations list +DEBUG:root: The attempted organizations resource already exists. Overwriting. + +DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION_NAME +DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' +``` + +### 3. Specifying JSON configuration files with `file-path` + +The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. + +An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: + +```yaml +compute-envs: + - name: 'my_aws_compute_environment' # required + workspace: 'my_organization/my_workspace' # required + credentials: 'my_aws_credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/my_aws_compute_environment.json' # required + overwrite: True +``` From 8ccc850dc9406ba6047db5cf95dcb655da0e3af2 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Tue, 12 Nov 2024 15:42:03 +0200 Subject: [PATCH 04/15] Updates WIP --- .../version-24.1/seqerakit/commands.mdx | 6 +- .../version-24.1/seqerakit/installation.mdx | 14 +-- .../version-24.1/seqerakit/quickstart.mdx | 12 +-- .../version-24.1/seqerakit/templates.mdx | 6 ++ .../seqerakit/yaml-configuration.mdx | 89 +++++++++---------- 5 files changed, 66 insertions(+), 61 deletions(-) create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates.mdx diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index 7c3c3280..0d0ff9f3 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -31,7 +31,7 @@ See [Defining your YAML file using CLI options](#defining-your-yaml-file-using-c ### Dryrun -It is recommended to confirm that your configuration and command are correct before creating resources in your Platform account, particularly when automating the end-to-end creation of multiple Platform entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your seqerakit command with the `--dryrun` flag: +Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your seqerakit command with the `--dryrun` flag: ```bash seqerakit file.yaml --dryrun @@ -67,7 +67,7 @@ You can target the creation of `pipelines` only by running: seqerakit test.yml --targets pipelines ``` -This will process only the pipelines block from the YAML file and ignore other blocks such as `workspaces` and `compute-envs`. +This will process only the pipelines block from the YAML file and ignore `workspaces` and `compute-envs`. To create both workspaces and pipelines, run: @@ -77,7 +77,7 @@ seqerakit test.yml --targets workspaces,pipelines ### Delete resources -Instead of adding or creating resources, recursively delete resources in your YAML file by specifying the `--delete` flag: +Instead of adding or creating resources, specify the `--delete` flag to recursively delete resources in your YAML file: ```bash seqerakit file.yml --delete diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx index abafb366..c9e2f2f3 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -13,7 +13,7 @@ tags: [seqerakit, cli, automation, installation] ### Pip -If you already have [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli#1-installation) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): +If you already have [Seqera Platform CLI](../cli/installation.mdx) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): ```console pip install seqerakit @@ -65,7 +65,7 @@ Install the development branch of `seqerakit` on your local machine to test the ### Configuration -Create a Seqera Platform access token using the [Seqera Platform](https://cloud.seqera.io/) web interface via the **Your Tokens** page in your profile. +Create a [Seqera](https://cloud.seqera.io/) access token via the **Your Tokens** page in your profile. `seqerakit` reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: @@ -73,23 +73,23 @@ Create a Seqera Platform access token using the [Seqera Platform](https://cloud. export TOWER_ACCESS_TOKEN= ``` -For Platform Enterprise installations, you must specify the custom API endpoint used to connect to the Platform. Export the API endpoint environment variable: +For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: ```bash export TOWER_API_ENDPOINT= ``` -By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud. +By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cloud. ### Usage -To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to the Platform is working as expected. This will run the `tw info` command under the hood: +To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to Seqera is working as expected. This will run the `tw info` command under the hood: ```bash seqerakit --info ``` -Use `--version` or `-v` to retrieve the current version of your seqerakit installation: +Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: ```bash seqerakit --version @@ -101,4 +101,4 @@ Use the `--help` or `-h` parameter to list the available commands and their asso seqerakit --help ``` -See [Commands](./commands.mdx) for detailed instructions to use seqerakit. \ No newline at end of file +See [Commands](./commands.mdx) for detailed instructions to use `seqerakit`. \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx b/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx index 6fac88c0..5cb9347d 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx @@ -7,11 +7,11 @@ tags: [seqerakit, cli, automation, commands, hello world] You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. -You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline. +You will need a Seqera account (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a workspace and a pre-defined compute environment where you can launch a pipeline. ### Launch via YAML -1. Create a YAML file called `hello-world-config.yml` with the contents below, and customise the `` and `` entries as required: +1. Create a YAML file called `hello-world-config.yml` with the contents below, and customize the `` and `` entries as needed: ```yaml launch: @@ -28,13 +28,13 @@ You will need to have an account on Seqera Platform (see [Plans and pricing](htt seqerakit hello-world-config.yml ``` -3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! +3. Log in to your Seqera instance and check the **Runs** page in the appropriate workspace for the pipeline you just launched. ### Launch via a Python script -You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python. +You can also launch the same pipeline via a Python script. This allows you to extend Platform CLI functionality by leveraging the flexibility and customization options available in Python. -1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `` and `` entries as required. +1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customize the `` and `` entries as needed. 2. Launch the pipeline with `seqerakit`: @@ -42,5 +42,5 @@ You can also launch the same pipeline via a Python script. This will essentially python launch_hello_world.py ``` -3. Login to your Seqera Platform instance and check the Runs page in the appropriate Workspace for the pipeline you just launched! +3. Log in to your Seqera instance and check the **Runs** page in the appropriate workspace for the pipeline you just launched. diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx new file mode 100644 index 00000000..1be86406 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -0,0 +1,6 @@ +--- +title: "Templates" +description: "seqerakit YAML configuration file templates and instructions" +date: "8 Nov 2024" +tags: [seqerakit, cli, automation, yaml, configuration] +--- \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index 8fc96699..7155005c 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -5,64 +5,63 @@ date: "21 Oct 2024" tags: [seqerakit, cli, automation, yaml, configuration] --- -## Defining your YAML file using CLI options - -All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for your desired entity. +All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for the resource you want to create. 1. Retrieve CLI Options -To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: + To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: -```console -$ tw pipelines add -h - -Usage: tw pipelines add [OPTIONS] PIPELINE_URL - -Add a workspace pipeline. - -Parameters: -* PIPELINE_URL Nextflow pipeline URL. - -Options: -* -n, --name= Pipeline name. - -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName - -d, --description= Pipeline description. - --labels=[,...] List of labels seperated by coma. - -c, --compute-env= Compute environment name. - --work-dir= Path where the pipeline scratch data is stored. - -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. - --params-file= Pipeline parameters in either JSON or YML format. - --revision= A valid repository commit Id, tag or branch name. - ... -``` -2. Define Key-Value Pairs in YAML + ```console + $ tw pipelines add -h -Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: + Usage: tw pipelines add [OPTIONS] PIPELINE_URL -```yaml -pipelines: - - name: 'my_first_pipeline' - url: 'https://github.com/username/my_pipeline' - workspace: 'my_organization/my_workspace' - description: 'My test pipeline' - labels: 'yeast,test_data' - compute-env: 'my_compute_environment' - work-dir: 's3://my_bucket' - profile: 'test' - params-file: '/path/to/params.yaml' - revision: '1.0' -``` + Add a workspace pipeline. + + Parameters: + * PIPELINE_URL Nextflow pipeline URL. + + Options: + * -n, --name= Pipeline name. + -w, --workspace= Workspace numeric identifier (TOWER_WORKSPACE_ID as default) or workspace reference as OrganizationName/WorkspaceName + -d, --description= Pipeline description. + --labels=[,...] List of labels seperated by coma. + -c, --compute-env= Compute environment name. + --work-dir= Path where the pipeline scratch data is stored. + -p, --profile=[,...] Comma-separated list of one or more configuration profile names you want to use for this pipeline execution. + --params-file= Pipeline parameters in either JSON or YML format. + --revision= A valid repository commit Id, tag or branch name. + ... + ``` + +1. Define Key-Value Pairs in YAML + + Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: + + ```yaml + pipelines: + - name: 'my_first_pipeline' + url: 'https://github.com/username/my_pipeline' + workspace: 'my_organization/my_workspace' + description: 'My test pipeline' + labels: 'yeast,test_data' + compute-env: 'my_compute_environment' + work-dir: 's3://my_bucket' + profile: 'test' + params-file: '/path/to/params.yaml' + revision: '1.0' + ``` -In this example: + In this example: -- `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. -- The corresponding values are user-defined + - `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. + - The corresponding values are user-defined ### Best Practices: - Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. - Use quotes around strings that contain special characters or spaces. - When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. -- For complex configurations, refer to the [Templates](./templates/) provided in this repository. +- For complex configurations, see [Templates](./templates.mdx). ## Templates From 4ad8624c71fbc73a26ab8a34f629dcf68060da2d Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Fri, 15 Nov 2024 14:23:05 +0200 Subject: [PATCH 05/15] Templates updates --- .../version-24.1/seqerakit/commands.mdx | 12 +- .../version-24.1/seqerakit/installation.mdx | 27 ++- .../version-24.1/seqerakit/templates.mdx | 171 +++++++++++++++++- .../seqerakit/templates/actions.yml | 2 +- .../seqerakit/templates/compute-envs.yml | 14 +- .../seqerakit/templates/credentials.yml | 4 +- .../seqerakit/templates/datasets.yml | 2 +- .../seqerakit/templates/labels.yml | 2 +- .../seqerakit/templates/launch.yml | 6 +- .../seqerakit/templates/members.yml | 4 +- .../seqerakit/templates/organizations.yml | 2 +- .../seqerakit/templates/participants.yml | 2 +- .../seqerakit/templates/pipelines.yml | 8 +- .../seqerakit/templates/secrets.yml | 2 +- .../seqerakit/templates/teams.yml | 2 +- .../seqerakit/templates/workspaces.yml | 2 +- .../seqerakit/yaml-configuration.mdx | 37 ++-- .../version-24.1-sidebars.json | 5 +- 18 files changed, 236 insertions(+), 68 deletions(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index 0d0ff9f3..8116f0e3 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -27,7 +27,7 @@ seqerakit --help $ cat file.yaml | seqerakit - ``` -See [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. +See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. ### Dryrun @@ -85,9 +85,9 @@ seqerakit file.yml --delete For example, if you have a `file.yml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yml --delete` to recursively delete the same resources. -### Using `tw`-specific CLI options +### Use `tw`-specific CLI options -`tw`-specific CLI options can be specified with the `--cli=` flag: +Specify `tw`-specific CLI options with the `--cli=` flag: ```bash seqerakit file.yaml --cli="--arg1 --arg2" @@ -101,11 +101,9 @@ The `--verbose` option for `tw` CLI is currently not supported by `seqerakit`. #### Example: HTTP-only connections -The Platform CLI expects to connect to a Seqera Platform instance that is secured by a TLS certificate. If your Platform Enterprise instance does not present a certificate, you will need to qualify and run your `tw` commands with the `--insecure` flag. +The Platform CLI expects to connect to a Seqera instance that is secured by a TLS certificate. If your Seqera Enterprise instance does not present a certificate, you must run your `tw` commands with the `--insecure` flag. -To use `tw` specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options you would like to use enclosed in double quotes. - -For example: +To use `tw`-specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options to use enclosed in double quotes: ```bash seqerakit file.yaml --cli="--insecure" diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx index c9e2f2f3..0af1503a 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -5,13 +5,24 @@ date: "21 Oct 2024" tags: [seqerakit, cli, automation, installation] --- + +`seqerakit` is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual seqerakit commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. + +As an extension of the Platform CLI, seqerakit enables: + +- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. +- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. +- **Automation**: End-to-end creation of Seqera entities, from adding an organization to launching pipelines. + +### Installation + `seqerakit` has three dependencies: 1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli/releases) 2. [Python (`>=3.8`)](https://www.python.org/downloads/) 3. [PyYAML](https://pypi.org/project/PyYAML/) -### Pip +#### Pip If you already have [Seqera Platform CLI](../cli/installation.mdx) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): @@ -25,7 +36,7 @@ Overwrite an existing installation to use the latest version: pip install --upgrade --force-reinstall seqerakit ``` -### Conda +#### Conda To install `seqerakit` and its dependencies via Conda, first configure the correct channels: @@ -37,7 +48,7 @@ conda config --set channel_priority strict Then create a conda environment with `seqerakit` installed: -``` +```console conda env create -n seqerakit seqerakit conda activate seqerakit ``` @@ -65,30 +76,32 @@ Install the development branch of `seqerakit` on your local machine to test the ### Configuration -Create a [Seqera](https://cloud.seqera.io/) access token via the **Your Tokens** page in your profile. +Create a [Seqera](https://cloud.seqera.io/) access token via **Your Tokens** in the user menu. `seqerakit` reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: ```bash -export TOWER_ACCESS_TOKEN= +export TOWER_ACCESS_TOKEN= ``` For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: ```bash -export TOWER_API_ENDPOINT= +export TOWER_API_ENDPOINT= ``` By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cloud. ### Usage -To confirm the installation of `seqerakit`, configuration of the Seqera Platform CLI and connection to Seqera is working as expected. This will run the `tw info` command under the hood: +To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected run this command: ```bash seqerakit --info ``` +This runs the `tw info` command under the hood. + Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: ```bash diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx index 1be86406..5a23adc7 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -3,4 +3,173 @@ title: "Templates" description: "seqerakit YAML configuration file templates and instructions" date: "8 Nov 2024" tags: [seqerakit, cli, automation, yaml, configuration] ---- \ No newline at end of file +toc_max_heading_level: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import Organizations from '!!raw-loader!./templates/organizations.yml'; +import Members from '!!raw-loader!./templates/members.yml'; +import Workspaces from '!!raw-loader!./templates/workspaces.yml'; +import Teams from '!!raw-loader!./templates/teams.yml'; +import Participants from '!!raw-loader!./templates/participants.yml'; +import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yml'; +import Pipelines from '!!raw-loader!./templates/pipelines.yml'; +import Credentials from '!!raw-loader!./templates/credentials.yml'; +import Datasets from '!!raw-loader!./templates/datasets.yml'; +import Labels from '!!raw-loader!./templates/labels.yml'; +import Launch from '!!raw-loader!./templates/launch.yml'; +import Secrets from '!!raw-loader!./templates/secrets.yml'; +import Actions from '!!raw-loader!./templates/actions.yml'; + +Customize YAML configuration templates to use in seqerakit commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single seqerakit command by combining them into a single configuration file. + +To use the templates on this page: +1. Copy the template text or download the YML files you need. +1. Edit the values to specify your resource details, and save as a YML file. +1. Specify the YML template file in your seqerakit commands: + - To create the resources specified in the file: + ```console + seqerakit /path/to/file.yml + ``` + - To delete the existing resources specified in the file: + ```console + seqerakit /path/to/file.yml --delete + ``` + +:::info +See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. +::: + +### Administration + +Manage organizations, organization members, workspaces, teams, and participants. + +#### Organizations + +Add or delete organizations. + + +{Organizations} + + +[Download organizations.yml](./templates/organizations.yml) + +#### Members + +Add or delete organization members. + + +{Members} + + +[Download members.yml](./templates/members.yml) + +#### Workspaces + +Add or delete workspaces. + + +{Workspaces} + + +[Download workspaces.yml](./templates/workspaces.yml) + +#### Teams + +Add or delete teams. + + +{Teams} + + +[Download teams.yml](./templates/teams.yml) + +#### Participants + +Add or delete participants in workspaces and teams. + + +{Participants} + + +[Download participants.yml](./templates/participants.yml) + +### Credentials + +Add or delete compute environment, Git, and container registry credentials in workspaces. + + +{Credentials} + + +[Download credentials.yml](./templates/credentials.yml) + +### Compute environments + +Add or delete compute environments. + + +{ComputeEnvironments} + + +[Download compute-envs.yml](./templates/compute-envs.yml) + +### Pipelines + +Add or delete pipelines in workspace Launchpads. + + +{Pipelines} + + +[Download pipelines.yml](./templates/pipelines.yml) + +### Launch + +Launch a Nextflow pipeline. + + +{Launch} + + +[Download launch.yml](./templates/launch.yml) + +### Datasets + +Add or delete workspace datasets for pipeline input data. + + +{Datasets} + + +[Download datasets.yml](./templates/datasets.yml) + +### Labels + +Add or delete labels and resource labels to apply to workspace compute environments, pipelines, and runs. + + +{Labels} + + +[Download labels.yml](./templates/labels.yml) + +### Secrets + +Add or delete user and workspace secrets. + + +{Secrets} + + +[Download secrets.yml](./templates/secrets.yml) + +### Actions + +Add or delete pipeline actions. + + +{Actions} + + +[Download actions.yml](./templates/actions.yml) \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml index d243b9a8..aaa8ec75 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw actions add" +## To see the full list of options available, run: "tw actions add -h" actions: - type: 'github' # required name: 'my_github_action' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml index b6bf1710..31b77645 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml @@ -1,7 +1,7 @@ -## To see the full list of options available run: "tw compute-envs add" +## To see the full list of options available, run: "tw compute-envs add -h" ## The options required to create compute environments can be specified: ## 1. Explicitly in this file -## 2. Via a JSON file exported from Seqera Platform with the "tw compute-envs export" command +## 2. Via a JSON file exported from Seqera with the "tw compute-envs export" command compute-envs: # To create a compute environment from a JSON configuration file (AWS Example) - name: 'my_aws_compute_environment' # required @@ -24,13 +24,13 @@ compute-envs: wave: False # optional fargate: False # optional fast-storage: False # optional - instance-types: 'c6i,r6i,m6i' # optional, comma separated list + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list no-ebs-auto-scale: True # optional max-cpus: 500 # required - labels: 'label1,label2' # optional, comma separated list + labels: 'label1,label2' # optional, comma-separated list vpc-id: 'vpc-1234567890' # optional - subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma separated list - security-groups: 'sg-1234567890,sg-1234567891' # optional, comma separated list - allow-buckets: 's3://my-bucket,s3://my-other-bucket' # optional, comma separated list + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://my-bucket,s3://my-other-bucket' # optional, comma-separated list wait: 'AVAILABLE' # optional overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml index b7fef41d..445f60b3 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml @@ -1,6 +1,6 @@ -## To see the full list of options available run: "tw credentials add" +## To see the full list of options available, run: "tw credentials add -h" ## To avoid exposing sensitive information about your credentials, -## use environment variables to supply passwords and secret keys +## use environment variables to supply passwords and secret keys. credentials: - type: 'github' # required name: 'my_github_credentials' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml index dce22bd8..4542a771 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw datasets add" +## To see the full list of options available, run: "tw datasets add -h" datasets: - name: 'my_dataset' # required description: 'My test dataset' # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml index 6a75adbd..00d45e0e 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw labels add" +## To see the full list of options available, run: "tw labels add -h" labels: - name: 'label_name' # required value: 'label_value' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml index c281c398..0e7e2049 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml @@ -1,7 +1,7 @@ -## To see the full list of options available run: "tw launch" +## To see the full list of options available, run: "tw launch -h" ## The options will vary if you are launching a pipeline: -## 1. Pre-configured in the Seqera Platform Launchpad where most options have already been specified -## 2. From source via a remote Git URL where you will need to explicitly specify all launch options +## 1. Pre-configured in the Seqera Launchpad with most options already specified +## 2. From source via a remote Git repository URL where you must explicitly specify all launch options ## Note: overwrite is not supported for "tw launch" launch: diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml index d3f2d825..278c7dd4 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml @@ -1,5 +1,5 @@ -## To see the full list of options available run: "tw members add" -participants: +## To see the full list of options available, run: "tw members add -h" +members: - user: 'bob@myorg.io' # required organization: 'myorg' # required overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml index acd4d18d..30bda2d7 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw organizations add" +## To see the full list of options available, run: "tw organizations add -h" organizations: - name: 'my_organization' # required full-name: 'my_organization' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml index e0d69ccb..8eda72fa 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw participants add" +## To see the full list of options available, run: "tw participants add -h" participants: - name: 'my_team' # required type: 'TEAM' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml index 30882265..778ed95b 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml @@ -1,11 +1,11 @@ -## To see the full list of options available run: "tw pipelines add" +## To see the full list of options available, run: "tw pipelines add -h" ## The options required to create pipelines can be specified: ## 1. Explicitly in this file -## 2. Via a JSON file exported from SeqeraPlatform with the "tw pipelines export" command +## 2. Via a JSON file exported from Seqera with the "tw pipelines export" command pipelines: - name: 'my_first_pipeline' # required workspace: 'my_organization/my_workspace' # required - description: 'My test pipeline' # optional + description: 'Options specified in this file' # optional compute-env: 'my_aws_compute_environment' # required work-dir: 's3://my_bucket' # optional profile: 'test' # optional @@ -18,7 +18,7 @@ pipelines: overwrite: True # optional - name: 'my_second_pipeline' # required workspace: 'my_organization/my_workspace' # required - description: 'My test pipeline' # optional + description: 'Options specified via JSON file' # optional compute-env: 'my_aws_compute_environment' # required file-path: './pipelines/my_pipeline.json' # required overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml index b5cd79bb..6e04ff3e 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw secrets add" +## To see the full list of options available, run: "tw secrets add -h" secrets: - name: 'my_secret' # required workspace: 'my_organization/my_workspace' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml index e6f921b2..69dd951b 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw teams add" +## To see the full list of options available, run: "tw teams add -h" teams: - name: 'my_team' # required organization: 'my_organization' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml index c11cea33..ccf9f75b 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml @@ -1,4 +1,4 @@ -## To see the full list of options available run: "tw workspaces add" +## To see the full list of options available, run: "tw workspaces add -h" workspaces: - name: 'my_workspace' # required full-name: 'my_workspace' # required diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index 7155005c..869cbc64 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -5,11 +5,11 @@ date: "21 Oct 2024" tags: [seqerakit, cli, automation, yaml, configuration] --- -All available options to provide as definitions in your YAML file can be determined by running the Seqera Platform CLI help command for the resource you want to create. +To determine the options to provide as definitions in your YAML file, run the Platform CLI help command for the resource you want to create. -1. Retrieve CLI Options +1. Retrieve CLI options: - To obtain a list of available CLI options for defining your YAML file, use the help command of the Seqera Platform CLI. For instance, if you want to configure a pipeline to be added to the Launchpad, you can view the options as follows: + Obtain a list of available CLI options for defining your YAML file with the Platform CLI help command. For example, to add a pipeline to your workspace, view the options as follows: ```console $ tw pipelines add -h @@ -34,9 +34,9 @@ All available options to provide as definitions in your YAML file can be determi ... ``` -1. Define Key-Value Pairs in YAML +1. Define key-value pairs in YAML: - Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For instance: + Translate each CLI option into a key-value pair in the YAML file. The structure of your YAML file should reflect the hierarchy and format of the CLI options. For example: ```yaml pipelines: @@ -54,32 +54,19 @@ All available options to provide as definitions in your YAML file can be determi In this example: - - `name`, `url`, `workspace`, etc., are the keys derived from the CLI options. - - The corresponding values are user-defined + - The keys (`name`, `url`, `workspace`, and so forth) are the keys derived from the CLI options. + - The corresponding values are user-defined. -### Best Practices: -- Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting. +#### Best practices + +- The indentation and structure of the YAML file must be correct — YAML is sensitive to formatting. - Use quotes around strings that contain special characters or spaces. -- When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above. +- To list multiple values (such as multiple `labels`, `instance-types`, or `allow-buckets`), separate values with commas. This is shown with `labels` in the preceding example. - For complex configurations, see [Templates](./templates.mdx). - ## Templates -We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization: - -- [organizations.yml](./templates/organizations.yml) -- [teams.yml](./templates/teams.yml) -- [workspaces.yml](./templates/workspaces.yml) -- [participants.yml](./templates/participants.yml) -- [credentials.yml](./templates/credentials.yml) -- [secrets.yml](./templates/secrets.yml) -- [compute-envs.yml](./templates/compute-envs.yml) -- [actions.yml](./templates/actions.yml) -- [datasets.yml](./templates/datasets.yml) -- [labels.yml](./templates/labels.yml) -- [pipelines.yml](./templates/pipelines.yml) -- [launch.yml](./templates/launch.yml) +See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. ## Real world example diff --git a/platform_versioned_sidebars/version-24.1-sidebars.json b/platform_versioned_sidebars/version-24.1-sidebars.json index 0ce5d9f4..0d6f8fba 100644 --- a/platform_versioned_sidebars/version-24.1-sidebars.json +++ b/platform_versioned_sidebars/version-24.1-sidebars.json @@ -213,9 +213,10 @@ "label": "seqerakit", "collapsed": true, "items": [ - "seqerakit/overview", "seqerakit/installation", - "seqerakit/commands" + "seqerakit/commands", + "seqerakit/yaml-configuration", + "seqerakit/templates" ] }, "pipeline-actions/overview" From 71965f990d79a8722de435b2ac6bf4e0aab24eb9 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Sun, 17 Nov 2024 21:50:30 +0200 Subject: [PATCH 06/15] Clean YAML page, add e2e template --- .../version-24.1/seqerakit/templates.mdx | 23 +++ .../seqerakit/templates/seqerakit-e2e.yml | 184 ++++++++++++++++++ .../seqerakit/yaml-configuration.mdx | 22 +-- 3 files changed, 209 insertions(+), 20 deletions(-) create mode 100644 platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx index 5a23adc7..2332ac78 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -20,6 +20,7 @@ import Labels from '!!raw-loader!./templates/labels.yml'; import Launch from '!!raw-loader!./templates/launch.yml'; import Secrets from '!!raw-loader!./templates/secrets.yml'; import Actions from '!!raw-loader!./templates/actions.yml'; +import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yml'; Customize YAML configuration templates to use in seqerakit commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single seqerakit command by combining them into a single configuration file. @@ -40,6 +41,28 @@ To use the templates on this page: See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. ::: +### End-to-end example + +A template to create the follwing resources: + +- An organization +- A workspace +- A team +- Participants +- Credentials +- Secrets +- Compute environments +- Datasets +- Pipelines + +The templates also contains `launch` entries to launch saved pipelines. + + +{EndToEnd} + + +[Download seqerakit-e2e.yml](./templates/seqerakit-e2e.yml) + ### Administration Manage organizations, organization members, workspaces, teams, and participants. diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml new file mode 100644 index 00000000..d70e2d95 --- /dev/null +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml @@ -0,0 +1,184 @@ +organizations: + - name: 'your-organization' + full-name: 'Your organization LLC' + description: 'Organization created E2E with seqerakit CLI scripting' + location: 'Global' + website: 'https://yourdomain.com/' + overwrite: True +teams: + - name: 'pipelines_team' + organization: 'your-organization' + description: 'Pipelines team @ Your organization' + members: + - 'user1@yourdomain.com' + - 'user2@yourdomain.com' + - 'user3@yourdomain.com' + overwrite: True +workspaces: + - name: 'workspace-1' + full-name: 'Workspace one' + organization: 'your-organization' + description: 'Workspace created E2E with seqerakit CLI scripting' + visibility: 'PRIVATE' + overwrite: True +participants: + - name: 'pipelines_team' + type: 'TEAM' + workspace: 'your-organization/workspace-1' + role: 'ADMIN' + overwrite: True + - name: 'user4@domain.com@seqera.io' + type: 'MEMBER' + workspace: 'your-organization/workspace-1' + role: 'LAUNCH' + overwrite: True +credentials: + - type: 'github' + name: 'github_credentials' + workspace: 'your-organization/workspace-1' + username: 'user1' + password: '$TOWER_GITHUB_PASSWORD' + overwrite: True + - type: 'container-reg' + name: 'dockerhub_credentials' + workspace: 'your-organization/workspace-1' + username: 'user1' + password: '$DOCKERHUB_PASSWORD' + registry: 'docker.io' + overwrite: True + - type: 'aws' + name: 'aws_credentials' + workspace: 'your-organization/workspace-1' + access-key: '$AWS_ACCESS_KEY_ID' + secret-key: '$AWS_SECRET_ACCESS_KEY' + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' + overwrite: True + - type: 'azure' + name: 'azure_credentials' + workspace: 'your-organization/workspace-1' + batch-key: '$AZURE_BATCH_KEY' + batch-name: 'yourbatch' + storage-key: '$AZURE_STORAGE_KEY' + storage-name: 'yourstorage' + overwrite: True + - type: 'google' + name: 'google_credentials' + workspace: 'your-organization/workspace-1' + key: '$GOOGLE_KEY' + overwrite: True +secrets: + - name: 'SENTIEON_LICENSE_BASE64' + workspace: 'your-organization/workspace-1' + value: '$SENTIEON_LICENSE_BASE64' + overwrite: True +compute-envs: + - name: 'your-aws-compute-environment' + workspace: 'your-organization/workspace-1' + credentials: 'aws_credentials' + wait: 'AVAILABLE' + file-path: 'path/to/aws-credentials.json' + overwrite: True + - name: 'your-azure-compute-environment' + workspace: 'your-organization/workspace-1' + credentials: 'azure_credentials' + wait: 'AVAILABLE' + file-path: 'path/to/azure-credentials.json' + overwrite: True + - name: 'your-gcp-compute-environment' + workspace: 'your-organization/workspace-1' + credentials: 'google_credentials' + wait: 'AVAILABLE' + file-path: 'path/to/gcp-credentials.json' + overwrite: True +datasets: + - name: 'rnaseq_samples' + description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' + header: true + workspace: 'your-organization/workspace-1' + file-path: 'path/to/rnaseq_samples.csv' + overwrite: True + - name: 'sarek_samples' + description: 'Samplesheet to run the nf-core/sarek pipeline end-to-end' + header: true + workspace: 'your-organization/workspace-1' + file-path: 'path/to/sarek_samples.csv' + overwrite: True + - name: 'viralrecon_illumina_samples' + description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Illumina data' + header: true + workspace: 'your-organization/workspace-1' + file-path: 'path/to/viralrecon_illumina_samples.csv' + overwrite: True + - name: 'viralrecon_nanopore_samples' + description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Nanopore data' + header: true + workspace: 'your-organization/workspace-1' + file-path: 'path/to/viralrecon_nanopore_samples.csv' + overwrite: True + - name: 'sentieon_samples' + description: 'Samplesheet to run the seqeralabs/nf-sentieon pipeline from end-to-end' + header: true + workspace: 'your-organization/workspace-1' + file-path: 'path/to/sentieon_samples.csv' + overwrite: True +pipelines: + - name: 'nf-core-rnaseq' + url: 'https://github.com/nf-core/rnaseq' + workspace: 'your-organization/workspace-1' + description: 'RNA sequencing analysis pipeline using STAR, RSEM, HISAT2 or Salmon with gene/isoform counts and extensive quality control.' + compute-env: 'your-aws-compute-environment' + work-dir: 's3://your-bucket/pipeline/working/directory' + profile: 'test' + revision: '3.12.0' + params: + outdir: 's3://your-bucket/nf-core-rnaseq/results' + config: 'path/to/nextflow.config' + pre-run: 'path/to/pipeline/pre_run.txt' + overwrite: True + - name: 'nf-core-sarek' + workspace: 'your-organization/workspace-1' + compute-env: 'your-azure-compute-environment' + file-path: 'path/to/nf-core-sarek_pipeline.json' + overwrite: True + - name: 'nf-core-viralrecon-illumina' + url: 'https://github.com/nf-core/viralrecon' + workspace: 'your-organization/workspace-1' + description: 'Pipeline for assembly and intrahost/low-frequency variant calling for viral samples.' + compute-env: 'your-gcp-compute-environment' + work-dir: 'gs://your-bucket/pipelines/work/directory' + profile: 'test' + revision: '2.6.0' + params-file: 'path/to/pipelines/nf_core_viralrecon_illumina_params.yml' + config: 'path/to/pipelines/nextflow.config' + pre-run: 'path/to/pipelines/pre_run.txt' + overwrite: True + - name: 'nf-sentieon' + workspace: 'your-organization/workspace-1' + compute-env: 'your-aws-compute-environment' + file-path: 'path/to/pipelines/nf_sentieon_pipeline.json' + overwrite: True +launch: + - name: 'nf-core-rnaseq-launchpad' + workspace: 'your-organization/workspace-1' + pipeline: 'nf-core-rnaseq' + - name: 'nf-core-rnaseq-remote' + workspace: 'your-organization/workspace-1' + pipeline: 'https://github.com/nf-core/rnaseq' + compute-env: 'your-azure-compute-environment' + work-dir: 'az://your-container/pipelines/work/directory' + profile: 'test' + revision: '3.12.0' + params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' + config: 'path/to/pipelines/nextflow.config' + pre-run: 'path/to/pipelines/pre_run.txt' + - name: 'nf-core-sarek' + workspace: 'your-organization/workspace-1' + pipeline: 'nf-core-sarek' + config: 'path/to/pipelines/nextflow.config' + pre-run: 'path/to/pipelines/pre_run.txt' + - name: 'nf-core-viralrecon-illumina' + workspace: 'your-organization/workspace-1' + pipeline: 'nf-core-viralrecon-illumina' + - name: 'nf-sentieon' + workspace: 'your-organization/workspace-1' + pipeline: 'nf-sentieon' diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index 869cbc64..f34265d3 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -9,7 +9,7 @@ To determine the options to provide as definitions in your YAML file, run the Pl 1. Retrieve CLI options: - Obtain a list of available CLI options for defining your YAML file with the Platform CLI help command. For example, to add a pipeline to your workspace, view the options as follows: + Obtain a list of available CLI options for defining your YAML file with the Platform CLI help command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: ```console $ tw pipelines add -h @@ -64,28 +64,10 @@ To determine the options to provide as definitions in your YAML file, run the Pl - To list multiple values (such as multiple `labels`, `instance-types`, or `allow-buckets`), separate values with commas. This is shown with `labels` in the preceding example. - For complex configurations, see [Templates](./templates.mdx). -## Templates +### Templates See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. -## Real world example - -Please see [`seqerakit-e2e.yml`](./examples/yaml/seqerakit-e2e.yml) for an end-to-end example that highlights how you can use `seqerakit` to create everything sequentially in Seqera Platform all the way from creating a new Organization to launching a pipeline. - -You can modify this YAML to similarly create Seqera Platform resources end-to-end for your setup. This YAML encodes environment variables to protect sensitive keys, usernames, and passwords that are required to create or add certain resources (i.e. credentials, compute environments). Prior to running it with `seqerakit examples/yaml/seqerakit-e2e.yml`, you will have to set the following environment variables: - -```console -$TOWER_GITHUB_PASSWORD -$DOCKERHUB_PASSWORD -$AWS_ACCESS_KEY_ID -$AWS_SECRET_ACCESS_KEY -$AWS_ASSUME_ROLE_ARN -$AZURE_BATCH_KEY -$AZURE_STORAGE_KEY -$GOOGLE_KEY -$SENTIEON_LICENSE_BASE64 -``` - ## YAML Configuration Options There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. From 199f11015e212b0d0a358a1069376f937bf35066 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Sun, 17 Nov 2024 21:55:14 +0200 Subject: [PATCH 07/15] Update yaml-configuration.mdx --- .../seqerakit/yaml-configuration.mdx | 26 ++++++++++--------- 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index f34265d3..90d5a8ea 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -68,13 +68,13 @@ To determine the options to provide as definitions in your YAML file, run the Pl See [Templates](./templates.mdx) for YAML file templates for each of the entities that can be created in Seqera. -## YAML Configuration Options +### YAML Configuration Options -There are several options that can be provided in your YAML configuration file, that are handled specially by seqerakit and/or are not exposed as `tw` CLI options. +Some options handled specially by seqerakit or not exposed as `tw` CLI options can be provided in your YAML configuration file. -### 1. Pipeline parameters using `params` and `params-file` +#### Pipeline parameters using `params` and `params-file` -To specify pipeline parameters, you may either use `params:` to specify a list of parameters, or use `params-file:` to point to a parameters file. +To specify pipeline parameters, use `params:` to specify a list of parameters or `params-file:` to point to a parameters file. For example, to specify pipeline parameters within your YAML: @@ -84,13 +84,13 @@ params: fasta: 's3://path/to/reference.fasta' ``` -Alternatively, to specify a file containing pipeline parameters: +To specify a file containing pipeline parameters: ```yaml params-file: '/path/to/my/parameters.yaml' ``` -Optionally, you may provide both: +Or provide both: ```yaml params-file: '/path/to/my/parameters.yaml' @@ -99,13 +99,15 @@ params: fasta: 's3://path/to/reference.fasta' ``` -**Note**: If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the provided `params-file`. +:::note +If duplicate parameters are provided, the parameters provided as key-value pairs inside the `params` nested dictionary of the YAML file will take precedence **over** values in the `params-file`. +::: -### 2. `overwrite` Functionality +#### `overwrite` -For every entity defined in your YAML file, you can specify `overwrite: True` to overwrite any existing entities in Seqera Platform of the same name. +For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. -`seqerakit` will first check to see if the name of the entity exists, if so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. +`seqerakit` will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. ```console DEBUG:root: Overwrite is set to 'True' for organizations @@ -117,9 +119,9 @@ DEBUG:root: Running command: tw organizations delete --name $SEQERA_ORGANIZATION DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NAME --full-name $SEQERA_ORGANIZATION_NAME --description 'Example of an organization' ``` -### 3. Specifying JSON configuration files with `file-path` +#### Specify JSON configuration files with `file-path` -The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. +The Platform CLI allows the export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: From 26a4b2f0d0572ed24f1218ccc13104e1071e2e75 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Mon, 18 Nov 2024 10:30:54 +0200 Subject: [PATCH 08/15] Remove old pages --- .../version-24.1/seqerakit/overview.mdx | 26 ----------- .../version-24.1/seqerakit/quickstart.mdx | 46 ------------------- 2 files changed, 72 deletions(-) delete mode 100644 platform_versioned_docs/version-24.1/seqerakit/overview.mdx delete mode 100644 platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx diff --git a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx b/platform_versioned_docs/version-24.1/seqerakit/overview.mdx deleted file mode 100644 index e5474126..00000000 --- a/platform_versioned_docs/version-24.1/seqerakit/overview.mdx +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "Overview" -description: "An overview of seqerakit" -date: "10 Oct 2024" -tags: [seqerakit, cli, automation] ---- - -`seqerakit` is a Python wrapper that allows you to set [Platform CLI](../cli/overview.mdx) command options with YAML configuration files. Individual seqerakit commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. - -### Key features - -As an extension of the Platform CLI, seqerakit enables: - -- **Infrastructure as code**: Users manage and provision their infrastructure from the command line. -- **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. -- **Automation**: End-to-end creation of Seqera Platform entities, from adding an organization to launching pipelines. - -### Availability - -seqerakit can be installed on macOS, Windows, and Linux. It is compatible with [Seqera Platform Cloud](https://cloud.seqera.io/) and Enterprise versions 21.08 and later. - -See [Installation](./installation.mdx) to get started. - -:::info -You can also invoke and run `seqerakit` via a [Python script](https://github.com/seqeralabs/seqera-kit#launch-via-a-python-script). -::: diff --git a/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx b/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx deleted file mode 100644 index 5cb9347d..00000000 --- a/platform_versioned_docs/version-24.1/seqerakit/quickstart.mdx +++ /dev/null @@ -1,46 +0,0 @@ ---- -title: "Quickstart" -description: "seqerakit Hello World pipeline example" -date: "21 Oct 2024" -tags: [seqerakit, cli, automation, commands, hello world] ---- - -You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform. - -You will need a Seqera account (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a workspace and a pre-defined compute environment where you can launch a pipeline. - -### Launch via YAML - -1. Create a YAML file called `hello-world-config.yml` with the contents below, and customize the `` and `` entries as needed: - - ```yaml - launch: - - name: 'hello-world' # Workflow name - workspace: '' # Workspace name - compute-env: '' # Compute environment - revision: 'master' # Pipeline revision - pipeline: 'https://github.com/nextflow-io/hello' # Pipeline URL - ``` - -2. Launch the pipeline with `seqerakit`: - - ```bash - seqerakit hello-world-config.yml - ``` - -3. Log in to your Seqera instance and check the **Runs** page in the appropriate workspace for the pipeline you just launched. - -### Launch via a Python script - -You can also launch the same pipeline via a Python script. This allows you to extend Platform CLI functionality by leveraging the flexibility and customization options available in Python. - -1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customize the `` and `` entries as needed. - -2. Launch the pipeline with `seqerakit`: - -```bash - python launch_hello_world.py -``` - -3. Log in to your Seqera instance and check the **Runs** page in the appropriate workspace for the pipeline you just launched. - From be69c8d8b35d7ea61b25c4e83d3cd72deb720aeb Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Wed, 20 Nov 2024 12:23:45 +0200 Subject: [PATCH 09/15] Update platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx Co-authored-by: Justine Geffen Signed-off-by: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> --- .../version-24.1/seqerakit/yaml-configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index 90d5a8ea..f3e19846 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -9,7 +9,7 @@ To determine the options to provide as definitions in your YAML file, run the Pl 1. Retrieve CLI options: - Obtain a list of available CLI options for defining your YAML file with the Platform CLI help command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: + Obtain a list of available CLI options for defining your YAML file with the Platform CLI `help` command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: ```console $ tw pipelines add -h From 8fd5458e2191aa17ec77aeca572d40c01ccf8075 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Thu, 21 Nov 2024 17:05:46 +0200 Subject: [PATCH 10/15] Seqerakit vs `seqerakit` --- .../version-24.1/seqerakit/commands.mdx | 6 +++--- .../version-24.1/seqerakit/installation.mdx | 14 +++++++------- .../version-24.1/seqerakit/templates.mdx | 4 ++-- .../version-24.1/seqerakit/yaml-configuration.mdx | 4 ++-- 4 files changed, 14 insertions(+), 14 deletions(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index 8116f0e3..6a0639e0 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -13,7 +13,7 @@ seqerakit --help ### Input -`seqerakit` supports input through paths to YAML configuration files or directly from standard input (stdin). +Seqerakit supports input through paths to YAML configuration files or directly from standard input (stdin). - Using file path: @@ -31,7 +31,7 @@ See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-o ### Dryrun -Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your seqerakit command with the `--dryrun` flag: +Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your `seqerakit` command with the `--dryrun` flag: ```bash seqerakit file.yaml --dryrun @@ -96,7 +96,7 @@ seqerakit file.yaml --cli="--arg1 --arg2" See [CLI commands](../cli/commands.mdx) or run `tw -h` for the full list of options. :::note -The `--verbose` option for `tw` CLI is currently not supported by `seqerakit`. +The `--verbose` option for `tw` CLI is currently not supported in `seqerakit` commands. ::: #### Example: HTTP-only connections diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx index 0af1503a..c622e37e 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -6,9 +6,9 @@ tags: [seqerakit, cli, automation, installation] --- -`seqerakit` is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual seqerakit commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. +Seqerakit is a Python wrapper that sets [Platform CLI](../cli/overview.mdx) command options using YAML configuration files. Individual commands and configuration parameters can be chained together to automate the end-to-end creation of all Seqera Platform entities. -As an extension of the Platform CLI, seqerakit enables: +As an extension of the Platform CLI, Seqerakit enables: - **Infrastructure as code**: Users manage and provision their infrastructure from the command line. - **Simple configuration**: All Platform CLI command-line options can be defined in simple YAML format. @@ -16,7 +16,7 @@ As an extension of the Platform CLI, seqerakit enables: ### Installation -`seqerakit` has three dependencies: +Seqerakit has three dependencies: 1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli/releases) 2. [Python (`>=3.8`)](https://www.python.org/downloads/) @@ -24,7 +24,7 @@ As an extension of the Platform CLI, seqerakit enables: #### Pip -If you already have [Seqera Platform CLI](../cli/installation.mdx) and Python installed on your system, install `seqerakit` directly from [PyPI](https://pypi.org/project/seqerakit/): +If you already have [Seqera Platform CLI](../cli/installation.mdx) and Python installed on your system, install Seqerakit directly from [PyPI](https://pypi.org/project/seqerakit/): ```console pip install seqerakit @@ -78,7 +78,7 @@ Install the development branch of `seqerakit` on your local machine to test the Create a [Seqera](https://cloud.seqera.io/) access token via **Your Tokens** in the user menu. -`seqerakit` reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: +Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: ```bash export TOWER_ACCESS_TOKEN= @@ -94,7 +94,7 @@ By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cl ### Usage -To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected run this command: +To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected, run this command: ```bash seqerakit --info @@ -114,4 +114,4 @@ Use the `--help` or `-h` parameter to list the available commands and their asso seqerakit --help ``` -See [Commands](./commands.mdx) for detailed instructions to use `seqerakit`. \ No newline at end of file +See [Commands](./commands.mdx) for detailed instructions to use Seqerakit. \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx index 2332ac78..733613ef 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -22,12 +22,12 @@ import Secrets from '!!raw-loader!./templates/secrets.yml'; import Actions from '!!raw-loader!./templates/actions.yml'; import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yml'; -Customize YAML configuration templates to use in seqerakit commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single seqerakit command by combining them into a single configuration file. +Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. To use the templates on this page: 1. Copy the template text or download the YML files you need. 1. Edit the values to specify your resource details, and save as a YML file. -1. Specify the YML template file in your seqerakit commands: +1. Specify the YML template file in your `seqerakit` commands: - To create the resources specified in the file: ```console seqerakit /path/to/file.yml diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index f3e19846..ba7fe423 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -70,7 +70,7 @@ See [Templates](./templates.mdx) for YAML file templates for each of the entitie ### YAML Configuration Options -Some options handled specially by seqerakit or not exposed as `tw` CLI options can be provided in your YAML configuration file. +Some options handled specially by `seqerakit` or not exposed as `tw` CLI options can be provided in your YAML configuration file. #### Pipeline parameters using `params` and `params-file` @@ -107,7 +107,7 @@ If duplicate parameters are provided, the parameters provided as key-value pairs For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. -`seqerakit` will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. +Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. ```console DEBUG:root: Overwrite is set to 'True' for organizations From bfbf602d567bdd4d22d82a91b132fc798b34e1b7 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Mon, 25 Nov 2024 21:08:30 +0200 Subject: [PATCH 11/15] Formatting, review suggestions --- .../version-24.1/seqerakit/commands.mdx | 20 +- .../version-24.1/seqerakit/installation.mdx | 16 +- .../version-24.1/seqerakit/templates.mdx | 50 +-- .../seqerakit/templates/actions.yml | 20 +- .../seqerakit/templates/compute-envs.yml | 20 +- .../seqerakit/templates/credentials.yml | 85 ++--- .../seqerakit/templates/datasets.yml | 12 +- .../seqerakit/templates/labels.yml | 2 +- .../seqerakit/templates/launch.yml | 23 +- .../seqerakit/templates/members.yml | 6 +- .../seqerakit/templates/organizations.yml | 12 +- .../seqerakit/templates/participants.yml | 16 +- .../seqerakit/templates/pipelines.yml | 38 +-- .../seqerakit/templates/secrets.yml | 8 +- .../seqerakit/templates/seqerakit-e2e.yml | 312 ++++++++---------- .../seqerakit/templates/teams.yml | 12 +- .../seqerakit/templates/workspaces.yml | 12 +- .../seqerakit/yaml-configuration.mdx | 8 +- 18 files changed, 323 insertions(+), 349 deletions(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index 6a0639e0..23ec2fec 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -7,7 +7,7 @@ tags: [seqerakit, cli, automation, commands] Use the `--help` or `-h` parameter to list available commands and options: -```bash +```shell-session seqerakit --help ``` @@ -17,14 +17,14 @@ Seqerakit supports input through paths to YAML configuration files or directly f - Using file path: - ```bash + ```shell-session seqerakit /path/to/file.yaml ``` - Using stdin: - ```console - $ cat file.yaml | seqerakit - + ```shell-session + cat /path/to/file.yaml | seqerakit - ``` See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. @@ -33,7 +33,7 @@ See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-o Confirm that your configuration and command are correct before creating resources in your Seqera account, particularly when automating the end-to-end creation of multiple entities at once. To print the commands that would be executed with Platform CLI when using a YAML file, run your `seqerakit` command with the `--dryrun` flag: -```bash +```shell-session seqerakit file.yaml --dryrun ``` @@ -63,7 +63,7 @@ pipelines: You can target the creation of `pipelines` only by running: -```bash +```shell-session seqerakit test.yml --targets pipelines ``` @@ -71,7 +71,7 @@ This will process only the pipelines block from the YAML file and ignore `worksp To create both workspaces and pipelines, run: -```bash +```shell-session seqerakit test.yml --targets workspaces,pipelines ``` @@ -79,7 +79,7 @@ seqerakit test.yml --targets workspaces,pipelines Instead of adding or creating resources, specify the `--delete` flag to recursively delete resources in your YAML file: -```bash +```shell-session seqerakit file.yml --delete ``` @@ -89,7 +89,7 @@ For example, if you have a `file.yml` that defines an organization, workspace, t Specify `tw`-specific CLI options with the `--cli=` flag: -```bash +```shell-session seqerakit file.yaml --cli="--arg1 --arg2" ``` @@ -105,6 +105,6 @@ The Platform CLI expects to connect to a Seqera instance that is secured by a TL To use `tw`-specific CLI options such as `--insecure`, use the `--cli=` flag, followed by the options to use enclosed in double quotes: -```bash +```shell-session seqerakit file.yaml --cli="--insecure" ``` diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx index c622e37e..13dda87b 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -59,18 +59,18 @@ Install the development branch of `seqerakit` on your local machine to test the 1. You must have [Python](https://www.python.org/downloads/) and [Git](https://git-scm.com/downloads) installed on your system. 1. To install directly from pip: - ```bash + ```shell-session pip install git+https://github.com/seqeralabs/seqera-kit.git@dev ``` 1. Alternatively, clone the repository locally and install manually: - ```bash + ```shell-session git clone https://github.com/seqeralabs/seqera-kit.git cd seqera-kit git checkout dev pip install . ``` 1. Verify your installation: - ```bash + ```shell-session pip show seqerakit ``` @@ -80,13 +80,13 @@ Create a [Seqera](https://cloud.seqera.io/) access token via **Your Tokens** in Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: -```bash +```shell-session export TOWER_ACCESS_TOKEN= ``` For Enterprise installations, specify the custom API endpoint used to connect to Seqera. Export the API endpoint environment variable: -```bash +```shell-session export TOWER_API_ENDPOINT= ``` @@ -96,7 +96,7 @@ By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Cl To confirm the installation of `seqerakit`, configuration of the Platform CLI, and connection to Seqera is working as expected, run this command: -```bash +```shell-session seqerakit --info ``` @@ -104,13 +104,13 @@ This runs the `tw info` command under the hood. Use `--version` or `-v` to retrieve the current version of your `seqerakit` installation: -```bash +```shell-session seqerakit --version ``` Use the `--help` or `-h` parameter to list the available commands and their associated options: -```bash +```shell-session seqerakit --help ``` diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx index 733613ef..bf3d0672 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -29,11 +29,11 @@ To use the templates on this page: 1. Edit the values to specify your resource details, and save as a YML file. 1. Specify the YML template file in your `seqerakit` commands: - To create the resources specified in the file: - ```console + ```shell-session seqerakit /path/to/file.yml ``` - To delete the existing resources specified in the file: - ```console + ```shell-session seqerakit /path/to/file.yml --delete ``` @@ -41,27 +41,7 @@ To use the templates on this page: See [Specify targets](./commands.mdx#specify-targets) to create or delete only selected resources from configuration templates that contain multiple resource entries. ::: -### End-to-end example - -A template to create the follwing resources: - -- An organization -- A workspace -- A team -- Participants -- Credentials -- Secrets -- Compute environments -- Datasets -- Pipelines - -The templates also contains `launch` entries to launch saved pipelines. - - -{EndToEnd} - - -[Download seqerakit-e2e.yml](./templates/seqerakit-e2e.yml) +See [End-to-end example](#end-to-end-example) for a template that contains examples of all Seqera resources that can be created with Seqerakit. ### Administration @@ -195,4 +175,26 @@ Add or delete pipeline actions. {Actions} -[Download actions.yml](./templates/actions.yml) \ No newline at end of file +[Download actions.yml](./templates/actions.yml) + +### End-to-end example + +A template to create the follwing resources: + +- An organization +- A workspace +- A team +- Participants +- Credentials +- Secrets +- Compute environments +- Datasets +- Pipelines + +The templates also contains `launch` entries to launch saved pipelines. + + +{EndToEnd} + + +[Download seqerakit-e2e.yml](./templates/seqerakit-e2e.yml) \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml index aaa8ec75..5b65933e 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml @@ -1,24 +1,24 @@ ## To see the full list of options available, run: "tw actions add -h" actions: - type: 'github' # required - name: 'my_github_action' # required + name: 'your-github-action' # required pipeline: 'https://github.com/my_username/my_repo' # required - workspace: 'my_organization/my_workspace' # required - compute-env: 'my_aws_compute_environment' # required - work-dir: 's3://my_bucket' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required revision: 'main' # required profile: 'test' # optional params: # optional - outdir: 's3://my_bucket/my_results' + outdir: 's3://your-bucket/results' overwrite: True # optional - type: 'tower' # required - name: 'my_tower_action' # required + name: 'your-tower-action' # required pipeline: 'https://github.com/my_username/my_repo' # required - workspace: 'my_organization/my_workspace' # required - compute-env: 'my_aws_compute_environment' # required - work-dir: 's3://my_bucket' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # required revision: 'main' # required profile: 'test' # optional params: # optional - outdir: 's3://my_bucket/my_results' + outdir: 's3://your-bucket/results' overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml index 31b77645..237f6642 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml @@ -4,21 +4,21 @@ ## 2. Via a JSON file exported from Seqera with the "tw compute-envs export" command compute-envs: # To create a compute environment from a JSON configuration file (AWS Example) - - name: 'my_aws_compute_environment' # required - workspace: 'my_organization/my_workspace' # required - credentials: 'my_aws_credentials' # required + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required wait: 'AVAILABLE' # optional - file-path: './compute-envs/my_aws_compute_environment.json' # required + file-path: './compute-envs/aws_compute_environment.json' # required overwrite: True # optional # To create a compute environment with options specified through YAML (AWS Example) - type: aws-batch # required - config-mode: forge # required for AWS and Azure - name: 'my_aws_compute_environment' # required - workspace: 'my_organization/my_workspace' # required - credentials: 'my_aws_credentials' # required + config-mode: forge # required for AWS and Azure (forge or manual) + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required region: 'eu-west-1' # required - work-dir: 's3://my-bucket' # required + work-dir: 's3://your-bucket' # required provisioning-model: 'SPOT' # optional fusion-v2: False # optional wave: False # optional @@ -31,6 +31,6 @@ compute-envs: vpc-id: 'vpc-1234567890' # optional subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list - allow-buckets: 's3://my-bucket,s3://my-other-bucket' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list wait: 'AVAILABLE' # optional overwrite: False # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml index 445f60b3..e63db59e 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml @@ -2,43 +2,48 @@ ## To avoid exposing sensitive information about your credentials, ## use environment variables to supply passwords and secret keys. credentials: - - type: 'github' # required - name: 'my_github_credentials' # required - workspace: 'my_organization/my_workspace' # required - username: 'my_username' # required - password: '$SEQPLATFORM_GITHUB_PASSWORD' # required - overwrite: True # optional - - type: 'container-reg' # required - name: 'my_dockerhub_credentials' # required - workspace: 'my_organization/my_workspace' # required - username: 'my_username' # required - password: '$DOCKERHUB_PASSWORD' # required - registry: 'docker.io' # required - overwrite: True # optional - - type: 'google' # required - name: 'my_google_credentials' # required - workspace: 'my_organization/my_workspace' # required - key: '$GOOGLE_KEY' # required - overwrite: True # optional - - type: 'aws' # required - name: 'my_aws_credentials' # required - workspace: 'my_organization/my_workspace' # required - access-key: '$AWS_ACCESS_KEY_ID' # required - secret-key: '$AWS_SECRET_ACCESS_KEY' # required - assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required - overwrite: True # optional - - type: 'azure' # required - name: 'my_azure_credentials' # required - workspace: 'my_organization/my_workspace' # required - batch-key: '$AZURE_BATCH_KEY' # required - batch-name: 'my_storage_name' # required - storage-key: '$AZURE_STORAGE_KEY' # required - storage-name: 'my_storage_name' # required - overwrite: True # optional - - type: 'codecommit' # required - name: 'codecommit_credentials' # required - workspace: 'my_organization/my_workspace' # required - access-key: '$CODECOMMIT_USER' # required - secret-key: '$CODECOMMIT_PASSWORD' # required - base-url: '$CODECOMMIT_BASEURL' # optional - overwrite: False # optional + - type: 'github' # required + name: 'your-github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$SEQPLATFORM_GITHUB_PASSWORD' # required + overwrite: True # optional + + - type: 'container-reg' # required + name: 'your-dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'your-username' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + + - type: 'google' # required + name: 'your-google-credentials' # required + workspace: 'your-organization/workspace-1' # required + key: '$GOOGLE_KEY' # required + overwrite: True # optional + + - type: 'aws' # required + name: 'your-aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + + - type: 'azure' # required + name: 'your-azure-credentials' # required + workspace: 'your-organization/workspace-1' # required + batch-key: '$AZURE_BATCH_KEY' # required + batch-name: 'your-batch-name' # required + storage-key: '$AZURE_STORAGE_KEY' # required + storage-name: 'your-storage-name' # required + overwrite: True # optional + + - type: 'codecommit' # required + name: 'your-codecommit-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$CODECOMMIT_USER' # required + secret-key: '$CODECOMMIT_PASSWORD' # required + base-url: '$CODECOMMIT_BASEURL' # optional + overwrite: False # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml index 4542a771..b93364dd 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml @@ -1,8 +1,8 @@ ## To see the full list of options available, run: "tw datasets add -h" datasets: - - name: 'my_dataset' # required - description: 'My test dataset' # optional - header: true # optional - workspace: 'my_organization/my_workspace' # required - file-path: './datasets/my_dataset.csv' # required - overwrite: True # optional \ No newline at end of file + - name: 'your-dataset' # required + description: 'Your dataset description' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: './datasets/dataset.csv' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml index 00d45e0e..f8496a7f 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml @@ -2,5 +2,5 @@ labels: - name: 'label_name' # required value: 'label_value' # required - workspace: 'my_organization/my_workspace' # required + workspace: 'your-organization/workspace-1' # required overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml index 0e7e2049..745c63c0 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml @@ -5,18 +5,19 @@ ## Note: overwrite is not supported for "tw launch" launch: - - name: 'my_launchpad_launch' # required - workspace: 'my_organization/my_workspace' # required - pipeline: 'my_launchpad_pipeline' # required + - name: 'launchpad-launch' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required params: # optional - outdir: 's3://my_bucket/my_results' - - name: 'my_remote_launch' # required - workspace: 'my_organization/my_workspace' # required - compute-env: 'my_aws_compute_environment' # required + outdir: 's3://your-bucket/results' + + - name: 'remote-launch' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required pipeline: 'https://github.com/my_username/my_repo' # required - work-dir: 's3://my_bucket' # optional + work-dir: 's3://your-bucket' # optional profile: 'test' # optional revision: 'main' # optional - params-file: './pipelines/my_params.yml' # optional - config: './pipelines/my_nextflow.config' # optional - pre-run: './pipelines/my_pre_run.txt' # optional + params-file: './pipelines/params.yml' # optional + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml index 278c7dd4..f06b9f4a 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml @@ -1,5 +1,5 @@ ## To see the full list of options available, run: "tw members add -h" members: - - user: 'bob@myorg.io' # required - organization: 'myorg' # required - overwrite: True # optional \ No newline at end of file + - user: 'user@domain.com' # required + organization: 'your-organization' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml index 30bda2d7..6cf0518d 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml @@ -1,8 +1,8 @@ ## To see the full list of options available, run: "tw organizations add -h" organizations: - - name: 'my_organization' # required - full-name: 'my_organization' # required - description: 'My test organisation' # optional - location: 'Global' # optional - website: 'https://my_organization/' # optional - overwrite: True # optional \ No newline at end of file + - name: 'your-organization' # required + full-name: 'Your Organization LLC' # required + description: 'Your organization description' # optional + location: 'Global' # optional + website: 'https://domain.com/' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml index 8eda72fa..3c5e77da 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml @@ -1,10 +1,10 @@ ## To see the full list of options available, run: "tw participants add -h" participants: - - name: 'my_team' # required - type: 'TEAM' # required - workspace: 'my_organization/my_workspace' # required - role: 'ADMIN' # required - - name: 'my_team_member@gmail.com' # required - type: 'MEMBER' # required - workspace: 'my_organization/my_workspace' # required - role: 'LAUNCH' # required \ No newline at end of file + - name: 'your-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + - name: 'team-member@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml index 778ed95b..017d7252 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml @@ -3,22 +3,22 @@ ## 1. Explicitly in this file ## 2. Via a JSON file exported from Seqera with the "tw pipelines export" command pipelines: - - name: 'my_first_pipeline' # required - workspace: 'my_organization/my_workspace' # required - description: 'Options specified in this file' # optional - compute-env: 'my_aws_compute_environment' # required - work-dir: 's3://my_bucket' # optional - profile: 'test' # optional - revision: 'main' # required - params: # optional - outdir: 's3://my-bucket/my_results' - config: './pipelines/my_nextflow.config' # optional - pre-run: './pipelines/my_pre_run.txt' # optional - url: 'https://github.com/my_username/my_repo' # required - overwrite: True # optional - - name: 'my_second_pipeline' # required - workspace: 'my_organization/my_workspace' # required - description: 'Options specified via JSON file' # optional - compute-env: 'my_aws_compute_environment' # required - file-path: './pipelines/my_pipeline.json' # required - overwrite: True # optional + - name: 'your-first-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified in this file' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket' # optional + profile: 'test' # optional + revision: 'main' # required + params: # optional + outdir: 's3://your-bucket/results' + config: './pipelines/nextflow.config' # optional + pre-run: './pipelines/pre-run.txt' # optional + url: 'https://github.com/my_username/my_repo' # required + overwrite: True # optional + - name: 'your-second-pipeline' # required + workspace: 'your-organization/workspace-1' # required + description: 'Options specified via JSON file' # optional + compute-env: 'your-aws-compute-environment' # required + file-path: './pipelines/pipeline.json' # required + overwrite: True # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml index 6e04ff3e..743faba3 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml @@ -1,6 +1,6 @@ ## To see the full list of options available, run: "tw secrets add -h" secrets: - - name: 'my_secret' # required - workspace: 'my_organization/my_workspace' # required - value: 'my_secret_value' # required - overwrite: True # optional \ No newline at end of file + - name: 'your-secret' # required + workspace: 'your-organization/workspace-1' # required + value: 'your-secret-value' # required + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml index d70e2d95..fcdea506 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml @@ -1,184 +1,148 @@ organizations: - - name: 'your-organization' - full-name: 'Your organization LLC' - description: 'Organization created E2E with seqerakit CLI scripting' - location: 'Global' - website: 'https://yourdomain.com/' - overwrite: True + - name: 'your-organization' # required + full-name: 'Your organization LLC' # required + description: 'Organization created E2E with seqerakit CLI scripting' # optional + location: 'Global' # optional + website: 'https://yourdomain.com/' # optional + overwrite: True # optional + teams: - - name: 'pipelines_team' - organization: 'your-organization' - description: 'Pipelines team @ Your organization' - members: - - 'user1@yourdomain.com' - - 'user2@yourdomain.com' - - 'user3@yourdomain.com' - overwrite: True + - name: 'pipelines-team' # required + organization: 'your-organization' # required + description: 'Pipelines team @ Your organization' # optional + members: # optional + - 'user1@domain.com' + - 'user2@domain.com' + - 'user3@domain.com' + overwrite: True # optional + workspaces: - - name: 'workspace-1' - full-name: 'Workspace one' - organization: 'your-organization' - description: 'Workspace created E2E with seqerakit CLI scripting' - visibility: 'PRIVATE' - overwrite: True + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Workspace created E2E with seqerakit CLI scripting' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional + participants: - - name: 'pipelines_team' - type: 'TEAM' - workspace: 'your-organization/workspace-1' - role: 'ADMIN' - overwrite: True - - name: 'user4@domain.com@seqera.io' - type: 'MEMBER' - workspace: 'your-organization/workspace-1' - role: 'LAUNCH' - overwrite: True + - name: 'pipelines-team' # required + type: 'TEAM' # required + workspace: 'your-organization/workspace-1' # required + role: 'ADMIN' # required + overwrite: True # optional + - name: 'user@domain.com' # required + type: 'MEMBER' # required + workspace: 'your-organization/workspace-1' # required + role: 'LAUNCH' # required + overwrite: True # optional + credentials: - - type: 'github' - name: 'github_credentials' - workspace: 'your-organization/workspace-1' - username: 'user1' - password: '$TOWER_GITHUB_PASSWORD' - overwrite: True - - type: 'container-reg' - name: 'dockerhub_credentials' - workspace: 'your-organization/workspace-1' - username: 'user1' - password: '$DOCKERHUB_PASSWORD' - registry: 'docker.io' - overwrite: True - - type: 'aws' - name: 'aws_credentials' - workspace: 'your-organization/workspace-1' - access-key: '$AWS_ACCESS_KEY_ID' - secret-key: '$AWS_SECRET_ACCESS_KEY' - assume-role-arn: '$AWS_ASSUME_ROLE_ARN' - overwrite: True - - type: 'azure' - name: 'azure_credentials' - workspace: 'your-organization/workspace-1' - batch-key: '$AZURE_BATCH_KEY' - batch-name: 'yourbatch' - storage-key: '$AZURE_STORAGE_KEY' - storage-name: 'yourstorage' - overwrite: True - - type: 'google' - name: 'google_credentials' - workspace: 'your-organization/workspace-1' - key: '$GOOGLE_KEY' - overwrite: True + - type: 'github' # required + name: 'github-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$TOWER_GITHUB_PASSWORD' # required + overwrite: True # optional + - type: 'container-reg' # required + name: 'dockerhub-credentials' # required + workspace: 'your-organization/workspace-1' # required + username: 'user1' # required + password: '$DOCKERHUB_PASSWORD' # required + registry: 'docker.io' # required + overwrite: True # optional + - type: 'aws' # required + name: 'aws-credentials' # required + workspace: 'your-organization/workspace-1' # required + access-key: '$AWS_ACCESS_KEY_ID' # required + secret-key: '$AWS_SECRET_ACCESS_KEY' # required + assume-role-arn: '$AWS_ASSUME_ROLE_ARN' # required + overwrite: True # optional + secrets: - - name: 'SENTIEON_LICENSE_BASE64' - workspace: 'your-organization/workspace-1' - value: '$SENTIEON_LICENSE_BASE64' - overwrite: True + - name: 'SENTIEON_LICENSE_BASE64' # required + workspace: 'your-organization/workspace-1' # required + value: '$SENTIEON_LICENSE_BASE64' # required + overwrite: True # optional + compute-envs: - - name: 'your-aws-compute-environment' - workspace: 'your-organization/workspace-1' - credentials: 'aws_credentials' - wait: 'AVAILABLE' - file-path: 'path/to/aws-credentials.json' - overwrite: True - - name: 'your-azure-compute-environment' - workspace: 'your-organization/workspace-1' - credentials: 'azure_credentials' - wait: 'AVAILABLE' - file-path: 'path/to/azure-credentials.json' - overwrite: True - - name: 'your-gcp-compute-environment' - workspace: 'your-organization/workspace-1' - credentials: 'google_credentials' - wait: 'AVAILABLE' - file-path: 'path/to/gcp-credentials.json' - overwrite: True +# To create a compute environment with options specified through YAML (AWS Example) + - type: aws-batch # required + config-mode: forge # required for AWS and Azure + name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + region: 'eu-west-1' # required + work-dir: 's3://your-bucket' # required + provisioning-model: 'SPOT' # optional + fusion-v2: False # optional + wave: False # optional + fargate: False # optional + fast-storage: False # optional + instance-types: 'c6i,r6i,m6i' # optional, comma-separated list + no-ebs-auto-scale: True # optional + max-cpus: 500 # required + labels: 'label1,label2' # optional, comma-separated list + vpc-id: 'vpc-1234567890' # optional + subnets: 'subnet-1234567890,subnet-1234567891' # optional, comma-separated list + security-groups: 'sg-1234567890,sg-1234567891' # optional, comma-separated list + allow-buckets: 's3://your-bucket,s3://your-other-bucket' # optional, comma-separated list + wait: 'AVAILABLE' # optional + overwrite: False # optional + +# To create a compute environment from a JSON configuration file (AWS Example) + - name: 'your-aws-compute-environment' # required + workspace: 'your-organization/workspace-1' # required + credentials: 'your-aws-credentials' # required + wait: 'AVAILABLE' # optional + file-path: './compute-envs/your_aws_compute_environment.json' # required + overwrite: True # optional + datasets: - - name: 'rnaseq_samples' - description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' - header: true - workspace: 'your-organization/workspace-1' - file-path: 'path/to/rnaseq_samples.csv' - overwrite: True - - name: 'sarek_samples' - description: 'Samplesheet to run the nf-core/sarek pipeline end-to-end' - header: true - workspace: 'your-organization/workspace-1' - file-path: 'path/to/sarek_samples.csv' - overwrite: True - - name: 'viralrecon_illumina_samples' - description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Illumina data' - header: true - workspace: 'your-organization/workspace-1' - file-path: 'path/to/viralrecon_illumina_samples.csv' - overwrite: True - - name: 'viralrecon_nanopore_samples' - description: 'Samplesheet to run the nf-core/viralrecon pipeline from end-to-end with Nanopore data' - header: true - workspace: 'your-organization/workspace-1' - file-path: 'path/to/viralrecon_nanopore_samples.csv' - overwrite: True - - name: 'sentieon_samples' - description: 'Samplesheet to run the seqeralabs/nf-sentieon pipeline from end-to-end' - header: true - workspace: 'your-organization/workspace-1' - file-path: 'path/to/sentieon_samples.csv' - overwrite: True + - name: 'rnaseq_samples' # required + description: 'Samplesheet to run the nf-core/rnaseq pipeline end-to-end' # optional + header: true # optional + workspace: 'your-organization/workspace-1' # required + file-path: 'path/to/rnaseq_samples.csv' # required + overwrite: True # optional + pipelines: - - name: 'nf-core-rnaseq' - url: 'https://github.com/nf-core/rnaseq' - workspace: 'your-organization/workspace-1' - description: 'RNA sequencing analysis pipeline using STAR, RSEM, HISAT2 or Salmon with gene/isoform counts and extensive quality control.' - compute-env: 'your-aws-compute-environment' - work-dir: 's3://your-bucket/pipeline/working/directory' - profile: 'test' - revision: '3.12.0' - params: - outdir: 's3://your-bucket/nf-core-rnaseq/results' - config: 'path/to/nextflow.config' - pre-run: 'path/to/pipeline/pre_run.txt' - overwrite: True - - name: 'nf-core-sarek' - workspace: 'your-organization/workspace-1' - compute-env: 'your-azure-compute-environment' - file-path: 'path/to/nf-core-sarek_pipeline.json' - overwrite: True - - name: 'nf-core-viralrecon-illumina' - url: 'https://github.com/nf-core/viralrecon' - workspace: 'your-organization/workspace-1' - description: 'Pipeline for assembly and intrahost/low-frequency variant calling for viral samples.' - compute-env: 'your-gcp-compute-environment' - work-dir: 'gs://your-bucket/pipelines/work/directory' - profile: 'test' - revision: '2.6.0' - params-file: 'path/to/pipelines/nf_core_viralrecon_illumina_params.yml' - config: 'path/to/pipelines/nextflow.config' - pre-run: 'path/to/pipelines/pre_run.txt' - overwrite: True - - name: 'nf-sentieon' - workspace: 'your-organization/workspace-1' - compute-env: 'your-aws-compute-environment' - file-path: 'path/to/pipelines/nf_sentieon_pipeline.json' - overwrite: True +# To create a pipeline with options specified through YAML + - name: 'nf-core-rnaseq' # required + url: 'https://github.com/nf-core/rnaseq' # required + workspace: 'your-organization/workspace-1' # required + description: 'RNA sequencing analysis pipeline with gene/isoform counts and extensive quality control.' # optional + compute-env: 'your-aws-compute-environment' # required + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # required + params: # optional + outdir: 's3://your-bucket/nf-core-rnaseq/results' + config: 'path/to/nextflow.config' # optional + pre-run: 'path/to/pipeline/pre_run.txt' # optional + overwrite: True # optional + +# To create a pipeline from a JSON configuration file + - name: 'nf-core-sarek' # required + workspace: 'your-organization/workspace-1' # required + compute-env: 'your-aws-compute-environment' # required + file-path: 'path/to/nf-core-sarek_pipeline.json' # required + overwrite: True # optional + launch: - - name: 'nf-core-rnaseq-launchpad' - workspace: 'your-organization/workspace-1' - pipeline: 'nf-core-rnaseq' - - name: 'nf-core-rnaseq-remote' - workspace: 'your-organization/workspace-1' - pipeline: 'https://github.com/nf-core/rnaseq' - compute-env: 'your-azure-compute-environment' - work-dir: 'az://your-container/pipelines/work/directory' - profile: 'test' - revision: '3.12.0' - params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' - config: 'path/to/pipelines/nextflow.config' - pre-run: 'path/to/pipelines/pre_run.txt' - - name: 'nf-core-sarek' - workspace: 'your-organization/workspace-1' - pipeline: 'nf-core-sarek' - config: 'path/to/pipelines/nextflow.config' - pre-run: 'path/to/pipelines/pre_run.txt' - - name: 'nf-core-viralrecon-illumina' - workspace: 'your-organization/workspace-1' - pipeline: 'nf-core-viralrecon-illumina' - - name: 'nf-sentieon' - workspace: 'your-organization/workspace-1' - pipeline: 'nf-sentieon' +# To launch a preconfigured pipeline from your workspace Launchpad + - name: 'nf-core-rnaseq-launchpad' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'nf-core-rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + +# To launch an unsaved pipeline, include the pipeline repository URL and launch details + - name: 'nf-core-rnaseq-remote' # required + workspace: 'your-organization/workspace-1' # required + pipeline: 'https://github.com/nf-core/rnaseq' # required + compute-env: 'your-aws-compute-environment' # optional (defaults to workspace primary compute environment) + work-dir: 's3://your-bucket/pipeline/working/directory' # optional + profile: 'test' # optional + revision: '3.12.0' # optional + params-file: 'path/to/pipelines/nf_core_rnaseq_params.yml' # optional + config: 'path/to/pipelines/nextflow.config' # optional + pre-run: 'path/to/pipelines/pre_run.txt' # optional diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml index 69dd951b..b1258de2 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml @@ -1,8 +1,8 @@ ## To see the full list of options available, run: "tw teams add -h" teams: - - name: 'my_team' # required - organization: 'my_organization' # required - description: 'My test team' # optional - members: # optional - - 'my_team_member@gmail.com' - overwrite: True # optional \ No newline at end of file + - name: 'your-team' # required + organization: 'your-organization' # required + description: 'Your team description.' # optional + members: # optional + - 'team-member@domain.com' + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml index ccf9f75b..ca939234 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml +++ b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml @@ -1,8 +1,8 @@ ## To see the full list of options available, run: "tw workspaces add -h" workspaces: - - name: 'my_workspace' # required - full-name: 'my_workspace' # required - organization: 'my_organization' # required - description: 'My test workspace' # optional - visibility: 'PRIVATE' # optional - overwrite: True # optional \ No newline at end of file + - name: 'workspace-1' # required + full-name: 'Workspace one' # required + organization: 'your-organization' # required + description: 'Your workspace description' # optional + visibility: 'PRIVATE' # optional + overwrite: True # optional \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index ba7fe423..e1db15a7 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -11,9 +11,11 @@ To determine the options to provide as definitions in your YAML file, run the Pl Obtain a list of available CLI options for defining your YAML file with the Platform CLI `help` command. For example, to add a pipeline to your workspace, view the options for adding a pipeline: - ```console - $ tw pipelines add -h + ```shell-session + tw pipelines add -h + ``` + ```shell-session Usage: tw pipelines add [OPTIONS] PIPELINE_URL Add a workspace pipeline. @@ -109,7 +111,7 @@ For every entity defined in your YAML file, specify `overwrite: True` to overwri Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. -```console +```shell-session DEBUG:root: Overwrite is set to 'True' for organizations DEBUG:root: Running command: tw -o json organizations list From c462d56f66ce9760351a813aff70b099821a5f4d Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Thu, 28 Nov 2024 11:18:41 +0200 Subject: [PATCH 12/15] Update platform_versioned_docs/version-24.1/seqerakit/commands.mdx Co-authored-by: Esha Joshi <128735622+ejseqera@users.noreply.github.com> Signed-off-by: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> --- .../version-24.1/seqerakit/commands.mdx | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index 23ec2fec..a387ae05 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -39,7 +39,24 @@ seqerakit file.yaml --dryrun ### Specify targets -When using a YAML file as input that defines multiple resources, use the `--targets` flag to specify which resources to create. This flag accepts a comma-separated list of resource names. +When using a YAML file as input that defines multiple resources, use the `--targets` flag to specify which resources to create. This flag accepts a comma-separated list of resource names. + +Supported resource names include: + +- actions +- compute-envs +- credentials +- datasets +- labels +- launch +- members +- organizations +- participants +- pipelines +- secrets +- teams +- workspaces + For example, given a `test.yml` file that defines the following resources: From 3ff8439805f820ff5e70d274d79bbfa1985d9336 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Thu, 28 Nov 2024 12:28:55 +0200 Subject: [PATCH 13/15] Update platform_versioned_docs/version-24.1/seqerakit/templates.mdx Co-authored-by: Esha Joshi <128735622+ejseqera@users.noreply.github.com> Signed-off-by: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> --- platform_versioned_docs/version-24.1/seqerakit/templates.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx index bf3d0672..b4744b80 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -25,7 +25,7 @@ import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yml'; Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. To use the templates on this page: -1. Copy the template text or download the YML files you need. +1. Copy the template text or download the YAML files you need. 1. Edit the values to specify your resource details, and save as a YML file. 1. Specify the YML template file in your `seqerakit` commands: - To create the resources specified in the file: From 875287453a5de8f27ebca2a871ee7dd1c17175e3 Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Thu, 28 Nov 2024 12:33:26 +0200 Subject: [PATCH 14/15] Tech review suggestions --- .../version-24.1/cli/installation.mdx | 6 +- .../version-24.1/seqerakit/commands.mdx | 16 ++--- .../version-24.1/seqerakit/installation.mdx | 4 +- .../version-24.1/seqerakit/templates.mdx | 60 +++++++++---------- .../templates/{actions.yml => actions.yaml} | 0 .../{compute-envs.yml => compute-envs.yaml} | 0 .../{credentials.yml => credentials.yaml} | 0 .../templates/{datasets.yml => datasets.yaml} | 0 .../templates/{labels.yml => labels.yaml} | 0 .../templates/{launch.yml => launch.yaml} | 0 .../templates/{members.yml => members.yaml} | 0 .../{organizations.yml => organizations.yaml} | 0 .../{participants.yml => participants.yaml} | 0 .../{pipelines.yml => pipelines.yaml} | 0 .../templates/{secrets.yml => secrets.yaml} | 0 .../{seqerakit-e2e.yml => seqerakit-e2e.yaml} | 0 .../templates/{teams.yml => teams.yaml} | 0 .../{workspaces.yml => workspaces.yaml} | 0 .../seqerakit/yaml-configuration.mdx | 4 +- 19 files changed, 45 insertions(+), 45 deletions(-) rename platform_versioned_docs/version-24.1/seqerakit/templates/{actions.yml => actions.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{compute-envs.yml => compute-envs.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{credentials.yml => credentials.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{datasets.yml => datasets.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{labels.yml => labels.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{launch.yml => launch.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{members.yml => members.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{organizations.yml => organizations.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{participants.yml => participants.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{pipelines.yml => pipelines.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{secrets.yml => secrets.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{seqerakit-e2e.yml => seqerakit-e2e.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{teams.yml => teams.yaml} (100%) rename platform_versioned_docs/version-24.1/seqerakit/templates/{workspaces.yml => workspaces.yaml} (100%) diff --git a/platform_versioned_docs/version-24.1/cli/installation.mdx b/platform_versioned_docs/version-24.1/cli/installation.mdx index 762b67ab..4acbbd0e 100644 --- a/platform_versioned_docs/version-24.1/cli/installation.mdx +++ b/platform_versioned_docs/version-24.1/cli/installation.mdx @@ -65,9 +65,9 @@ tw info Details -------------------------+---------------------- Tower API endpoint | - Tower API version | 1.25.0 - Tower version | 24.2.0_cycle22 - CLI version | 0.9.4 (f3e846e) + Tower API version | 1.25.0 + Tower version | 24.2.0_cycle27 + CLI version | 0.10.3 (a86eeed) CLI minimum API version | 1.15 Authenticated user | diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index a387ae05..b1f34cf7 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -18,13 +18,13 @@ Seqerakit supports input through paths to YAML configuration files or directly f - Using file path: ```shell-session - seqerakit /path/to/file.yaml + seqerakit file.yaml ``` - Using stdin: ```shell-session - cat /path/to/file.yaml | seqerakit - + cat file.yaml | seqerakit - ``` See [Define your YAML file using CLI options](#define-your-yaml-file-using-cli-options) for guidance on formatting your input YAML files. @@ -58,7 +58,7 @@ Supported resource names include: - workspaces -For example, given a `test.yml` file that defines the following resources: +For example, given a `test.yaml` file that defines the following resources: ```yaml workspaces: @@ -81,15 +81,15 @@ pipelines: You can target the creation of `pipelines` only by running: ```shell-session -seqerakit test.yml --targets pipelines +seqerakit test.yaml --targets pipelines ``` -This will process only the pipelines block from the YAML file and ignore `workspaces` and `compute-envs`. +This command will create only the pipelines defined in the YAML file and ignore `workspaces` and `compute-envs`. To create both workspaces and pipelines, run: ```shell-session -seqerakit test.yml --targets workspaces,pipelines +seqerakit test.yaml --targets workspaces,pipelines ``` ### Delete resources @@ -97,10 +97,10 @@ seqerakit test.yml --targets workspaces,pipelines Instead of adding or creating resources, specify the `--delete` flag to recursively delete resources in your YAML file: ```shell-session -seqerakit file.yml --delete +seqerakit file.yaml --delete ``` -For example, if you have a `file.yml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yml --delete` to recursively delete the same resources. +For example, if you have a `file.yaml` that defines an organization, workspace, team, credentials, and compute environment that have already been created, run `seqerakit file.yaml --delete` to recursively delete the same resources. ### Use `tw`-specific CLI options diff --git a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx index 13dda87b..fb4f54f9 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/installation.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/installation.mdx @@ -18,7 +18,7 @@ As an extension of the Platform CLI, Seqerakit enables: Seqerakit has three dependencies: -1. [Seqera Platform CLI (`>=0.9.2`)](https://github.com/seqeralabs/tower-cli/releases) +1. [Seqera Platform CLI (`>=0.10.1`)](https://github.com/seqeralabs/tower-cli/releases) 2. [Python (`>=3.8`)](https://www.python.org/downloads/) 3. [PyYAML](https://pypi.org/project/PyYAML/) @@ -76,7 +76,7 @@ Install the development branch of `seqerakit` on your local machine to test the ### Configuration -Create a [Seqera](https://cloud.seqera.io/) access token via **Your Tokens** in the user menu. +Create a [Seqera](https://cloud.seqera.io/tokens) access token via **Your Tokens** in the user menu. Seqerakit reads your access token from the `TOWER_ACCESS_TOKEN` environment variable: diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx index bf3d0672..b4ca9bf5 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/templates.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/templates.mdx @@ -7,20 +7,20 @@ toc_max_heading_level: 4 --- import CodeBlock from '@theme/CodeBlock'; -import Organizations from '!!raw-loader!./templates/organizations.yml'; -import Members from '!!raw-loader!./templates/members.yml'; -import Workspaces from '!!raw-loader!./templates/workspaces.yml'; -import Teams from '!!raw-loader!./templates/teams.yml'; -import Participants from '!!raw-loader!./templates/participants.yml'; -import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yml'; -import Pipelines from '!!raw-loader!./templates/pipelines.yml'; -import Credentials from '!!raw-loader!./templates/credentials.yml'; -import Datasets from '!!raw-loader!./templates/datasets.yml'; -import Labels from '!!raw-loader!./templates/labels.yml'; -import Launch from '!!raw-loader!./templates/launch.yml'; -import Secrets from '!!raw-loader!./templates/secrets.yml'; -import Actions from '!!raw-loader!./templates/actions.yml'; -import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yml'; +import Organizations from '!!raw-loader!./templates/organizations.yaml'; +import Members from '!!raw-loader!./templates/members.yaml'; +import Workspaces from '!!raw-loader!./templates/workspaces.yaml'; +import Teams from '!!raw-loader!./templates/teams.yaml'; +import Participants from '!!raw-loader!./templates/participants.yaml'; +import ComputeEnvironments from '!!raw-loader!./templates/compute-envs.yaml'; +import Pipelines from '!!raw-loader!./templates/pipelines.yaml'; +import Credentials from '!!raw-loader!./templates/credentials.yaml'; +import Datasets from '!!raw-loader!./templates/datasets.yaml'; +import Labels from '!!raw-loader!./templates/labels.yaml'; +import Launch from '!!raw-loader!./templates/launch.yaml'; +import Secrets from '!!raw-loader!./templates/secrets.yaml'; +import Actions from '!!raw-loader!./templates/actions.yaml'; +import EndToEnd from '!!raw-loader!./templates/seqerakit-e2e.yaml'; Customize YAML configuration templates to use in `seqerakit` commands to create, update, or delete Seqera resources. Create or delete multiple resources with a single command by combining them into a single configuration file. @@ -30,11 +30,11 @@ To use the templates on this page: 1. Specify the YML template file in your `seqerakit` commands: - To create the resources specified in the file: ```shell-session - seqerakit /path/to/file.yml + seqerakit /path/to/file.yaml ``` - To delete the existing resources specified in the file: ```shell-session - seqerakit /path/to/file.yml --delete + seqerakit /path/to/file.yaml --delete ``` :::info @@ -55,7 +55,7 @@ Add or delete organizations. {Organizations} -[Download organizations.yml](./templates/organizations.yml) +[Download organizations.yaml](./templates/organizations.yaml) #### Members @@ -65,7 +65,7 @@ Add or delete organization members. {Members} -[Download members.yml](./templates/members.yml) +[Download members.yaml](./templates/members.yaml) #### Workspaces @@ -75,7 +75,7 @@ Add or delete workspaces. {Workspaces} -[Download workspaces.yml](./templates/workspaces.yml) +[Download workspaces.yaml](./templates/workspaces.yaml) #### Teams @@ -85,7 +85,7 @@ Add or delete teams. {Teams} -[Download teams.yml](./templates/teams.yml) +[Download teams.yaml](./templates/teams.yaml) #### Participants @@ -95,7 +95,7 @@ Add or delete participants in workspaces and teams. {Participants} -[Download participants.yml](./templates/participants.yml) +[Download participants.yaml](./templates/participants.yaml) ### Credentials @@ -105,7 +105,7 @@ Add or delete compute environment, Git, and container registry credentials in wo {Credentials} -[Download credentials.yml](./templates/credentials.yml) +[Download credentials.yaml](./templates/credentials.yaml) ### Compute environments @@ -115,7 +115,7 @@ Add or delete compute environments. {ComputeEnvironments} -[Download compute-envs.yml](./templates/compute-envs.yml) +[Download compute-envs.yaml](./templates/compute-envs.yaml) ### Pipelines @@ -125,7 +125,7 @@ Add or delete pipelines in workspace Launchpads. {Pipelines} -[Download pipelines.yml](./templates/pipelines.yml) +[Download pipelines.yaml](./templates/pipelines.yaml) ### Launch @@ -135,7 +135,7 @@ Launch a Nextflow pipeline. {Launch} -[Download launch.yml](./templates/launch.yml) +[Download launch.yaml](./templates/launch.yaml) ### Datasets @@ -145,7 +145,7 @@ Add or delete workspace datasets for pipeline input data. {Datasets} -[Download datasets.yml](./templates/datasets.yml) +[Download datasets.yaml](./templates/datasets.yaml) ### Labels @@ -155,7 +155,7 @@ Add or delete labels and resource labels to apply to workspace compute environme {Labels} -[Download labels.yml](./templates/labels.yml) +[Download labels.yaml](./templates/labels.yaml) ### Secrets @@ -165,7 +165,7 @@ Add or delete user and workspace secrets. {Secrets} -[Download secrets.yml](./templates/secrets.yml) +[Download secrets.yaml](./templates/secrets.yaml) ### Actions @@ -175,7 +175,7 @@ Add or delete pipeline actions. {Actions} -[Download actions.yml](./templates/actions.yml) +[Download actions.yaml](./templates/actions.yaml) ### End-to-end example @@ -197,4 +197,4 @@ The templates also contains `launch` entries to launch saved pipelines. {EndToEnd} -[Download seqerakit-e2e.yml](./templates/seqerakit-e2e.yml) \ No newline at end of file +[Download seqerakit-e2e.yaml](./templates/seqerakit-e2e.yaml) \ No newline at end of file diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/actions.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/actions.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/actions.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/compute-envs.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/credentials.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/datasets.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/labels.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/labels.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/labels.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/launch.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/launch.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/launch.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/members.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/members.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/members.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/members.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/organizations.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/participants.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/participants.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/participants.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/pipelines.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/secrets.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/seqerakit-e2e.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/teams.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/teams.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/teams.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml b/platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yaml similarity index 100% rename from platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yml rename to platform_versioned_docs/version-24.1/seqerakit/templates/workspaces.yaml diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index e1db15a7..4ef09682 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -109,7 +109,7 @@ If duplicate parameters are provided, the parameters provided as key-value pairs For every entity defined in your YAML file, specify `overwrite: True` to overwrite any existing Seqera entities of the same name. -Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. +Seqerakit will first check to see if the name of the entity exists. If so, it will invoke a `tw delete` command before attempting to create it based on the options defined in the YAML file. ```shell-session DEBUG:root: Overwrite is set to 'True' for organizations @@ -125,7 +125,7 @@ DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NA The Platform CLI allows the export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file. -An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template: +An example of the `file-path` option is provided in the [compute-envs.yaml](./templates/compute-envs.yaml) template: ```yaml compute-envs: From e3c3b73d0e3a9baf59c9c03282f4a7b86c3d4d3d Mon Sep 17 00:00:00 2001 From: Llewellyn vd Berg <113503285+llewellyn-sl@users.noreply.github.com> Date: Thu, 28 Nov 2024 12:44:03 +0200 Subject: [PATCH 15/15] Add resource names list --- .../version-24.1/seqerakit/commands.mdx | 27 +++++++++---------- .../seqerakit/yaml-configuration.mdx | 16 +++++++++++ 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx index b1f34cf7..c2931274 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/commands.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/commands.mdx @@ -43,20 +43,19 @@ When using a YAML file as input that defines multiple resources, use the `--targ Supported resource names include: -- actions -- compute-envs -- credentials -- datasets -- labels -- launch -- members -- organizations -- participants -- pipelines -- secrets -- teams -- workspaces - +- `actions` +- `compute-envs` +- `credentials` +- `datasets` +- `labels` +- `launch` +- `members` +- `organizations` +- `participants` +- `pipelines` +- `secrets` +- `teams` +- `workspaces` For example, given a `test.yaml` file that defines the following resources: diff --git a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx index 4ef09682..572c5083 100644 --- a/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx +++ b/platform_versioned_docs/version-24.1/seqerakit/yaml-configuration.mdx @@ -5,6 +5,22 @@ date: "21 Oct 2024" tags: [seqerakit, cli, automation, yaml, configuration] --- +Seqerakit supports the creation and deletion of the following Seqera Platform resources, listed here with their respective Platform CLI resource names: + +- Pipeline actions: `actions` +- Compute environments: `compute-envs` +- Credentials: `credentials` +- Datasets: `datasets` +- Labels (including resource labels): `labels` +- Pipeline launch: `launch` +- Organization members: `members` +- Organizations: `organizations` +- Workspace and team participants: `participants` +- Pipelines: `pipelines` +- Pipeline secrets: `secrets` +- Teams: `teams` +- Workspaces: `workspaces` + To determine the options to provide as definitions in your YAML file, run the Platform CLI help command for the resource you want to create. 1. Retrieve CLI options: