Add data link usage (#442)

* Add data link usage * Clean up whitespace * More detail for browse * More info for browse options * Incorporate suggestions * Fixed whitespace fix * Fix whitespace * Fix whitespace again * Align whitespace * Whitespace again * Fix headers
seqeralabs · Aug 12, 2024 · 9e0650e · 9e0650e
1 parent ba9b0d2
commit 9e0650e
Showing 1 changed file with 134 additions and 1 deletion.
diff --git a/USAGE.md b/USAGE.md
@@ -413,7 +413,7 @@ $ tw runs dump -i 5z4AMshti4g0GK -o file.tar.gz
 Run `tw workspaces -h` to view supported workspace operations.
 Run `tw workspaces add -h` to view the required and optional fields for adding your workspace.
 
-Workspaces provide the context in which a user launches workflow executions, defines the available resources, and manages who can access those resources. Workspaces contain pipelines, runs, actions, datasets, compute environments, credentials, and secrets. Access permissions are controlled with participants, collaborators, and teams.
+Workspaces provide the context in which a user launches workflow executions, defines the available resources, and manages who can access those resources. Workspaces contain pipelines, runs, actions, compute environments, credentials, datasets, data links, and secrets. Access permissions are controlled with participants, collaborators, and teams.
 
 See [User workspaces][user-workspaces] for more information.
 
@@ -484,6 +484,138 @@ $ tw  participants update --name=collaborator@mydomain.com --type=COLLABORATOR -
   Participant 'collaborator@mydomain.com' has now role 'maintain' for workspace 'shared-workspace'
 ```
 
+## Data Links
+
+Run `tw data-links -h` to view supported data link operations.  
+
+Data links allow you to work with public and private cloud storage buckets in Data Explorer in the specified workspace. See [Data Explorer][data-explorer] for more information. AWS S3, Azure Blob Storage, and Google Cloud Storage are supported. The full list of operations are:
+
+- `list`: List data links in a workspace
+- `add`: Add a custom data link to a workspace
+- `update`: Update a custom data link in a workspace
+- `delete`: Delete a custom data link from a workspace
+- `browse`: Browse the contents of a data link in a workspace
+
+### List data links
+
+Run `tw data-links list -h` to view all the optional fields for listing data links in a workspace. If a workspace is not defined, the `TOWER_WORKSPACE_ID` workspace is used by default. Data links can be one of two types:
+
+- `v1-cloud-<id>`: **cloud** data links auto-discovered using credentials attached to the workspace
+- `v1-user-<id>`: **custom** data links created by users
+
+```console
+$ tw data-links list -w seqeralabs/showcase
+
+  Data links at [seqeralabs / showcase] workspace:
+
+ ID                                        | Provider | Name                           | Resource ref                                                    | Region    
+-------------------------------------------+----------+--------------------------------+-----------------------------------------------------------------+-----------
+ v1-cloud-833bb845bd9ec1970c4a7b0bb7b8c4ad | aws      | e2e-data-explorer-tests-aws    | s3://e2e-data-explorer-tests-aws                                | eu-west-2 
+ v1-cloud-60700a33ec3fae68d424cf948fa8d10c | aws      | nf-tower-bucket                | s3://nf-tower-bucket                                            | eu-west-1 
+ v1-user-09705781697816b62f9454bc4b9434b4  | aws      | vscode-analysis-demo           | s3://seqera-development-permanent-bucket/studios-demo/vscode/   | eu-west-2 
+ v1-user-0dede00fabbc4b9e2610261822a2d6ae  | aws      | seqeralabs-showcase            | s3://seqeralabs-showcase                                        | eu-west-1 
+ v1-user-171aa8801cabe4af71500335f193d649  | aws      | projectA-rnaseq-analysis       | s3://seqeralabs-showcase/demo/nf-core-rnaseq/                   | eu-west-1 
+
+<snip>
+
+ v1-user-bb4fa9625a44721510c47ac1cb97905b  | aws      | genome-in-a-bottle             | s3://giab                                                       | us-east-1 
+ v1-user-e7bf26921ba74032bd6ae1870df381fc  | aws      | NCBI_Sequence_Read_Archive_SRA | s3://sra-pub-src-1/                                             | us-east-1 
+
+  Showing from 0 to 99 from a total of 16 entries. 
+```
+
+### Add a custom data link
+
+Run `tw data-links add -h` to view all the required and optional fields for adding a custom data link to a workspace. Users with the `MAINTAIN` role and above for a workspace can add custom data links. The data link `name`, `uri` and `provider` (one of `aws`, `azure`, or `google`) fields are required. If adding a custom data link for a private bucket, the `credentials` identifier field is also required. Adding a custom data link for a public bucket doesn't require credentials.
+
+```console
+$ tw data-links add -w seqeralabs/showcase -n FOO -u az://seqeralabs.azure-benchmarking -p azure -c seqera_azure_credentials
+
+  Data link created:
+
+ ID                                       | Provider | Name | Resource ref                       | Region 
+------------------------------------------+----------+------+------------------------------------+--------
+ v1-user-152116183ee325463901430bb9efb8c9 | azure    | FOO  | az://seqeralabs.azure-benchmarking |        
+```
+
+### Update a custom data link
+
+Run `tw data-links update -h` to view all the required and optional fields for updating a custom data link in a workspace. Users with the `MAINTAIN` role and above for a workspace can update custom data links.
+
+```console
+$ tw data-links update -w seqeralabs/showcase -i v1-user-152116183ee325463901430bb9efb8c9 -n BAR
+
+  Data link updated:
+
+ ID                                       | Provider | Name | Resource ref                       | Region 
+------------------------------------------+----------+------+------------------------------------+--------
+ v1-user-152116183ee325463901430bb9efb8c9 | azure    | BAR  | az://seqeralabs.azure-benchmarking |        
+```
+
+### Delete a custom data link
+
+Run `tw data-links delete -h` to view all the required and optional fields for deleting a custom data link from a workspace. Users with the `MAINTAIN` role and above for a workspace can delete custom data links.
+
+```console
+$ tw data-links delete -w seqeralabs/showcase -i v1-user-152116183ee325463901430bb9efb8c9
+
+  Data link 'v1-user-152116183ee325463901430bb9efb8c9' deleted at '138659136604200' workspace.
+```
+
+### Browse the contents of a data link
+
+Run `tw data-links browse -h` to view all the required and optional fields for browsing a data link in a workspace. Define the data link id using the required `-i` or `--id` argument, which can be found by first using the `list` operation for a workspace. In the example below, a `name` is optionally defined to only retrieve data links with names that start with the given word:
+
+```console
+$ tw data-links list -w seqeralabs/showcase -n 1000genomes
+
+  Data links at [seqeralabs / showcase] workspace:
+
+ ID                                       | Provider | Name        | Resource ref     | Region    
+------------------------------------------+----------+-------------+------------------+-----------
+ v1-user-6d8f44c239e2a098b3e02e918612452a | aws      | 1000genomes | s3://1000genomes | us-east-1 
+
+  Showing from 0 to 99 from a total of 1 entries.
+
+$ tw data-links browse -w seqeralabs/showcase -i v1-user-6d8f44c239e2a098b3e02e918612452a         
+
+  Content of 's3://1000genomes' and path 'null':
+
+ Type   | Name                                       | Size     
+--------+--------------------------------------------+----------
+ FILE   | 20131219.populations.tsv                   | 1663     
+ FILE   | 20131219.superpopulations.tsv              | 97       
+ FILE   | CHANGELOG                                  | 257098   
+ FILE   | README.alignment_data                      | 15977    
+ FILE   | README.analysis_history                    | 5289     
+ FILE   | README.complete_genomics_data              | 5967     
+ FILE   | README.crams                               | 563      
+ FILE   | README.ebi_aspera_info                     | 935      
+ FILE   | README.ftp_structure                       | 8408     
+ FILE   | README.pilot_data                          | 2082     
+ FILE   | README.populations                         | 1938     
+ FILE   | README.sequence_data                       | 7857     
+ FILE   | README_missing_files_20150612              | 672      
+ FILE   | README_phase3_alignments_sequence_20150526 | 136      
+ FILE   | README_phase3_data_move_20150612           | 273      
+ FILE   | alignment.index                            | 3579471  
+ FILE   | analysis.sequence.index                    | 54743580 
+ FILE   | exome.alignment.index                      | 3549051  
+ FILE   | sequence.index                             | 67069489 
+ FOLDER | 1000G_2504_high_coverage/                  | 0        
+ FOLDER | alignment_indices/                         | 0        
+ FOLDER | changelog_details/                         | 0        
+ FOLDER | complete_genomics_indices/                 | 0        
+ FOLDER | data/                                      | 0        
+ FOLDER | hgsv_sv_discovery/                         | 0        
+ FOLDER | phase1/                                    | 0        
+ FOLDER | phase3/                                    | 0        
+ FOLDER | pilot_data/                                | 0        
+ FOLDER | release/                                   | 0        
+ FOLDER | sequence_indices/                          | 0        
+ FOLDER | technical/                                 | 0        
+```
+
 [compute-envs]: https://docs.seqera.io/platform/latest/compute-envs/overview
 [credentials]: https://docs.seqera.io/platform/latest/credentials/overview
 [git-integration]: https://docs.seqera.io/platform/latest/git/overview
@@ -493,3 +625,4 @@ $ tw  participants update --name=collaborator@mydomain.com --type=COLLABORATOR -
 [tower-agent]: https://github.com/seqeralabs/tower-agent
 [user-workspaces]: https://docs.seqera.io/platform/latest/orgs-and-teams/workspace-management
 [wave-docs]: https://www.nextflow.io/docs/latest/wave.html
+[data-explorer]: https://docs.seqera.io/platform/latest/data/data-explorer