Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation update #55

Merged
merged 73 commits into from
Apr 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
70ddccc
Refactoring code, linting terraform, understandable env variables and…
anuradhawick Mar 10, 2023
2b29958
FIX - variable name, and code formatting
anuradhawick Mar 10, 2023
a8ab938
refactored the /info endpoint
anuradhawick Mar 14, 2023
62e726c
refactored the /configuration endpoint
anuradhawick Mar 14, 2023
37eaafb
Refactoring /info, /map and /configuration
anuradhawick Mar 14, 2023
ecd5cfa
Refactor /entry_types and /analyses endpoints
anuradhawick Mar 14, 2023
287bd11
refactoring /biosamples endpoint
anuradhawick Mar 14, 2023
f09defb
refactor runs, individuals, and biosamples
anuradhawick Mar 14, 2023
3c10b65
Refactoring code, linting terraform, understandable env variables and…
anuradhawick Mar 10, 2023
48a0bd3
FIX - variable name, and code formatting
anuradhawick Mar 10, 2023
56a13d0
rebasing with dev
anuradhawick Mar 15, 2023
f7a77f8
refactored the /configuration endpoint
anuradhawick Mar 14, 2023
97850b7
Refactoring /info, /map and /configuration
anuradhawick Mar 14, 2023
e6a838f
Refactor /entry_types and /analyses endpoints
anuradhawick Mar 14, 2023
6a4b8b0
refactoring /biosamples endpoint
anuradhawick Mar 14, 2023
b7165ec
refactor runs, individuals, and biosamples
anuradhawick Mar 14, 2023
0fec1d9
refactor biosamples, cohorts and function signatures for easy error h…
anuradhawick Mar 15, 2023
da3b2f2
Refactoring Cohorts
anuradhawick Mar 15, 2023
da96d12
merger
anuradhawick Mar 15, 2023
bf26376
Refactored all major endpoints
anuradhawick Mar 15, 2023
0c7ea30
full pydantic parsing of requests
anuradhawick Mar 15, 2023
187dee0
Update entity search conditions, TODO - update with passing Class as …
anuradhawick Mar 15, 2023
2167015
refactor response builder
anuradhawick Mar 15, 2023
96edcf8
Dynamic variables for all Beacon/Organisation variables
anuradhawick Mar 15, 2023
740f6d0
Merge pull request #41 from anuradhawick/refactoring
anuradhawick Mar 15, 2023
d758cd8
FIX - treating Similarity.EXACT and Similarity.HIGH equally when desc…
anuradhawick Mar 16, 2023
07e456c
Merge branch 'aehrc:refactoring' into refactoring
anuradhawick Mar 16, 2023
df616e9
Update readme
anuradhawick Mar 16, 2023
42a52e4
Black formatting and setting Include Descendant Terms to True by default
anuradhawick Mar 16, 2023
70888f5
add lru cache for simple functions generating dictionaries
anuradhawick Mar 16, 2023
a736503
Refactored environment variables into a single util
anuradhawick Mar 16, 2023
f505049
Remove unused import
anuradhawick Mar 16, 2023
6b209d8
Refactoring environment variables
anuradhawick Mar 17, 2023
e05e34d
Refactor symlinks into a single lambda layer
anuradhawick Mar 17, 2023
531e5e9
Refactor shared code into python a single module and put them in a layer
anuradhawick Mar 20, 2023
9bb51fe
update simulator TODO - improve cohorts, datasets
anuradhawick Mar 20, 2023
f6ef674
remove stale file
anuradhawick Mar 20, 2023
e812d09
Add cache tables
anuradhawick Mar 20, 2023
524ca51
Refactor simulation variables
anuradhawick Mar 20, 2023
986654e
Refactor simulation variables
anuradhawick Mar 20, 2023
d39a076
CTAS based queries for all entities
anuradhawick Mar 21, 2023
f9aafe4
Refactor variant query communication approach to be lazy with futures…
anuradhawick Mar 22, 2023
4b4c2e0
merger refactoring branch to dev
anuradhawick Mar 22, 2023
bc20e97
remove stale file
anuradhawick Mar 22, 2023
087fac1
Merge branch 'aehrc:dev' into dev
anuradhawick Mar 22, 2023
e3250da
Merge pull request #46 from anuradhawick/dev
anuradhawick Mar 22, 2023
8e2fa25
minor refactoring
anuradhawick Mar 22, 2023
17ce11d
Merge branch 'dev' of https://github.com/anuradhawick/terraform-aws-s…
anuradhawick Mar 22, 2023
ca89a06
Change upload destinations to support CTAS queries
anuradhawick Mar 30, 2023
248b3bd
refactor code for simulator
anuradhawick Mar 30, 2023
6d4ed35
Merge pull request #47 from anuradhawick/dev
anuradhawick Mar 30, 2023
98097f3
improvements to data submission performance
anuradhawick Mar 30, 2023
20b6191
Significantly faster submission times (<10s for 2504 samples)
anuradhawick Mar 30, 2023
58a5647
Refactor use of executors
anuradhawick Mar 30, 2023
6d45a83
refactored submit dataset, addd submit cohort endpoint, minor perform…
anuradhawick Mar 31, 2023
93faacd
Refactored the variant querying code
anuradhawick Apr 4, 2023
ee6330b
Refactoring
anuradhawick Apr 4, 2023
33b6abc
interrim commit before batching the variant querying requests
anuradhawick Apr 4, 2023
052d93e
Faster variant queries with retries
anuradhawick Apr 5, 2023
8dff566
Bug fix
anuradhawick Apr 5, 2023
c506406
Merge pull request #49 from anuradhawick/dev
anuradhawick Apr 5, 2023
c23b293
Cleanup
anuradhawick Apr 12, 2023
064715f
compress based on payload size (stringified)
anuradhawick Apr 12, 2023
55b29d5
Merge pull request #50 from anuradhawick/dev
anuradhawick Apr 12, 2023
6fcd62c
Improved query builder/parser
anuradhawick Apr 13, 2023
46010b7
performance tuning and payload compression bug fix
anuradhawick Apr 18, 2023
75a67d3
Merge branch 'aehrc:dev' into dev
anuradhawick Apr 18, 2023
70660be
Merge remote-tracking branch 'origin/master' into dev
anuradhawick Apr 18, 2023
0889d3f
Merge pull request #51 from anuradhawick/dev
anuradhawick Apr 18, 2023
5cc9da4
documentation update and logo URL
anuradhawick Apr 19, 2023
5627620
Merge pull request #53 from anuradhawick/dev
anuradhawick Apr 19, 2023
a4dfa2b
merge
anuradhawick Apr 19, 2023
2aa6808
Merge pull request #54 from anuradhawick/dev
anuradhawick Apr 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 8 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Serverless Beacon
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

![assets/logo-black.png](assets/logo-black.png)

## Why serverless?
Serverless means the service does not require any servers to be provisioned. The
idea is to minimise running costs, as well as support arbitrary scalablility. It
Expand All @@ -21,7 +23,7 @@ cd terraform-aws-serverless-beacon

### Option 1: Setting up the development environment

Skip to next section if you're only interested in deployment or using a different architecture compared to AWS lambda environment.
Skip to next section if you're only interested in deployment or using a different architecture compared to AWS lambda environment. The following setup must be performed on a latest Amazon Linux instance to match the lambda runtimes. If this is not a viable option, please resort to using Docker.

Run the following shell commands to setup necessary build tools. Valid for Amazon Linux development instances.

Expand All @@ -42,11 +44,11 @@ make
sudo make install
```

Make sure you have a terraform version newer than `Terraform v1.1.6` if you're not using the docker image. Run the following command to get the terraform binary.
Make sure you have a terraform version newer than `Terraform v1.3.7` if you're not using the docker image. Run the following command to get the terraform binary.

```bash
wget https://releases.hashicorp.com/terraform/1.2.8/terraform_1.2.8_linux_386.zip
sudo unzip terraform_1.2.8_linux_386.zip -d /usr/bin/
wget https://releases.hashicorp.com/terraform/1.2.8/terraform_1.3.7_linux_386.zip
sudo unzip terraform_1.3.7_linux_386.zip -d /usr/bin/
```

### Option 2: Using the docker image
Expand Down Expand Up @@ -98,12 +100,6 @@ terraform apply
Your beacon deployment could be a part of a larger program with a front-end and other services. In that case, on the parent folder that the repo folder resides, create a `main.tf` file.
```bash
# main.tf

provider "aws" {
# aws region
region = "REGION"
}

module "serverless-beacon" {
# repo folder
source = "./terraform-aws-serverless-beacon"
Expand All @@ -119,8 +115,8 @@ module "serverless-beacon" {
# aws region
region = "REGION"
}

```
Please refer to [./examples](./examples) to find a minimal and a complete setup.
## Development

All the layers needed for the program to run are in layers folder. To add a new layer for immediate use with additional configs, run the following commands. Once the decision to use the library is finalised update the `init.sh` script to automate the process.
Expand Down Expand Up @@ -149,7 +145,7 @@ cp lib terraform-aws-serverless-beacon/layers/<Library Name>/

* Collaborative development

Please make a copy of `backend.tf.template` with suited parameters and rename as `backend.tf`. Refer to documentation for more information [https://www.terraform.io/language/settings/backends/configuration](https://www.terraform.io/language/settings/backends/configuration).
Please make a copy of `backend.tf.template` with suited parameters and rename as `backend.tf`. Refer to documentation for more information [https://www.terraform.io/language/settings/backends/configuration](https://www.terraform.io/language/settings/backends/configuration). If this is not done, make sure the terraform lock and state files are stored securely to avoid infrastructure-vs-code inconsistencies. Please refer to [./examples/full](./examples/full) to find a an example backend.

## API

Expand Down
Binary file added assets/logo-black.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/logo-tile.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 3 additions & 4 deletions backend.tf.template
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
terraform {
backend "s3" {
bucket = "sbeacon-terraform-states"
key = "sbeacon"
region = "ap-southeast-2"
bucket = "sbeacon-terraform-states"
key = "sbeacon"
region = "ap-southeast-2"
dynamodb_table = "terraform-state-locks"
}
}

8 changes: 8 additions & 0 deletions examples/full/backend.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
terraform {
backend "s3" {
bucket = "sbeacon-terraform-states"
key = "sbeacon"
region = "ap-southeast-2"
dynamodb_table = "terraform-state-locks"
}
}
38 changes: 29 additions & 9 deletions examples/full/main.tf
Original file line number Diff line number Diff line change
@@ -1,15 +1,35 @@
provider "aws" {
region = "ap-southeast-2"
}

module "serverless-beacon" {
region = "us-east-1"
source = "../.."
beacon-id = "au.csiro-serverless.beacon"
beacon-name = "CSIRO Serverless Beacon"
organisation-id = "CSIRO"
organisation-name = "CSIRO"
common-tags = {
stack = "serverless-beacon"
stack = "serverless-beacon"
environment = "dev"
}
variants-bucket-prefix = "sbeacon-variants-"
metadata-bucket-prefix = "sbeacon-metadata-"
lambda-layers-bucket-prefix = "sbeacon-lambda-layers-"
beacon-id = "au.csiro.sbeacon"
beacon-name = "CSIRO Serverless Beacon"
beacon-api-version = "v2.0.0"
beacon-environment = "dev"
beacon-description = "Serverless Beacon (sBeacon)"
beacon-version = "v0.1.0"
beacon-welcome-url = "https://bioinformatics.csiro.au/"
beacon-alternative-url = "https://bioinformatics.csiro.au/"
beacon-create-datetime = "2018-11-26H00:00:00Z"
beacon-update-datetime = "2023-03-16H00:00:00Z"
beacon-handovers = "[]"
beacon-documentation-url = "https://github.com/EGA-archive/beacon2-ri-api"
beacon-default-granularity = "boolean"
beacon-uri = "https://beacon.csiro.au"
organisation-id = "CSIRO"
organisation-name = "CSIRO"
beacon-org-description = "CSIRO, Australia"
beacon-org-address = "AEHRC, Westmead NSW, Australia"
beacon-org-welcome-url = "https://beacon.csiro.au"
beacon-org-contact-url = "https://bioinformatics.csiro.au/get-in-touch/"
beacon-org-logo-url = "https://raw.githubusercontent.com/aehrc/terraform-aws-serverless-beacon/master/assets/logo-tile.png"
beacon-service-type-group = "au.csiro"
beacon-service-type-artifact = "beacon"
beacon-service-type-version = "1.0"
}
2 changes: 1 addition & 1 deletion examples/full/outputs.tf
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
output "api_url" {
value = module.serverless-beacon.api_url
value = module.serverless-beacon.api_url
description = "URL used to invoke the API."
}
17 changes: 8 additions & 9 deletions examples/minimum/main.tf
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
provider "aws" {
region = "ap-southeast-2"
}

module "serverless-beacon" {
source = "../.."
beacon-id = "au.csiro-serverless.beacon"
beacon-name = "CSIRO Serverless Beacon"
organisation-id = "CSIRO"
organisation-name = "CSIRO"
source = "../.."
beacon-id = "au.csiro-serverless.beacon"
variants-bucket-prefix = "sbeacon-"
metadata-bucket-prefix = "sbeacon-metadata-"
lambda-layers-bucket-prefix = "sbeacon-lambda-layers-"
beacon-name = "CSIRO Serverless Beacon"
organisation-id = "CSIRO"
organisation-name = "CSIRO"
}
2 changes: 1 addition & 1 deletion examples/minimum/outputs.tf
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
output "api_url" {
value = module.serverless-beacon.api_url
value = module.serverless-beacon.api_url
description = "URL used to invoke the API."
}
4 changes: 2 additions & 2 deletions variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ variable "beacon-org-address" {
variable "beacon-org-welcome-url" {
type = string
description = "Value for beacon organisation welcome url"
default = "https://beacon.csiro.au"
default = "https://bioinformatics.csiro.au/"
}

variable "beacon-org-contact-url" {
Expand All @@ -155,7 +155,7 @@ variable "beacon-org-contact-url" {
variable "beacon-org-logo-url" {
type = string
description = "Value for beacon organisation logo url"
default = "https://bioinformatics.csiro.au/covid-19-sbeacon/"
default = "https://raw.githubusercontent.com/aehrc/terraform-aws-serverless-beacon/master/assets/logo-tile.png"
}

# Beacon service variables
Expand Down