Repository for common actions used in Sneaks And Data CI/CD processes
Available actions are:
- semver_release
- install_poetry
- build_helm_chart
- create_package
- generate_version
- install_azcopy
- login_to_aks
- deploy_poetry_project_to_azfs
- deploy_dbt_project_to_azfs
- deploy_data_schemas_to_azfs
- run_azcopy
- get_azure_share_sas
- setup_gh_app
- update_airflow_variables
- contribute_changes
- activate_workflow
- setup_aws_ca
Creates a new GitHub release based on git tags and semantic versioning
NOTE: This action uses github.ref
variable for target branch name (see: https://cli.github.com/manual/gh_release_create).
Name | Description | Optional |
---|---|---|
major_v | major version of current release | False |
minor_v | minor version of current release | False |
Name | Description |
---|---|
version | generated new version of the release |
name: Release a new version
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Create Release
uses: SneaksAndData/github-actions/semver_release@v0.0.17
with:
major_v: 0
minor_v: 0
Installs poetry to build environment and restores dependencies using custom and private pypi indices. Optionally can export dependency tree to requirements.txt file.
Name | Description | Optional | Default value |
---|---|---|---|
pypi_repo_url | URL of python package index (for custom packages) | True | "" |
pypi_token_username | Package index authentication username. | True | "" |
pypi_token | Package index authentication token or password. | True | "" |
export_requirements | Set to true if need to generate requirements.txt. Optional defaults to false. |
True | false |
export_credentials | If export_requirements is set to true, it exports requirements.txt with --with-credentials flag. Otherwise, does nothing. | True | true |
requirements_path | Path to requirements.txt to be generated (relative to sources root). | True | .container/requirements.txt |
install_preview | Install preview version of Poetry. | True | false |
version | Version to install. If value is 'latest', script will install the latest available version of Poetry. | True | latest |
install_extras | List of optional dependencies to install, separated by space. If value is 'all', all extras will be installed | True | |
install_only_dependencies | If set to true, installs only dependencies for project, adds the parameter --no-root to poetry install command. |
True | false |
skip_dependencies | If set to true, installs only poetry without installing dependencies. | True | false |
export_dev_requirements | If export_requirements is set to true, it exports dev requirements.txt with --without-dev flag. Otherwise, does nothing. | True | true |
no_binary_dependencies | Dependencies that must be built from source - equivalent to installer.no-binary setting in Poetry. Example: "bottleneck,numpy" | True | "" |
Name | Description |
---|---|
custom_repo_name | Name of configured custom repository for poetry push command |
name: Install poetry and package dependencies
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Install Poetry and dependencies
uses: SneaksAndData/github-actions/install_poetry@v0.0.17
with:
pypi_repo_url: ${{ secrets.AZOPS_PYPI_REPO_URL }}
pypi_token_username: ${{ secrets.AZOPS_PAT_USER }}
pypi_token: ${{ secrets.AZOPS_PAT }}
export_requirements: True # if you want to generate requirements.txt
requirements_path: ".container/requirements.txt"
install_extras: "azure datadog"
Allows to build helm chart and push it to remote container repository.
NOTE: to be able to use this action, your repository should contain version tags. This action relies on git tags to be present in order to generate an artifact tag.
Name | Description | Optional | Default value |
---|---|---|---|
container_registry_address | Container registry address | False | |
application: | Application name | False | |
container_registry_user | Container registry username | False | |
container_registry_token | Container registry access token | False | |
helm_version | Version of helm to install | True | 3.9.2 |
helm_directory | Location of helm chart related to project root | True | .helm |
app_version | Application version to use for the chart. If omitted, the latest tag will be used. | True | |
chart_version | Chart version to use for the chart. If omitted, the latest tag will be used. | True |
No outputs defined
name: Build and publish Helm chart
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Build and Push Chart (DEV)
uses: SneaksAndData/github-actions/build_helm_chart@v0.0.17
with:
application: beast
container_registry_user: ${{secrets.AZCR_DEV_USER}}
container_registry_token: ${{secrets.AZCR_DEV_TOKEN}}
container_registry_address: ${{secrets.AZCR_DEV_USER}}
Creates a development version of a python package according to PEP-440 from an open pull request and uploads it to a provided python index.
Version format is {Major}.{Minor}.{Patch}a{PR_NUMBER}dev{COMMENT_NUMBER}
where PR_NUMBER is number of pull request and
COMMENT_NUMBER is number of comment which triggered a build.
NOTES:
- To use this action, your repository should contain version tags. This action relies on git tags to be present in order to generate an artifact tag.
- This action should be placed in separate job with issue_comment pull request trigger. (see Usage below)
- This action requires to poetry ~1.2 being installed in build environment (for example, by install_poetry action)
⚠️ If the inputrepo_url
is not provided, this action will push the package to a public repository (https://pypi.org). In this case, the inputpublic_package_index_token
should be provided.
Name | Description | Optional | Default value |
---|---|---|---|
repo_url | Package index URL | True | Empty |
repo_token_username | Package index authentication username | True | Empty |
repo_token | Package index authentication token or password. | True | Empty |
package_name | Name of package to create. This should match name of root project directory | False | |
version | Version of package. If not provided, a new development version will be generated | Empty | |
public_package_index_token | Access token for publishing to a public repository (https://pypi.org) | True | Empty |
No outputs defined
Add to .github/workflows
a workflow file defined as follwoing and replace value in property package_name
with name
of root folder of your package:
on: issue_comment
jobs:
pr_commented:
name: Build package on PR comment
runs-on: ubuntu-latest
if: ${{ github.event.issue.pull_request && github.event.comment.body == 'create_package' && github.event.issue.state == 'open' }}
steps:
- uses: actions/checkout@v2
with:
ref: refs/pull/${{github.event.issue.number}}/merge
fetch-depth: 0
- name: Install Poetry and dependencies
uses: SneaksAndData/github-actions/install_poetry@v0.0.17
with:
pypi_repo_url: ${{ secrets.AZOPS_PYPI_REPO_URL }}
pypi_token_username: ${{ secrets.AZOPS_PAT_USER }}
pypi_token: ${{ secrets.AZOPS_PAT }}
skip_dependencies: true
- name: Create package
uses: SneaksAndData/github-actions/create_package@v0.0.17
with:
pypi_repo_url: ${{ secrets.AZOPS_PYPI_UPLOAD }}
pypi_token_username: ${{ secrets.AZOPS_PAT_USER }}
pypi_token: ${{ secrets.AZOPS_PAT }}
package_name: python_project
Generates project version based on current git commit and git tags.
NOTES:
- To use this action, your repository should contain version tags. This action relies on git tags to be present in order to generate a version.
- Generated version is will not be compatible with PEP-440, so this versions should not be used with python packages. Although, this action can be used with source code deployments of python applications.
No inputs defined
Name | Description |
---|---|
version | generated version string |
name: Print version
on:
workflow_dispatch
jobs:
print_version:
name: print version
runs-on: ubuntu-latest
if: ${{ github.ref != 'refs/heads/main' }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Get project version
uses: SneaksAndData/github-actions/generate_version@v0.0.17
id: version
- run: echo "$version"
env:
version: ${{steps.version.outputs.version}}
Installs azcopy v10 in current build directory.
No inputs defined
No outputs defined
name: Install azcopy
on:
workflow_dispatch:
jobs:
install_azcopy:
name: install azcopy
runs-on: ubuntu-latest
steps:
- name: Install azcopy v10
uses: SneaksAndData/github-actions/install_azcopy@v0.0.17
Get AKS login credentials for kubectl. Service principal used must have a permission to list cluster credentials.
Name | Description | Optional | Default value |
---|---|---|---|
cluster_sp_client_id | Cluster service principal application id | False | |
cluster_sp_client_password | Cluster service principal password | False | |
tenant_id | Azure tenant ID | False | |
subscription_id | Azure subscription ID | False | |
cluster_name | Name of the cluster | False |
No outputs defined
name: Login to AKS
on:
jobs:
login_to_aks:
name: Login to AKS
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Get cluster credentials
uses: SneaksAndData/github-actions/login_to_aks@v0.0.17
with:
cluster_sp_client_id: $AZURE_CLIENT_ID
cluster_sp_client_password: $AZURE_CLIENT_SECRET
tenant_id: $AZURE_TENANT_ID
subscription_id: $AZURE_SUBSCRIPTION_ID
cluster_name: $AZURE_AKS_NAME
Copy python site-packages of current virtual environment and installs application into it.
Name | Description | Optional | Default value |
---|---|---|---|
project_version | Version of the project | False | |
project_name | Name of the project | False | |
project_directory | Directory name inside the project (if differs from project name) | True | "" |
destination | Directory or SAS for upload | False | |
python_version | Project python version | True | 3.9 |
deployment_root | Root directory in the file share | False |
NOTES:
- To use this action, your project should use poetry for virtual environment management. Ensure that you installed the latest version of poetry and project dependencies (for instance, by install_poetry action).
No outputs defined
name: Prepare python deployment
on:
workflow_dispatch:
jobs:
prepare_deployment:
name: Prepare python code for deployment
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Get project version
uses: SneaksAndData/github-actions/generate_version@v0.0.17
id: version
- name: Generate SAS for upload
uses: SneaksAndData/github-actions/get_azure_share_sas@v0.0.17
with:
directory_name: share-name/path/within/share
account_key: ${{ secrets.ACCOUNT_KEY }}
account_name: ${{ secrets.ACCOUNT_NAME }}
id: sas
- name: Prepare site-packages for deployment
uses: SneaksAndData/github-actions/deploy_poetry_project_to_azfs@v0.0.17
with:
deployment_root: /python
project_version: ${{ steps.version.outputs.version }}
destination: ${{ steps.sas.outputs.authorized_destination }}
project_name: python_project
Prepare DBT models for deployment to an Azure file share.
Name | Description | Optional | Default value |
---|---|---|---|
project_version | Version of the project | False | |
project_name | Name of the project | False | |
destination | Directory or SAS for upload | False | |
deployment_root | Root directory in the file share | False |
No outputs defined
name: Prepare deployment
on:
workflow_dispatch:
jobs:
prepare_deployment:
name: Prepare dbt output for deployment
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Get project version
uses: SneaksAndData/github-actions/generate_version@v0.0.17
id: version
- name: Generate SAS for upload
uses: SneaksAndData/github-actions/get_azure_share_sas@v0.0.17
with:
directory_name: share-name/path/within/share
account_key: ${{ secrets.ACCOUNT_KEY }}
account_name: ${{ secrets.ACCOUNT_NAME }}
id: sas
- name: Prepare dbt for deployment
uses: SneaksAndData/github-actions/deploy_dbt_project_to_azfs@v0.0.17
with:
deployment_root: /dbt
project_version: ${{ steps.version.outputs.version }}
destination: ${{ steps.sas.outputs.authorized_destination }}
project_name: dbt_project
Prepare DBT schemas for deployment to an Azure file share.
Name | Description | Optional | Default value |
---|---|---|---|
project_version | Version of the project | False | |
project_name | Name of the project | False | |
destination | Directory or SAS for upload | False | |
deployment_root | Root directory in the file share | False |
No outputs defined
name: Prepare deployment
on:
workflow_dispatch:
jobs:
validate_commit:
name: Prepare schemas output for deployment
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Get project version
uses: SneaksAndData/github-actions/generate_version@v0.0.17
id: version
- name: Generate SAS for upload
uses: SneaksAndData/github-actions/get_azure_share_sas@v0.0.17
with:
directory_name: share-name/path/within/share
account_key: ${{ secrets.ACCOUNT_KEY }}
account_name: ${{ secrets.ACCOUNT_NAME }}
id: sas
- name: Prepare dbt for deployment
uses: SneaksAndData/github-actions/deploy_data_schemas_to_azfs@v0.0.17
with:
deployment_root: /dbt
project_version: ${{ steps.version.outputs.version }}
destination: ${{ steps.sas.outputs.authorized_destination }}
project_name: dbt_project
Invoke azcopy copy command in pipeline.
Name | Description | Optional | Default value |
---|---|---|---|
source | Source directory or SAS url to copy | False | |
target | Target directory or SAS url | False | |
mode | azcopy action mode (copy or sync) | True | copy |
put_md5 | If true sets --put-md5 parameter to azcopy |
True | True |
delete_destination | azcopy --delete-destination flag | True | False |
No outputs defined
name: Copy files
on:
workflow_dispatch:
jobs:
copy_files:
name: Copy files
steps:
- name: Generate SAS for upload
uses: SneaksAndData/github-actions/get_azure_share_sas@v0.0.17
with:
directory_name: share-name/path/within/share
account_key: ${{ secrets.ACCOUNT_KEY }}
account_name: ${{ secrets.ACCOUNT_NAME }}
id: sas
- name: Copy data
uses: SneaksAndData/github-actions/run_azcopy@v0.0.17
with:
source: source/directory/on/build/agent
target: ${{ steps.sas.outputs.authorized_destination }}
Generates new temporary Shared Access Signature for a file share, attached to a storage account.
Name | Description | Optional | Default Value |
---|---|---|---|
directory_name | Path within file share | False | |
account_key | Name of the storage account of the share | False | |
account_name | Key of the storage account of the share | False | |
expiration_date | Expiration date in format that can be used by the date command |
True | +10 minutes |
directory_type | Type of directory (blob or fileshare) | True | fileshare |
NOTES:
- For the expiration date format see man 1 date
Name | Description |
---|---|
authorized_destination | URL of the file share with attached shared access signature |
name: Release a new version
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Generate SAS for upload
uses: SneaksAndData/github-actions/get_azure_share_sas@v0.0.17
with:
directory_name: share-name/path/within/share
account_key: ${{ secrets.ACCOUNT_KEY }}
account_name: ${{ secrets.ACCOUNT_NAME }}
id: sas
- name: Copy data
uses: SneaksAndData/github-actions/run_azcopy@v0.0.17
with:
source_directory: source/directory/on/build/agent
target: ${{ steps.sas.outputs.authorized_destination }}
Configure git client in the workflow job to authenticate to GitHub using a GitHub App instead of builtin repo-scoped GITHUB_TOKEN. next workflow steps.
Name | Description | Optional | Default Value |
---|---|---|---|
app_private_key | Private key of application | False | |
app_installation_id | Application installation Id | False | |
git_user_email | User email for git client | False | |
git_user_name | User name for git client | False |
Name | Description |
---|---|
access_token | Access token generated by GitHub API |
name: Checkout repo using github app
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Reconfigure Git to use App
id: setup_gh_app
uses: SneaksAndData/github-actions/setup_gh_app@v0.0.17
with:
app_private_key: ${{ secrets.APP_PRIVATE_KEY }}
app_installation_id: 1234567
app_id: 89123
git_user_name: "Github App"
git_user_email: "user@example.com"
- name: Checkout Airflow Variables
uses: actions/checkout@v3
with:
repository: SneaksAndData/airflow-variables
token: ${{ steps.setup_gh_app.outputs.access_token }}
path: airflow-variables
fetch-depth: 0
Update airflow configuration of a specified dbt project to the current version (git tag) and provided execution graph
Name | Description | Optional | Default Value |
---|---|---|---|
project_name | Name of the project | False | |
project_version | Version of the project | False | |
project_graph | Graph generated by metadata-manager | False | |
working_directory | User name for git client | False | |
airflow_variable | JSON-encoded airflow variable | False |
Name | Description | Optional | Default Value |
---|---|---|---|
airflow_variable | JSON-encoded airflow variable | False |
name: Update airflow-variables
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Update Project Version
uses: SneaksAndData/github-actions/update_airflow_variables@v0.0.17
id: update_variables
with:
project_name: dbt-project
working_directory: github-repository
project_version: 0.0.1
project_graph: "{}"
Create pull request in repository
Name | Description | Optional | Default Value |
---|---|---|---|
project_name | Name of the project | False | |
project_version | Version of the project | False | |
working_directory | Directory with airflow-variables GitHub repo | False | |
access_token | An access token with push and create pull request permissions | False | |
merge | True if action should merge changes. Otherwise just create a branch and open a Pull Request | False |
No outputs defined
name: Create pull request
on:
workflow_dispatch:
jobs:
contribute_changes:
runs-on: ubuntu-latest
steps:
- name: Contribute Variable Changes
uses: SneaksAndData/github-actions/contribute_changes@v0.0.17
with:
project_name: dbt-project
working_directory: github-repository
project_version: 0.0.1
access_token: ${{ secrets.ACCESS_TOKEN }}
merge: false
Triggers a specified GitHub Workflow file with parameters.
Name | Description | Optional | Default Value |
---|---|---|---|
access_token | An access token with push and create pull request permissions | False | |
repo_name | Repository to deploy | False | |
workflow_name | Name of the workflow to activate | False | |
deploy_environment | Environment name to deploy | True | production |
No outputs defined
name: Deploy latest tag
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Deploy variables
uses: SneaksAndData/github-actions/activate_workflow@v0.0.17
with:
access_token: ${{ secrets.ACCESS_TOKEN }}
repo_name: github-repo
workflow_name: Deploy Variables to airflow
Create pull request in repository
Name | Description | Optional | Default Value |
---|---|---|---|
access_token | An access token with push and create pull request permissions | False | |
run_title | Repository to deploy | False | |
repo_name | Repository name | False | |
workflow_name | Name of the workflow | False | |
branch_name | Name of the branch | True | main |
No outputs defined
name: Deploy latest tag
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Deploy variables
uses: SneaksAndData/github-actions/activate_workflow@v0.0.17
with:
access_token: ${{ secrets.ACCESS_TOKEN }}
run_title: "Updating Project github-repo to version 1.1.1"
repo_name: github-repo
workflow_name: Prepare Helm chart
Read airflow variable, escape newlines for using content in other steps.
Name | Description | Optional | Default Value |
---|---|---|---|
project_name | Project name | False | |
root_directory | Root directory with variables repository | False | |
variables_sub_path | Subdirectory with JSON-encoded file with variable | False |
Name | Description |
---|---|
airflow_variable | JSON-encoded airflow variable |
name: Deploy latest tag
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Read airflow variable
uses: SneaksAndData/github-actions/read_airflow_variable@v0.0.17
with:
project_name: ${{ env.PROJECT_NAME }}
id: read
Setup AWS CodeArtifact credentials
Name | Description | Optional | Default Value |
---|---|---|---|
aws_access_key | AWS access key | False | |
aws_access_key_id | AWS access key ID | False | |
mode | Setup for read or publish | False | |
aws_ca_domain | AWS CodeArtifact domain | False | |
aws_ca_domain_owner | AWS CodeArtifact domain owner name | False | |
aws_ca_repository | AWS CodeArtifact repository name | False | |
aws_region | AWS region where the artifact storage is located | True | eu-central-1 |
Name | Description |
---|---|
url | Python artifact storage URL (pip or twine-compatible) |
user | User Name |
token | Access token |
name: Deploy latest tag
on:
workflow_dispatch:
jobs:
create_release:
runs-on: ubuntu-latest
steps:
- name: Setup AWS CA
uses: SneaksAndData/github-actions/setup_aws_ca@v0.1.1
with:
aws_access_key: ${{ env.AWS_ACCESS_KEY }}
aws_access_key_id: ${{ env.AWS_ACCESS_KEY_ID }}
mode: read
aws_ca_domain: some-domain
aws_ca_domain_owner: some-domain-owner
aws_ca_repository: some-repository
id: aws_ca
- name: Install Poetry and dependencies
uses: SneaksAndData/github-actions/install_poetry@v0.1.0
with:
pypi_repo_url: ${{ steps.aws_ca.outputs.url }}
pypi_token_username: ${{ steps.aws_ca.outputs.user }}
pypi_token: ${{ steps.aws_ca.outputs.token }}