okdata-pipeline

Collection of pipeline components for Origo Dataplattform.

Components

• converters
  • csv
  • json
  • xls
• lambda_invoker
• validators
  • csv
  • json
• writers
  • s3

Setup

1. Install Serverless Framework
2. python3.11 -m venv .venv
3. Install Serverless plugins: make init
4. Install Python toolchain: python3 -m pip install (--user) tox black pip-tools
   • If running with --user flag, add $HOME/.local/bin to $PATH

Formatting code

Code is formatted using black: make format.

Running tests

Tests are run using tox: make test.

For tests and linting we use pytest, flake8, and black.

Deploy

Deploy to both dev and prod is automatic via GitHub Actions on push to main.

You can also deploy from a local machine to dev with:

make deploy

Or to prod with:

make deploy-prod