Release v2.3.0: Bump users-package version to 3.0 as well as general bug fixes and improvements

.github/workflows/pr.yaml

@@ -11,20 +11,20 @@ on:
 
 jobs:
   changelog:
-    uses: obervinov/_templates/.github/workflows/changelog.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/changelog.yaml@v1.2.9
 
   pylint:
-    uses: obervinov/_templates/.github/workflows/pylint.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/pylint.yaml@v1.2.9
 
   pytest:
-    uses: obervinov/_templates/.github/workflows/pytest-with-vault.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/pytest-with-vault.yaml@v1.2.9
 
   pyproject:
-    uses: obervinov/_templates/.github/workflows/pyproject.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/pyproject.yaml@v1.2.9
 
   pr:
-    uses: obervinov/_templates/.github/workflows/pr.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/pr.yaml@v1.2.9
 
   build-pr-image:
-    uses: obervinov/_templates/.github/workflows/docker.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/docker.yaml@v1.2.9
     needs: [changelog, pylint, pytest, pyproject]

.github/workflows/release.yaml

@@ -10,14 +10,14 @@ on:
 
 jobs:
   create-release:
-    uses: obervinov/_templates/.github/workflows/release.yaml@v1.2.8
+    uses: obervinov/_templates/.github/workflows/release.yaml@v1.2.9
 
   cleanup-untagged-images:
     runs-on: ubuntu-latest
     steps:
       - name: Delete untagged images from GitHub Container Registry
         continue-on-error: true
-        uses: Chizkiyahu/delete-untagged-ghcr-action@v3
+        uses: Chizkiyahu/delete-untagged-ghcr-action@v4
         with:
           token: ${{ secrets.PAT_GHCR_CLEANUP }}
           package_name: 'pyinstabot-downloader'
@@ -26,5 +26,5 @@ jobs:
           owner_type: 'user'
 
   # milestone:
-  #   uses: obervinov/_templates/.github/workflows/milestone.yaml@v1.2.8
+  #   uses: obervinov/_templates/.github/workflows/milestone.yaml@v1.2.9
   #   needs: [create-release]

CHANGELOG.md

@@ -3,6 +3,24 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/) and this project adheres to [Semantic Versioning](http://semver.org/).
 
 
+## v2.3.0 - 2024-10-04
+### What's Changed
+**Full Changelog**: https://github.com/obervinov/pyinstabot-downloader/compare/v2.2.1...v2.3.0 by @obervinov in https://github.com/obervinov/pyinstabot-downloader/pull/95
+#### 💥 Breaking Changes
+* now all user data is stored in the database
+* psql credentials are now written out via Vault Database Engine
+#### 🚀 Features
+* bump workflow version to `1.2.9`
+* bump vault-package to major version `3.0.0`
+* bump users-package to major version `3.0.2`
+* bump telegram-package to major version `2.0.1`
+* add tests for database and metrics modules
+* add proxy support for all dependencies with `requests` library
+* [Switch reading of the database connection configuration to db engine](https://github.com/obervinov/pyinstabot-downloader/issues/33)
+#### 🐛 Bug Fixes
+* general bug fixes and improvements
+
+
 ## v2.2.1 - 2024-08-24
 ### What's Changed
 **Full Changelog**: https://github.com/obervinov/pyinstabot-downloader/compare/v2.2.0...v2.2.1 by @obervinov in https://github.com/obervinov/pyinstabot-downloader/pull/94

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.9.19-alpine3.20
+FROM python:3.9.20-alpine3.20
 
 ### External argumetns ###
 ARG PROJECT_DESCRIPTION

README.md

@@ -20,7 +20,7 @@
   - [Target storage of the content](#target-storage-of-the-content)
   - [Bot configuration source and supported parameters](#bot-configuration-source-and-supported-parameters)
   - [Bot persistent data storage](#bot-persistent-data-storage)
-- [How to run project](#-how-to-run-project)
+- [How to run project locally](#-how-to-run-project-locally)
 
 
 ## <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/book.png" width="25" title="about"> About this project
@@ -50,22 +50,29 @@ This project is a Telegram bot that allows you to upload posts from your Instagr
 </br>
 
 ## <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/requirements.png" width="25" title="requirements"> Requirements
-- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/vault.png" width="15" title="vault"> Vault server - [a storage of secrets for bot with kv v2 engine](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2)
-- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/dropbox.ico" width="15" title="dropbox"> Dropbox [api token](https://dropbox.tech/developers/generate-an-access-token-for-your-own-account)</img> or <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/mega.png" width="15" title="mega"> Mega.nz [account](https://mega.nz)</img> or <img src="https://github.com/obervinov/_templates/blob/main/icons/file.png" width="15" title="webdav"> WebDav provider [url, username and password](https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html)</img>
-- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/telegram.png" width="15" title="telegram"> Telegram bot api token - [instructions for creating bot and getting a token of api](https://learn.microsoft.com/en-us/azure/bot-service/bot-service-channel-connect-telegram?view=azure-bot-service-4.0)
-- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/instagram.png" width="15" title="instagram"> Instagram username/password - [login and password from the instagram account, it is advisable to create a new account](https://www.instagram.com/accounts/emailsignup/)
-- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/postgres.png" width="15" title="postgresql"> Postgresql - [a storage of project persistent data](https://www.postgresql.org/download/)
+- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/vault.png" width="15" title="vault"> **Vault server**
+   - [store the project configuration in kv2](https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2)
+   - [generate access credentials in the database](https://developer.hashicorp.com/vault/docs/secrets/databases)
+   - [prepare the vault server](scripts/vault-init.sh)
+- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/mega.png" width="15" title="mega"> **Cloud Storage** (choose one)
+   - dropbox: [api token](https://dropbox.tech/developers/generate-an-access-token-for-your-own-account)
+   - mega: [account](https://mega.nz)
+   - webdav: [url, username and password](https://docs.nextcloud.com/server/latest/user_manual/en/files/access_webdav.html)
+- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/telegram.png" width="15" title="telegram"> **Telegram**
+   - bot: [api token](https://learn.microsoft.com/en-us/azure/bot-service/bot-service-channel-connect-telegram?view=azure-bot-service-4.0)
+- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/instagram.png" width="15" title="instagram"> **Instagram** (choose one)
+   - account: [username/password](https://www.instagram.com/accounts/emailsignup/) or [a ready uploaded session from the browser](https://raw.githubusercontent.com/instaloader/instaloader/master/docs/codesnippets/615_import_firefox_session.py)
+- <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/postgres.png" width="15" title="postgresql"> **Postgresql**
+   - database: [empty database](scripts/psql-init.sh)
 </br>
 
 ## <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/build.png" width="25" title="build"> Environment variables
 | Variable  | Description | Default value |
 | ------------- | ------------- | ------------- |
 | `LOGGER_LEVEL` | [The logging level of the logging module](https://docs.python.org/3/library/logging.html#logging-levels) | `INFO` |
-| `BOT_NAME` | The name of the bot, used to determine the unique mount point in the vault | `pyinstabot-downloader` |
+| `TELEGRAM_BOT_NAME` | The name of the bot, used to determine the unique mount point in the vault | `pyinstabot-downloader` |
 | `MESSAGES_CONFIG` | The path to the message template file | `src/configs/messages.json` |
-| `VAULT_ADDR`  | The address at which the vault server will be available to the bot | `None` |
-| `VAULT_APPROLE_ID` | [Approle id created during vault setup](https://developer.hashicorp.com/vault/docs/auth/approle) | `None` |
-| `VAULT_APPROLE_SECRETID`  | [Approle secret id created during vault setup](https://developer.hashicorp.com/vault/docs/auth/approle) | `None` |
+| `VAULT_*` | All supported vault environment variables can be found [here](https://github.com/obervinov/vault-package/tree/v3.0.0?tab=readme-ov-file#-supported-environment-variables) | - |
 </br>
 
 ## <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/config.png" width="25" title="config"> Prepare and configure environment
@@ -87,21 +94,38 @@ This project is a Telegram bot that allows you to upload posts from your Instagr
 </br>
 
 ### Bot configuration source and supported parameters
-<img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/vault.png" width="15" title="vault"> All bot configuration is stored in the `Vault Secrets`</br>
-_except for the part of the configuration that configures the connection to `Vault`_</br>
+<img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/vault.png" width="15" title="vault"> All bot configuration is stored in the `Vault Secrets` (_except for the part of the configuration that configures the connection to `Vault`_)
+</br>
+
+- `pyinstabot-downloader-database` - vault database engine mount point, returns the temporary username and password for the database. More information about the database engine can be found [here](https://developer.hashicorp.com/vault/docs/secrets/databases/postgresql) and [here](https://developer.hashicorp.com/vault/tutorials/db-credentials/database-secrets)
+  </br>
+
 - `configuration/database`: database connection parameters
   ```json
   {
-    "database": "pyinstabot-downloader",
+    "dbname": "pyinstabot-downloader",
     "host": "postgresql.example.com",
-    "password": "qwerty123",
     "port": "5432",
-    "user": "user1",
     "connections": "10"
   }
   ```
   </br>
+
+- `configuration/telegram`: telegram bot configuration
+  ```json
+  {
+    "token": "123456:qwerty"
+  }
+  ```
+  </br>
+
 - `configuration/downloader-api`: downloader module configuration (for downloading content from instagram)
+
+  Clarification of non-obvious parameters
+  - `fatal-status-codes`: a list of status codes that are considered fatal and stop downloader module work
+  - `iphone-support`: if `True` the downloader module will use the iphone user-agent
+  - `login-method`: the method of logging into the instagram account (`session`, `password`, `anonymous`)
+  - `session-base64`: the session file content in base64 format (only for `session` login method)
   ```json
   {
     "enabled": "True",
@@ -115,24 +139,20 @@ _except for the part of the configuration that configures the connection to `Vau
     "username": "username1"
   }
   ```
-  Clarification of non-obvious parameters
-  - `fatal-status-codes`: a list of status codes that are considered fatal and stop downloader module work
-  - `iphone-support`: if `True`, the downloader module will use the iphone user-agent
-  - `login-method`: the method of logging into the instagram account (`session`, `password`, `anonymous`)
-  - `session-base64`: the session file content in base64 format
-  </br>
-- `configuration/telegram`: telegram bot configuration
-  ```json
-  {
-    "token": "123456:qwerty"
-  }
-  ```
   </br>
+
 - `configuration/uploader-api`: uploader module configuration (for upload content to the target storage)
+
+  Clarification of non-obvious parameters
+  - `destination-directory`: the directory in the target storage where the content will be uploaded
+  - `exclude-types`: a list of file extensions that will be excluded from the upload (for example, `.txt` - text from the post)
+  - `source-directory`: the directory where the content will be stored before uploading (temporary directory)
+  - `storage-type`: the type of storage where the content will be uploaded (`dropbox`, `mega`, `webdav`)
+  - `url`: the url of the target webdav directory (only for `webdav` storage)
   ```json
   {
-    "destination-directory": "cloud-directory/",
     "enabled": "True",
+    "destination-directory": "cloud-directory/",
     "exclude-types": "[\".txt\", \".tmp\"]",
     "password": "qwerty123",
     "source-directory": "data/",
@@ -141,25 +161,21 @@ _except for the part of the configuration that configures the connection to `Vau
     "url": "https://webdav.example.com/directory"
   }
   ```
-  Clarification of non-obvious parameters
-  - `destination-directory`: the directory in the target storage where the content will be uploaded
-  - `exclude-types`: a list of file extensions that will be excluded from the upload (for example, `.txt` - text from the post)
-  - `source-directory`: the directory where the content will be stored before uploading (temporary directory)
-  - `storage-type`: the type of storage where the content will be uploaded (`dropbox`, `mega`, `webdav`)
-  - `url`: the url of the target webdav directory (only for `webdav` storage)
   </br>
-- `configuration/users/<telegram_user_id>`: user permissions configuration
+
+- `configuration/users/<telegram_user_id>`: users permissions and attributes
+
+  Clarification of non-obvious parameters
+  - `requests`: the number of requests that the user can make per day and per hour, as well as the random shift in minutes (scheduling of message processing from the queue works on the basis of this parameter)
+  - `roles`: list of roles that allow to use the corresponding functionality ([available roles](src/configs/constants.py#L11-L15)).
+  - `status`: allowed or denied user access to the bot
   ```json
   {
     "requests": "{\"requests_per_day\": 10, \"requests_per_hour\": 1, \"random_shift_minutes\": 60}",
     "roles": "[\"post\", \"posts_list\"]",
     "status": "allowed"
   }
   ```
-  Clarification of non-obvious parameters
-  - `requests`: the number of requests that the user can make per day and per hour, as well as the random shift in minutes (scheduling of message processing from the queue works on the basis of this parameter)
-  - `roles`: list of roles that allow to use the corresponding functionality ([available roles](src/configs/constants.py#L19-L23)).
-  - `status`: allowed or denied user access to the bot
 
 #### You can use an existing vault-server or launch a new one using docker-compose
 Scripts for configuring the vault-server are located in the [vault-init.sh](scripts/vault-init.sh)
@@ -173,11 +189,11 @@ cd pyinstabot-downloader
 docker-compose -f docker-compose.yml up vault-server -d
 
 # Initialize and unseal new vault-server
+export VAULT_ADDR=http://0.0.0.0:8200
 vault operator init
 vault operator unseal
 
 # Run the script for configuring the vault-server for this bot project
-export VAULT_ADDR=http://localhost:8200
 export VAULT_TOKEN=hvs.123456qwerty
 ./scripts/vault-init.sh
 ```
@@ -196,15 +212,16 @@ export VAULT_TOKEN=hvs.123456qwerty
 </br>
 
 ### Bot persistent data storage
-<img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/postgres.png" width="15" title="postgres"> Persistent data storage is implemented using `Postgresql`</br>
-You can familiarize yourself with the
+<img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/postgres.png" width="15" title="postgres"> Persistent data storage is implemented using `Postgresql`
 - data structure, tables and assignment of tables [here](src/configs/databases.json)
 - migrations [here](src/migrations/)
 
-The database structure is created automatically when the bot starts. Bot checks the database structure and creates missing tables if necessary.
-After checking the database structure, the bot executes the migrations in the order of their numbering.</br>
-All that is required is a database and the rights of the owner of this data database.
-To quickly prepare an instance, you can execute the[psql-init.sh](scripts/psql-init.sh) script
+The database structure is created automatically when the bot starts:
+  1. bot checks the database structure and creates missing tables if necessary
+  2. after checking the database structure, the bot executes the migrations in the order of their numbering
+
+To quickly prepare an instance, you can execute the [psql-init.sh](scripts/psql-init.sh) script
+
 ```bash
 git clone https://github.com/obervinov/pyinstabot-downloader.git
 cd pyinstabot-downloader
@@ -221,15 +238,16 @@ export PGDATABASE=postgres
 **What data is stored in tables:**
 - users requests queue
 - users metadata
-- history of processed users requests 
-- migration history
-- messages sent by the bot
+- history of users requests
+- history of processed messages
+- migrations history
+- service messages by the bot
 </br>
 
 ## <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/docker.png" width="25" title="docker"> How to run project locally
 ```sh
 export VAULT_APPROLE_ID={change_me}
-export VAULT_APPROLE_SECRETID={change_me}
+export VAULT_APPROLE_SECRET_ID={change_me}
 export VAULT_ADDR={change_me}
 docker compose -f docker-compose.yml up -d
 ```
@@ -238,4 +256,4 @@ docker compose -f docker-compose.yml up -d
 ## <img src="https://github.com/obervinov/_templates/blob/v1.2.2/icons/github-actions.png" width="25" title="github-actions"> GitHub Actions
 | Name  | Version |
 | ------------------------ | ----------- |
-| GitHub Actions Templates | [v1.2.8](https://github.com/obervinov/_templates/tree/v1.2.8) |
+| GitHub Actions Templates | [v1.2.9](https://github.com/obervinov/_templates/tree/v1.2.9) |

SECURITY.md

@@ -11,4 +11,4 @@ Versions supported to fix vulnerabilities
 
 ## Reporting a Vulnerability
 
-To report a vulnerability to me, just open the issue https://github.com/obervinov/pyinstabot-downloader/security/advisories/new
+To report a vulnerability to me, just open the issue https://github.com/obervinov/pyinstabot-downloader/security/advisories/new

docker-compose.yml

@@ -46,14 +46,16 @@ services:
             args:
                 PROJECT_NAME: pyinstabot-downloader
                 PROJECT_DESCRIPTION: "This project is a Telegram bot that allows you to upload posts from your Instagram profile to clouds like Dropbox, Mega or any WebDav compatible cloud storage."
-                PROJECT_VERSION: 2.2.0
+                PROJECT_VERSION: 2.3.0
         container_name: pyinstabot-downloader
         restart: always
         environment:
             - TELEGRAM_BOT_NAME=pyinstabot-downloader
             - VAULT_APPROLE_ID=${VAULT_APPROLE_ID}
-            - VAULT_APPROLE_SECRETID=${VAULT_APPROLE_SECRETID}
+            - VAULT_APPROLE_SECRET_ID=${VAULT_APPROLE_SECRET_ID}
             - VAULT_ADDR=${VAULT_ADDR}
+            - VAULT_AUTH_TYPE=approle
+            - VAULT_NAMESPACE=pyinstabot-downloader
             - LOGGER_LEVEL=DEBUG
             - MESSAGES_CONFIG=configs/messages.json
         volumes: