Instructions for integrating piccolo inside existing application?

[mavwolverine]

We are currently using asyncpg in a blacksheep application. Since piccolo is based on top of asyncpg and provides migration support, we want to migrate to piccolo.

But the only instructions available, are for creating new project.

It would be great if some documentation could be provided on integrating into existing projects.

Thanks

[dantownsend]

I'll add this to the docs at some point.

I'd recommend creating a new project using piccolo asgi new and selecting BlackSheep - that way you can see how everything hangs together.

You can then copy over the relevant bits to your existing project.

Alternatively, doing it from scratch, you'll need to do the following:

Create a new Piccolo project file

Create a new piccolo_conf.py file in the root of your project:

piccolo project new

This contains your database details, and is used to register Piccolo apps.

Create a new app

The app contains your tables and migrations. Run this at the root of your project:

# Replace 'my_app' with whatever you want to call your app
piccolo app new my_app

Register this new app

Register this new app in piccolo_conf.py. For example:

APP_REGISTRY = AppRegistry(
    apps=[
        "my_app.piccolo_app",
    ]
)

While you're at it, make sure the database credentials are correct in piccolo_conf.py.

Make Piccolo tables for your current database

Now, if you run:

piccolo schema generate

It will output Piccolo tables for your current database. Copy the output into my_project/tables.py. Double check that everything looks correct.

In my_project/piccolo_app.py make sure it's tracking these tables for migration purposes.

from piccolo.conf.apps import AppConfig, table_finder

APP_CONFIG = AppConfig(
    ...
    table_classes=table_finder(["my_app.tables"], exclude_imported=True),
)

Create an initial migration

piccolo migrations new my_app --auto

These tables already exist in your database, as it's an existing project, so you need to fake apply this initial migration:

piccolo migrations forwards my_app --fake

Using the Piccolo tables in your app

Now you're basically setup - to make database queries:

from my_app.tables import MyTable

async def my_blacksheep_endpoint():
    data = await MyTable.select()
    return data

Making new migrations

Just modify the files in tables.py, and then run:

piccolo migrations new my_app --auto
piccolo migrations forwards my_app

---

I hope that helps.

[mavwolverine]

Thank you Dan for those instructions. Will give them a try.

One question on piccolo_conf.py.
Is there any way to make it configurable (based on environment)?

https://github.com/Neoteroi/essentials-configuration is a library from the blacksheep creator.

""" Load app configuration """
import os

from configuration.common import Configuration, ConfigurationBuilder
from configuration.env import EnvironmentVariables
from configuration.yaml import YAMLFile

environment_name = os.environ.get("APP_ENVIRONMENT", "dev")


def load_configuration() -> Configuration:
    """Create Configuration object from config"""
    builder = ConfigurationBuilder(
        YAMLFile(os.path.join("config", "config.yaml")),
        YAMLFile(os.path.join("config", f"config.{environment_name}.yaml"), optional=True),
        EnvironmentVariables(prefix="APP_"),
    )

    return builder.build()

The above code reads config from config.yaml, overwrites some of those config from config.${APP_ENVIRONMENT}.yaml and other Environment variables.

That way, we can have different db credentials for dev, qa and prod environments.

Is there any way to make this possible with piccolo_conf.py?

Thanks

[dantownsend]

Yes, it's possible. I haven't used essentials-configuration, but I use something similar called python-dotenv. So my piccolo_conf.py looks like this:

import os

from dotenv import load_dotenv
from piccolo.engine.postgres import PostgresEngine

# Loads environment variables from a .env file.
load_dotenv()


DB = PostgresEngine(
    config={
        "port": int(os.environ.get("PG_PORT", "5432")),
        "user": os.environ.get("PG_USER", "postgres"),
        "password": os.environ.get("PG_PASSWORD", ""),
        "database": os.environ.get("PG_DATABASE", "my_project"),
        "host": os.environ.get("PG_HOST", "localhost"),
    }
)

[mavwolverine]

That did give me an idea.

from app.configuration import load_configuration

config = load_configuration()

DB = PostgresEngine(
    config={
        "port": int(config.pg_port),
        "user": config.pg_user,
        "password": config.pg_password,
        "database": config.pg_database,
        "host": config.pg_host,
    }
)

I would have to call load_configuration() twice , once here and once in the app, but I guess still works.

Thanks