docs: add django example #445

rdmolony · 2023-03-01T16:50:32Z

Hey there,

I created an example of using devenv alongside Django via poetry including running unit tests by starting the database in the background.

I rely on start-postgres to spin up Postgres in the background (see here similarly to our chat on Discord on spinning up a database in GitHub Actions for testing Django projects

Let me know if you find this useful & if so, if you have any feedback

There's also a corresponding repo - https://github.com/rdmolony/devenv-django

rdmolony · 2023-03-01T18:02:58Z

I'm not sure if the start-postgres workaround is a good idea.

I'm having a bit of trouble trying to figure out how to run start-mysql or start-postgres outside of devenv up. Ideally, I'd like to run individual process like I might using docker-compose run postgres, so that I can just spin up the database in CI & run tests against it

If interested see example GitHub Action runs here rdmolony/devenv-django#1 against branch https://github.com/rdmolony/devenv-django/tree/reproduce-github-actions-passing-on-failing-test

rdmolony · 2023-03-02T10:46:00Z

I refactored to use devenv up as per rdmolony/devenv-django#2 as this generalises to services other than postgres required to spin up prior to running tests

rdmolony · 2023-03-02T15:27:53Z

Adapted https://github.com/cachix/devenv/blob/dfd9c4857c4e4b910b4270c6a35af9f9ba824917/examples/python-poetry/devenv.nix so using languages.python.poetry.enable to initialise poetry

bobvanderlinden

Nice, i like that this adds an example for a whole framework. Though the example is quite big and I think some of the parts could potentially moved to other modules if they're needed.

I'm hoping an example configuration for running a Django application would be just a few lines (like, languages.python.django.enable and enabling postgres).

examples/devenv-django/devenv.nix

bobvanderlinden · 2023-03-09T07:24:22Z

examples/devenv-django/devenv.nix

+
+  env = {
+    PYTHON_VERSION = python_version;
+    DATABASE_URL = "postgresql://${db_user}@${db_host}:${db_port}/${db_name}";


Could be a nice addition to the postgres module to automatically set the database url. Though I don't know how standardized this envvar is.

A bit of digging shows that this style of database URL is used by ...

libpq https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING ... source https://stackoverflow.com/questions/3582552/what-is-the-format-for-the-postgresql-connection-string-url

sqlalchemy uses pretty much the same style but often includes the database driver like postgresql+psycopg2://... https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls

I use it here via https://github.com/jazzband/dj-database-url which is a dependency of https://github.com/joke2k/django-environ which is used here to load secrets from a .env file

I don't know how it applies to other databases but I'd guess it's common

domenkozar · 2023-07-05T09:59:12Z

I feel like we need to expose wait_for_db from devenv itself for this example to get really small.

I'd love to merge this, but I'd like to make it minimal.

Can we get rid of some of the stuff that's not really required (like gitignore, CI, etc).

examples/python-django/devenv.nix

rdmolony · 2023-07-06T13:03:37Z

Let me know if you need anything further here or upstream re wait_for_db!

domenkozar · 2024-03-22T08:43:42Z

We now have wait_for_port: https://devenv.sh/tests/

rdmolony · 2024-03-28T17:07:31Z

I simplified the example, it's much simpler via wait_for_port, however, it now throws ...

Having upgraded to devenv 1.0.2 (x86_64-linux)

rowan@Windows:/mnt/c/Users/New User/Code/third-party/devenv/examples/python-django$ devenv test
• Overriding .devenv to .devenv.dPJ3fscqlnMO
• Building tests ...
error:
       … while evaluating the attribute 'config'
         at «github:cachix/devenv-nixpkgs/06fb0f1c643aee3ae6838dda3b37ef0abc3c763b»/lib/modules.nix:322:9:
          321|         options = checked options;
          322|         config = checked (removeAttrs config [ "_module" ]);
             |         ^
          323|         _module = checked (config._module);

       … while calling the 'seq' builtin
         at «github:cachix/devenv-nixpkgs/06fb0f1c643aee3ae6838dda3b37ef0abc3c763b»/lib/modules.nix:322:18:
          321|         options = checked options;
          322|         config = checked (removeAttrs config [ "_module" ]);
             |                  ^
          323|         _module = checked (config._module);

       … while calling the 'throw' builtin
         at «github:cachix/devenv-nixpkgs/06fb0f1c643aee3ae6838dda3b37ef0abc3c763b»/lib/modules.nix:298:18:
          297|                     ''
          298|             else throw baseMsg
             |                  ^
          299|         else null;

       error:
       error: The option `devenv.dotfile' does not exist. Definition values:
       - In `<unknown-file>': "/mnt/c/Users/New User/Code/third-party/devenv/examples/python-django/.devenv.dPJ3fscqlnMO"

✖ Command produced the following output:


✔ Building tests in 15.2s.
Error:   × Command `/nix/store/l24rmcpdnachwhz4rjwb9d89m3gxfy5n-nix-2.21-devenv/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty
  │ false --option eval-cache false --keep-going --max-jobs 2 build .#devenv.test --no-link --print-out-paths` failed with with exit code 1

I previously used .env to store SECRET_KEY for Django, however, I have since moved this to an environment variable in devenv.nix & still I see this thrown. I don't understand it.

domenkozar · 2024-04-01T06:40:50Z

You need to delete your devenv.lock in that folder

rdmolony · 2024-04-04T09:03:41Z

Thanks @domenkozar, the tests pass now, however, the behaviour doesn't match what I expect. I changed examples/python-django/myapp/test.py to a failing test & devenv test still shows Tests passed :) -

(I switched self.assertEquals(1,1) to self.assertEquals(1,2))

• Overriding .devenv to .devenv.bwFPAwU1VW4y
• Building tests ...
• Using Cachix: devenv
✔ Building tests in 6.0s.
• Building processes ...
✔ Building processes in 3.7s.
• Starting processes ...• Building shell ...
✔ Building shell in 3.8s.
• PID is 256663
• Stop:      $ devenv processes stop
✔ Starting processes in 3.8s.
• Running tests ...
• Building shell ...
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'warning: unknown setting 'upgrade-nix-store-path-url'
warning: unknown setting 'upgrade-nix-store-path-url'
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'Installing dependencies from lock file
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'
No dependencies to install or update
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'[runserver  ] • Building shell ...
[postgres       ] The files belonging to this database system will be owned by user "rowan".
[postgres       ] This user must also own the server process.
[postgres       ] 
[postgres       ] The database cluster will be initialized with locale "C".
[postgres       ] The default text search configuration will be set to "english".
[postgres       ] 
[postgres       ] Data page checksums are disabled.
[postgres       ] 
[postgres       ] creating directory /home/rowan/code/devenv/examples/python-django/.devenv.bwFPAwU1VW4y/state/postgres ... ok
[postgres       ] creating subdirectories ... ok
[postgres       ] selecting dynamic shared memory implementation ... posix
[postgres       ] selecting default max_connections ... 100
[postgres       ] selecting default shared_buffers ... 128MB
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'[postgres   ] selecting default time zone ... Europe/London
[postgres       ] creating configuration files ... ok
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'[postgres   ] running bootstrap script ... ok
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'[runserver  ] • Using Cachix: devenv
evaluating derivation 'git+file:///home/rowan/code/devenv?dir=examples/python-django#devShells.x86_64-linux.default'[postgres   ] performing post-bootstrap initialization ... ok
[postgres       ] syncing data to disk ... ok
[postgres       ] 
[postgres       ] 
[postgres       ] Success. You can now start the database server using:
[postgres       ] 
[postgres       ]     /nix/store/ph0bg8kq0g47h6rczgifrrbnag1isb56-postgresql-15.6/bin/pg_ctl -D /home/rowan/code/devenv/examples/python-django/.devenv.bwFPAwU1VW4y/state/postgres -l logfile start
[postgres       ] 
[postgres       ] initdb: warning: enabling "trust" authentication for local connections
[postgres       ] initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
[postgres       ] 
[postgres       ] PostgreSQL initdb process complete.
[postgres       ] 
[postgres       ] 
[postgres       ] PostgreSQL is setting up the initial database.
[postgres       ] 
✔ Building shell in 3.2s.
[postgres       ] waiting for server to start....2024-04-04 08:59:21.033 GMT [261579] LOG:  starting PostgreSQL 15.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.0, 64-bit
[postgres       ] 2024-04-04 08:59:21.041 GMT [261579] LOG:  listening on Unix socket "/tmp/devenv-0480122/postgres/.s.PGSQL.5432"
[postgres       ] 2024-04-04 08:59:21.051 GMT [261601] LOG:  database system was shut down at 2024-04-04 08:59:20 GMT
[postgres       ] 2024-04-04 08:59:21.063 GMT [261579] LOG:  database system is ready to accept connections
evaluating derivation 'flake:nixpkgs#bashInteractive'[postgres  ]  done
[postgres       ] server started
[postgres       ] Checking presence of database: db
[postgres       ] 0
[postgres       ] Creating database: db
evaluating derivation 'flake:nixpkgs#bashInteractive'[postgres  ] CREATE DATABASE
[postgres       ] CREATE ROLE
[postgres       ] waiting for server to shut down....2024-04-04 08:59:21.202 GMT [261579] LOG:  received fast shutdown request
[postgres       ] 2024-04-04 08:59:21.206 GMT [261579] LOG:  aborting any active transactions
[postgres       ] 2024-04-04 08:59:21.208 GMT [261579] LOG:  background worker "logical replication launcher" (PID 261604) exited with exit code 1
[postgres       ] 2024-04-04 08:59:21.212 GMT [261599] LOG:  shutting down
[postgres       ] 2024-04-04 08:59:21.216 GMT [261599] LOG:  checkpoint starting: shutdown immediate
evaluating derivation 'flake:nixpkgs#bashInteractive'[postgres  ] 2024-04-04 08:59:21.384 GMT [261599] LOG:  checkpoint complete: wrote 922 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.024 s, sync=0.120 s, total=0.173 s; sync files=305, longest=0.051 s, average=0.001 s; distance=4226 kB, estimate=4226 kB
[postgres       ] 2024-04-04 08:59:21.389 GMT [261579] LOG:  database system is shut down
[postgres       ]  done
[postgres       ] server stopped
evaluating derivation 'flake:nixpkgs#bashInteractive'[postgres  ] 2024-04-04 08:59:21.465 GMT [259877] LOG:  starting PostgreSQL 15.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.0, 64-bit
[postgres       ] 2024-04-04 08:59:21.466 GMT [259877] LOG:  listening on IPv4 address "127.0.0.1", port 5432
evaluating derivation 'flake:nixpkgs#bashInteractive'[postgres  ] 2024-04-04 08:59:21.482 GMT [259877] LOG:  listening on Unix socket "/tmp/devenv-0480122/postgres/.s.PGSQL.5432"
[postgres       ] 2024-04-04 08:59:21.502 GMT [262148] LOG:  database system was shut down at 2024-04-04 08:59:21 GMT
[postgres       ] 2024-04-04 08:59:21.512 GMT [259877] LOG:  database system is ready to accept connections
warning: unknown setting 'upgrade-nix-store-path-url'
warning: unknown setting 'upgrade-nix-store-path-url'
• Setting up shell environment ...
warning: unknown setting 'upgrade-nix-store-path-url'
warning: unknown setting 'upgrade-nix-store-path-url'
• Testing ...
✔ Running tests in 5.4s.
• Stopping process with PID 256663
Stopping processes...
• Tests passed :)
[postgres       ] 2024-04-04 08:59:23.189 GMT [259877] LOG:  received fast shutdown request
[postgres       ] 2024-04-04 08:59:23.202 GMT [259877] LOG:  aborting any active transactions
[postgres       ] 2024-04-04 08:59:23.205 GMT [259877] LOG:  background worker "logical replication launcher" (PID 262173) exited with exit code 1
[postgres       ] 2024-04-04 08:59:23.205 GMT [262146] LOG:  shutting down
[postgres       ] 2024-04-04 08:59:23.221 GMT [262146] LOG:  checkpoint starting: shutdown immediate
[postgres       ] 2024-04-04 08:59:23.236 GMT [262146] PANIC:  could not rename file "pg_logical/replorigin_checkpoint.tmp" to "pg_logical/replorigin_checkpoint": No such file or directory

If I run devenv up to launch Postgres & in a separate shell run devenv shell; python manage.py test then I see my test logs which correctly show my tests have passed or failed

Am I missing something here?

examples/python-django/devenv.nix

domenkozar · 2024-04-08T08:28:43Z

Can you fix conflicts?

rdmolony · 2024-04-08T08:37:24Z

I rebased on main (dangerous tool!) & then realised that it was my bad that devenv.lock was out of sync. All good now

devenv.lock

Co-authored-by: bobvanderlinden Co-authored-by: domenkozar

examples/python-django/devenv.lock

examples/python-django/devenv.yaml

Co-authored-by: Domen Kožar <domen@enlambda.com>

domenkozar · 2024-04-09T10:05:00Z

Thank you!

domenkozar force-pushed the main branch 4 times, most recently from e82dccf to 2cae5ec Compare March 2, 2023 11:27

bobvanderlinden reviewed Mar 9, 2023

View reviewed changes

domenkozar force-pushed the main branch from e7de27e to 47eb4d1 Compare April 26, 2023 17:03

domenkozar reviewed Jul 6, 2023

View reviewed changes

examples/python-django/devenv.nix Outdated Show resolved Hide resolved

domenkozar force-pushed the main branch from 3730e4c to 72affdb Compare March 25, 2024 15:39

domenkozar reviewed Apr 5, 2024

View reviewed changes

examples/python-django/devenv.nix Outdated Show resolved Hide resolved

domenkozar reviewed Apr 8, 2024

View reviewed changes

examples/python-django/devenv.nix Outdated Show resolved Hide resolved

rdmolony force-pushed the add-devenv-django-example branch from 4c4c776 to 5ef0591 Compare April 8, 2024 08:34

domenkozar reviewed Apr 8, 2024

View reviewed changes

devenv.lock Outdated Show resolved Hide resolved

docs: add django example

03ae5d6

Co-authored-by: bobvanderlinden Co-authored-by: domenkozar

rdmolony force-pushed the add-devenv-django-example branch from d8b8054 to 03ae5d6 Compare April 8, 2024 19:55

domenkozar reviewed Apr 9, 2024

View reviewed changes

examples/python-django/devenv.lock Outdated Show resolved Hide resolved

domenkozar reviewed Apr 9, 2024

View reviewed changes

examples/python-django/devenv.yaml Outdated Show resolved Hide resolved

rdmolony and others added 2 commits April 9, 2024 08:51

remove devenv lock & yaml

0c168b5

remove devenv shell from runserver process

a299873

Co-authored-by: Domen Kožar <domen@enlambda.com>

domenkozar approved these changes Apr 9, 2024

View reviewed changes

domenkozar merged commit 7925cce into cachix:main Apr 9, 2024
197 of 251 checks passed

rdmolony deleted the add-devenv-django-example branch April 9, 2024 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add django example #445

docs: add django example #445

rdmolony commented Mar 1, 2023

rdmolony commented Mar 1, 2023 •

edited

Loading

rdmolony commented Mar 2, 2023 •

edited

Loading

rdmolony commented Mar 2, 2023

bobvanderlinden left a comment

bobvanderlinden Mar 9, 2023

rdmolony Mar 10, 2023

domenkozar commented Jul 5, 2023

rdmolony commented Jul 6, 2023

domenkozar commented Mar 22, 2024

rdmolony commented Mar 28, 2024 •

edited

Loading

domenkozar commented Apr 1, 2024

rdmolony commented Apr 4, 2024 •

edited

Loading

domenkozar commented Apr 8, 2024

rdmolony commented Apr 8, 2024

domenkozar commented Apr 9, 2024

docs: add django example #445

docs: add django example #445

Conversation

rdmolony commented Mar 1, 2023

rdmolony commented Mar 1, 2023 • edited Loading

rdmolony commented Mar 2, 2023 • edited Loading

rdmolony commented Mar 2, 2023

bobvanderlinden left a comment

Choose a reason for hiding this comment

bobvanderlinden Mar 9, 2023

Choose a reason for hiding this comment

rdmolony Mar 10, 2023

Choose a reason for hiding this comment

domenkozar commented Jul 5, 2023

rdmolony commented Jul 6, 2023

domenkozar commented Mar 22, 2024

rdmolony commented Mar 28, 2024 • edited Loading

domenkozar commented Apr 1, 2024

rdmolony commented Apr 4, 2024 • edited Loading

domenkozar commented Apr 8, 2024

rdmolony commented Apr 8, 2024

domenkozar commented Apr 9, 2024

rdmolony commented Mar 1, 2023 •

edited

Loading

rdmolony commented Mar 2, 2023 •

edited

Loading

rdmolony commented Mar 28, 2024 •

edited

Loading

rdmolony commented Apr 4, 2024 •

edited

Loading