Reorganization of the documentation structure (#2279)

Top-level reorganization as first step of restructuring the documentation.

docs/source/developer_guide/internals.rst → docs/source/developer_guide/core/internals.rst

@@ -402,3 +402,85 @@ For the **complete** API documentation see :py:mod:`aiida.orm`.
     :maxdepth: 2
 
     orm_overview
+
+Deprecated features, renaming, and adding new methods
++++++++++++++++++++++++++++++++++++++++++++++++++++++
+In case a method is renamed or removed, this is the procedure to follow:
+
+1. (If you want to rename) move the code to the new function name.
+   Then, in the docstring, add something like::
+
+     .. versionadded:: 0.7
+        Renamed from OLDMETHODNAME
+
+2. Don't remove directly the old function, but just change the code to use
+   the new function, and add in the docstring::
+
+     .. deprecated:: 0.7
+        Use :meth:`NEWMETHODNAME` instead.
+
+   Moreover, at the beginning of the function, add something like::
+
+     import warnings
+
+     # If we call this DeprecationWarning, pycharm will properly strike out the function
+     from aiida.common.warnings import AiidaDeprecationWarning as DeprecationWarning  # pylint: disable=redefined-builtin
+     warnings.warn("<Deprecation warning here - MAKE IT SPECIFIC TO THIS DEPRECATION, as it will be shown only once per different message>", DeprecationWarning)
+        
+     # <REST OF THE FUNCTION HERE>
+ 
+   (of course replace the parts between ``< >`` symbols with the
+   correct strings).
+
+   The advantage of the method above is:
+
+   - pycharm will still show the method crossed out
+   - Our ``AiidaDeprecationWarning`` does not inherit from ``DeprecationWarning``, so it will not be "hidden" by python
+   - User can disable our warnings (and only those) by using AiiDA
+     properties with::
+       
+       verdi devel setproperty warnings.showdeprecations False
+
+Changing the config.json structure
+++++++++++++++++++++++++++++++++++
+
+In general, changes to ``config.json`` should be avoided if possible. However, if there is a need to modify it, the following procedure should be used to create a migration:
+
+1. Determine whether the change will be backwards-compatible. This means that an older version of AiiDA will still be able to run with the new ``config.json`` structure. It goes without saying that it's preferable to change ``config.json`` in a backwards-compatible way.
+
+2. In ``aiida/common/additions/config_migration/_migrations.py``, increase the ``CURRENT_CONFIG_VERSION`` by one. If the change is **not** backwards-compatible, set ``OLDEST_COMPATIBLE_CONFIG_VERSION`` to the same value.
+
+3. Write a function which transforms the old config dict into the new version. It is possible that you need user input for the migration, in which case this should also be handled in that function.
+
+4. Add an entry in ``_MIGRATION_LOOKUP`` where the key is the version **before** the migration, and the value is a ``ConfigMigration`` object. The ``ConfigMigration`` is constructed from your migration function, and the **hard-coded** values of ``CURRENT_CONFIG_VERSION`` and ``OLDEST_COMPATIBLE_CONFIG_VERSION``. If these values are not hard-coded, the migration will break as soon as the values are changed again.
+
+5. Add tests for the migration, in ``aiida/common/additions/config_migration/test_migrations.py``. You can add two types of tests:
+
+    * Tests that run the entire migration, using the ``check_and_migrate_config`` function. Make sure to run it with ``store=False``, otherwise it will overwrite your ``config.json`` file. For these tests, you will have to update the reference files.
+    * Tests that run a single step in the migration, using the ``ConfigMigration.apply`` method. This can be used if you need to test different edge cases of the migration.
+
+  There are examples for both types of tests.
+
+Daemon and signal handling
+++++++++++++++++++++++++++
+
+While the AiiDA daemon is running, interrupt signals (``SIGINT`` and ``SIGTERM``) are captured so that the daemon can shut down gracefully. This is implemented using Python's ``signal`` module, as shown in the following dummy example:
+
+.. code:: python
+
+    import signal
+
+    def print_foo(*args):
+        print('foo')
+
+    signal.signal(signal.SIGINT, print_foo)
+
+You should be aware of this while developing code which runs in the daemon. In particular, it's important when creating subprocesses. When a signal is sent, the whole process group receives that signal. As a result, the subprocess can be killed even though the Python main process captures the signal. This can be avoided by creating a new process group for the subprocess, meaning that it will not receive the signal. To do this, you need to pass ``preexec_fn=os.setsid`` to the ``subprocess`` function:
+
+.. code:: python
+
+    import os
+    import subprocess
+
+    print(subprocess.check_output('sleep 3; echo bar', preexec_fn=os.setsid))
+

docs/source/developer_guide/core/modifying_the_schema.rst

@@ -0,0 +1,122 @@
+Mofidying the schema
+++++++++++++++++++++
+
+Django
+------
+The Django database schema can be found in :py:mod:`aiida.backends.djsite.db.models`.
+
+If you need to change the database schema follow these steps:
+
+1. Make all the necessary changes to :py:mod:`aiida.backends.djsite.db.models`
+2. Create a new migration file.  From ``aiida/backends/djsite``, run::
+
+     python manage.py makemigrations
+
+   This will create the migration file in ``aiida/backends/djsite/db/migrations`` whose
+   name begins with a number followed by some description.  If the description
+   is not appropriate then change to it to something better but retain the
+   number.
+
+3. Open the generated file and make the following changes::
+
+    from aiida.backends.djsite.db.migrations import upgrade_schema_version
+    ...
+    REVISION = # choose an appropriate version number (hint: higher than the last migration!)
+    DOWN_REVISION = # the revision number of the previous migration
+    ...
+    class Migration(migrations.Migration):
+      ...
+      operations = [
+        ..
+        upgrade_schema_version(REVISION, DOWN_REVISION)
+      ]
+
+5. Change the ``LATEST_MIGRATION`` variable in
+   ``aiida/backends/djsite/db/migrations/__init__.py`` to the name of your migration
+   file::
+
+     LATEST_MIGRATION = '0003_my_db_update'
+
+   This let's AiiDA get the version number from your migration and make sure the
+   database and the code are in sync.
+6. Migrate your database to the new version, (again from ``aiida/backends/djsite``),
+   run::
+
+     python manage.py migrate
+
+
+SQLAlchemy
+----------
+The SQLAlchemy database schema can be found in ``aiida/backends/sqlalchemy/models``
+
+If you need to change the database schema follow these steps:
+
+1. Make all the necessary changes to the model than you would like to modify
+   located in the ``aiida/backends/sqlalchemy/models`` directory.
+2. Create new migration file by going to ``aiida/backends/sqlalchemy`` and
+   executing::
+
+    ./alembic_manage.py revision "This is a new revision"
+
+   This will create a new migration file in ``aiida/backends/sqlalchemy/migrations/versions``
+   whose names begins with an automatically generated hash code and the
+   provided message for this new migration. Of course you can change the
+   migration message to a message of your preference. Please look at the
+   generatedvfile and ensure that migration is correct. If you are in doubt
+   about the operations mentioned in the file and its content, you can have a
+   look at the Alembic documentation.
+3. Your database will be automatically migrated to the latest revision as soon
+   as you run your first verdi command. You can also migrate it manually with
+   the help of the alembic_manage.py script as you can see below.
+
+Overview of alembic_manage.py commands
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The alembic_manage.py provides several options to control your SQLAlchemy
+migrations. By executing::
+
+    ./alembic_manage.py --help
+
+you will get a full list of the available arguments that you can pass and
+commands. Briefly, the available commands are:
+
+* **upgrade** This command allows you to upgrade to the later version. For the
+  moment, you can only upgrade to the latest version.
+* **downgrade** This command allows you to downgrade the version of your
+  database. For the moment, you can only downgrade to the base version.
+* **history** This command lists the available migrations in chronological
+  order.
+* **current** This command displays the current version of the database.
+* **revision** This command creates a new migration file based on the model
+  changes.
+
+.. _first_alembic_migration:
+
+Debugging Alembic
+~~~~~~~~~~~~~~~~~
+Alembic migrations should work automatically and migrate your database to the
+latest version. However, if you were using SQLAlchemy before we introduced
+Alembic, you may get a message like to following during the first migration::
+
+    sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation
+    "db_dbuser" already exists [SQL: '\nCREATE TABLE db_dbuser (\n\tid SERIAL
+    NOT NULL, \n\temail VARCHAR(254), \n\tpassword VARCHAR(128),
+    \n\tis_superuser BOOLEAN NOT NULL, \n\tfirst_name VARCHAR(254),
+    \n\tlast_name VARCHAR(254), \n\tinstitution VARCHAR(254), \n\tis_staff
+    BOOLEAN, \n\tis_active BOOLEAN, \n\tlast_login TIMESTAMP WITH TIME ZONE,
+    \n\tdate_joined TIMESTAMP WITH TIME ZONE, \n\tCONSTRAINT db_dbuser_pkey
+    PRIMARY KEY (id)\n)\n\n']
+
+In this case, you should create manually the Alembic table in your database and
+add a line with the database version number. To do so, use psql to connect
+to the desired database::
+
+    psql aiidadb_sqla
+
+(you should replace ``aiidadb_sqla`` with the name of the database that you
+would like to modify). Then, execute the following commands::
+
+    CREATE TABLE alembic_version (version_num character varying(32) not null, PRIMARY KEY(version_num));
+    INSERT INTO alembic_version VALUES ('e15ef2630a1b');
+    GRANT ALL ON alembic_version TO aiida;
+
+