Merge #3008 #3015

3008: Removed from __future__ r=jngrad a=jrfinkbeiner

Fixes #3004

3015: Update checkpointing docs r=jngrad a=RudolfWeeber

Fixes #2808

Co-authored-by: Jan Finkbeiner <st144298@stud.uni-stuttgart.de>
Co-authored-by: Rudolf Weeber <weeber@icp.uni-stuttgart.de>
Co-authored-by: Jean-Noël Grad <jgrad@icp.uni-stuttgart.de>

doc/doxygen/gen_doxyconfig.py

@@ -19,7 +19,6 @@
 #
 # This script generates the file doxy-features
 #
-from __future__ import print_function
 import inspect
 import sys
 import os

doc/sphinx/io.rst

@@ -5,159 +5,84 @@ Input and Output
 
 .. _No generic checkpointing:
 
-No generic checkpointing
-------------------------
+Checkpointing and restoring a simulation
+----------------------------------------
 
-One of the most asked-for feature that seems to be missing in is
-*checkpointing*, a simple way to tell to store and restore the current
+One of the most asked-for feature that seems to be missing is
+*checkpointing*, a simple way to store and restore the current
 state of the simulation, and to be able to write this state to or read
 it from a file. This would be most useful to be able to restart a
 simulation from a specific point in time.
 
 Unfortunately, it is impossible to provide a simple command
-(``checkpoint``), out of two reasons. The main reason is that has no way
-to determine what information constitutes the actual state of the
-simulation. On the one hand, scripts sometimes use Tcl-variables that
-contain essential information about a simulation, the stored values of
+(``checkpoint``), out of two reasons. The main reason is that it has no
+way to determine what information constitutes the actual state of the
+simulation. Scripts sometimes use variables that
+contain essential information about a simulation: the stored values of
 an observable that was computed in previous time steps, counters, etc.
 These would have to be contained in a checkpoint. However, not all
-Tcl-variables are of interest. For example, Tcl has a number of
-automatically set variables that contain information about the hostname,
-the machine type, etc. These variables should most probably *not* be
-included the simulation state. has no way to distinguish between these
-variables. On the other hand, the core has a number of internal
-variables, the particle coordinates. While most of these are probably
-good candidates for being included into a checkpoint, this is not
-necessarily so. For example, when you have particles in your system that
-have fixed coordinates, should these be stored in a checkpoint, or not?
-If the system contains mostly fixed particles and only very few moving
-particles, this would increase the memory size of a checkpoint
-needlessly. And what about the interactions in the system, or the bonds?
-Should these be stored in a checkpoint, or are they generated by the
-script?
+variables are of interest. 
 
 Another problem with a generic checkpoint would be the control flow of
 the script. In principle, the checkpoint would have to store where in
 the script the checkpointing function was called to be able to return
-there. All this is even further complicated by the fact that is running
-in parallel.
-
-Instead, in |es| the user has to specify what information needs to be saved to a
-file to be able to restore the simulation state. When floating point numbers
-are stored in text files (the particle positions), there is only a limited
-precision. Therefore, it is not possible to bitwise reproduce a simulation
-state using this text files. When you need bitwise reproducibility, you will have
-to use checkpointing , which stores positions, forces and velocities in binary
-format.
-
-.. _(Almost) generic checkpointing in Python:
-
-(Almost) generic checkpointing in Python
-----------------------------------------
-
-Referring to the previous section, generic checkpointing poses
-difficulties in many ways. Fortunately, the Python checkpointing module
-presented in this section provides a comfortable workflow for an almost
-generic checkpointing.
+there. All this is even further complicated by the fact that |es| is
+running in parallel.
 
-The idea is to let the user initially define which data is of interest
-for checkpointing and thus solve the above mentioned problem. Once this
-is done, checkpoints can then be saved simply by calling one save
-function.
-
-The checkpoint data can then later be restored easily by calling one
+Having said that, |es| does provide functionality which aims to store the state of the simulation engine.
+In addition, variables declared in the simulation script can be added to the checkpoint.
+The checkpoint data can then later be restored by calling one
 load function that will automatically process the checkpoint data by
-setting the user variables and the checkpointed properties in .
-
-In addition, the checkpointing module is also able to catch signals that
+setting the user variables and restore the components of the simulation.
+Furthermore, the checkpointing can be triggered by system signals that
 are invoked for example when the simulation is aborted by the user or by
 a timeout.
 
-The checkpointing module can be imported with::
+The checkpointing functionality is difficult to test for all possible simulation setups. Therefore, it is to be used with care.
+It is strongly recommended to keep track of the times in the simulation run where a checkpoint was written and restored and manually verify that the observables of interest do not jump or drift after restoring the checkpoint.
+Moreover, please carefully read the limitations mentioned below.
 
+Checkpointing is implemented by the :class:`espressomd.checkpointing.Checkpoint` class. It is instanced as follows::
+    
     from espressomd import checkpointing
+    checkpoint = checkpointing.Checkpoint(checkpoint_id="mycheckpoint", checkpoint_path=".")
 
-    [ checkpoint_path= ]
-
-Determines the identifier for a checkpoint. Legal characters for an id
+Here, `checkpoint_id` denotes the identifier for a checkpoint. Legal characters for an id
 are "0-9", "a-zA-Z", "-", "_".
+The parameter `checkpoint_path`, specifies the relative or absolute path where the checkpoints are
+stored. The current working directory is assumed, when this parameter is skipped.
 
-Specifies the relative or absolute path where the checkpoints are
-stored.
-
-For example ``checkpoint = checkpointing.Checkpoint(checkpoint_id="mycheckpoint")``
-would create the new checkpoint with id "mycheckpoint" and all the
-checkpointing data will be stored in the current directory.
-
-After the system and checkpointing user variables are set up they can be
+After the simulation system and user variables are set up, they can be
 registered for checkpointing.
 Name the string of the object or user variable that should be registered for
 checkpointing.
 
 To give an example::
 
-    myvar = "some variable value"
-    skin = 0.4
-    checkpoint.register("myvar")
-    checkpoint.register("skin")
-
+    my_var = "some variable value"
     system = espressomd.System(box_l=[100.0, 100.0, 100.0])
     # ... set system properties like time_step here ...
     checkpoint.register("system")
+    checkpoint.register("my_var")
+    # ...
 
-    system.thermostat.set_langevin(kT=1.0, gamma=1.0)
-    checkpoint.register("system.thermostat")
-
-    # ... set system.non_bonded_inter here ...
-    checkpoint.register("system.non_bonded_inter")
-
-    # ... add particles to the system with system.part.add(...) here ...
-    checkpoint.register("system.part")
+will register the user variable `my_var` and the instance of the simulation system. The checkpoint can be saved via::
 
-    # ... set charges of particles here ...
-    from espressomd import electrostatics
-    p3m = electrostatics.P3M(prefactor=1.0, accuracy=1e-2)
-    system.actors.add(p3m)
-    checkpoint.register("p3m")
 
-will register the user variables ``skin`` and ``myvar``, system properties, a
-Langevin thermostat, non-bonded interactions, particle properties and a p3m
-object for checkpointing. It is important to note that the checkpointing of
-|es| will only save basic system properties. This excludes for example the
-system thermostat or the particle data. For this reason one has to explicitly
-register and for checkpointing.
+    checkpoint.save()
 
-Analogous to this, objects that have been registered for checkpointing but are
-no longer needed in the next checkpoints can be unregistered with ``checkpoint
-unregister var``.  A list of all registered object names can be generated with
-``checkpoint get_registered_objects``.  A new checkpoint with a consecutive
-index that contains the latest data of the registered objects can then be
-created by calling ``checkpoint save [checkpoint_index]``.
+To trigger the checkpoint when Ctrl+C is pressed during a running simulation, the corresponding signal has to be registered::
 
-An existing checkpoint can be loaded with ``checkpoint load
-[checkpoint_index]``.
 
-If no is passed the last checkpoint will be loaded. Concerning the procedure of
-registering objects for checkpointing it is good to know that all registered
-objects saved in a checkpoint will be automatically re-registered after loading
-this checkpoint.
-
-In practical implementations it might come in handy to check if there are any
-available checkpoints for a given checkpoint id. This can be done with
-``checkpoint has_checkpoints`` which returns a bool value.
-
-As mentioned in the introduction, the checkpointing module also enables
-to catch signals in order to save a checkpoint and quit the simulation.
-Therefore one has to register the signal which should be caught with
-``checkpoint register_signal signum=int_number``.
-
-The registered signals are associated with the checkpoint id and will be automatically
-re-registered when the same checkpoint id is used later.
+    import signal
+    # signal.SIGINT: signal 2, is sent when ctrl+c is pressed
+    checkpoint.register_signal(signal.SIGINT)
 
-Following the example above, the next example loads the last checkpoint,
-restores the state of all checkpointed objects and registers a signal.
+In the above example checkpointing is triggered, when the user 
+interrupts by pressing Ctrl+C. In this case a new checkpoint is written and the simulation
+quits.
 
-.. code::
+An existing checkpoint can be loaded with::
 
     import espressomd
     from espressomd import checkpointing
@@ -166,25 +91,28 @@ restores the state of all checkpointed objects and registers a signal.
     checkpoint = checkpointing.Checkpoint(checkpoint_id="mycheckpoint")
     checkpoint.load()
 
-    system = espressomd.System(box_l=[100.0, 100.0, 100.0])
-    system.cell_system.skin = skin
-    system.actors.add(p3m)
+This will restore the state of the objects registered for checkpointing.
+The checkpointing instance itself will also be restored. I.e., the same variables will be registered for the next checkpoint and the same system signals will be caught as in the initial setup of the checkpointing.
 
-    # signal.SIGINT: signal 2, is sent when ctrl+c is pressed
-    checkpoint.register_signal(signal.SIGINT)
+Be aware of the following limitations: 
 
-    # integrate system until user presses ctrl+c while True:
-    system.integrator.run(1000)
+  * Checkpointing makes use of the `pickle` python package. Objects will only be restored as far as they support pickling. This is the case for Python's basic data types, `numpy` arrays and many other objects. Still, pickling support cannot be taken for granted.
 
-The above example runs as long as the user interrupts by pressing
-ctrl+c. In this case a new checkpoint is written and the simulation
-quits.
+  * Pickling support of the Espresso system instance and contained objects such as bonded and non-bonded interactions and electrostatics methods. However, there are many more combinations of active interactions and algorithms then can be tested.
+
+  * The active actors, i.e., the content of `system.actors`, are checkpointed. For lattice Boltzmann fluids, this only includes the parameters such as the lattice constant (`agrid`). The actual flow field has to be saved separately with the lattice-Boltzmann specific methods 
+    :meth:`espressomd.lb.HydrodynamicInteraction.save_checkpoint`
+    and loaded via :meth:`espressomd.lb.HydrodynamicInteraction.load_checkpoint` after restoring the checkpoint
+
+  * References between Python objects are not maintained during checkpointing. For example, if an instance of a shape and an instance of a constraint containing the shape are checkpointed, these two objects are equal before checkpointing but independent copies which have the same parameters after restoring the checkpoint. Changing one will no longer afect the other.
+
+  * The state of the cell system as well as the MPI node grid are checkpointed. Therefore, checkpoints can only be loaded, when the script runs on the same number of MPI ranks.
+
+  * Checkpoints are not compatible between different |es| versions.
+
+  * Checkpoints may depend on the presence of other Python modules at specific versions. It may therefore not be possible to load a checkpoint in a different environment than where it was loaded. 
 
-It is perhaps surprising that one has to explicitly create :class:`~espressomd.system.System` again.
-But this is necessary as not all |es| modules like :class:`~espressomd.cellsystem` or
-:class:`~espressomd.actors` have implementations for checkpointing yet. By calling :class:`~espressomd.system.System` these modules
-are created and can be easily initialized with checkpointed user variables
-(like ``skin``) or checkpointed submodules (like ``p3m``).
+For additional methods of the checkpointing class, see :class:`espressomd.checkpointing.Checkpoint`.
 
 .. _Writing H5MD-Files:
 

doc/sphinx/visualization.rst

@@ -228,7 +228,6 @@ following code snippet demonstrates the visualization of the director property
 and individual settings for two particle types (requires the ``ROTATION``
 feature)::
 
-    from __future__ import print_function
     import numpy
     from espressomd import *
     from espressomd.visualization_opengl import *

doc/tutorials/01-lennard_jones/01-lennard_jones.ipynb

@@ -16,19 +16,18 @@
     "2. [Background](#Background)\n",
     "3. [The Lennard-Jones Potential](#The-Lennard-Jones-Potential)\n",
     "4. [Units](#Units)\n",
-    "5. [Python versions](#Python-versions)\n",
-    "6. [First steps](#First-steps)\n",
-    "7. [Overview of a simulation script](#Overview-of-a-simulation-script)\n",
+    "5. [First steps](#First-steps)\n",
+    "6. [Overview of a simulation script](#Overview-of-a-simulation-script)\n",
     "    1. [System setup](#System-setup)\n",
     "    2. [Choosing the thermodynamic ensemble, thermostat](#Choosing-the-thermodynamic-ensemble,-thermostat)\n",
     "    3. [Placing and accessing particles](#Placing-and-accessing-particles)\n",
     "    4. [Setting up non-bonded interactions](#Setting-up-non-bonded-interactions)\n",
     "    5. [Warmup](#Warmup)\n",
     "    6. [Integrating equations of motion and taking measurements](#Integrating-equations-of-motion-and-taking-measurements)\n",
     "    7. [Simple Error Estimation on Time Series Data](#Simple-Error-Estimation-on-Time-Series-Data)\n",
-    "8. [Exercises](#Exercises)\n",
+    "7. [Exercises](#Exercises)\n",
     "    1. [Binary Lennard-Jones Liquid](#Binary-Lennard-Jones-Liquid)\n",
-    "9. [References](#References)\n",
+    "8. [References](#References)\n",
     "    "
    ]
   },
@@ -92,24 +91,6 @@
     "words LJ units."
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Python versions\n",
-    "\n",
-    "Espresso can be used both, with python 2 and python 3. In these tutorials, we use python 3 print syntax. If you are on python 2, issue the following command to activate it:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from __future__ import print_function"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},

doc/tutorials/02-charged_system/02-charged_system-1.ipynb

@@ -39,7 +39,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from __future__ import print_function\n",
     "from espressomd import System, electrostatics\n",
     "import espressomd\n",
     "import numpy\n",

doc/tutorials/02-charged_system/02-charged_system-2.ipynb

@@ -67,7 +67,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from __future__ import print_function\n",
     "from espressomd import System, electrostatics, electrostatic_extensions\n",
     "from espressomd.shapes import Wall\n",
     "import espressomd\n",

doc/tutorials/04-lattice_boltzmann/04-lattice_boltzmann_part3.ipynb

@@ -57,7 +57,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from __future__ import print_function, division\n",
     "import espressomd\n",
     "espressomd.assert_features(['CUDA', 'LENNARD_JONES'])\n",
     "from espressomd import System, interactions, lb, polymer\n",

doc/tutorials/06-active_matter/EXERCISES/enhanced_diffusion.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 import numpy as np
 import os
 import sys

doc/tutorials/06-active_matter/EXERCISES/flow_field.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 import numpy as np
 import os
 

doc/tutorials/06-active_matter/EXERCISES/rectification_geometry.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 from math import cos, pi, sin
 import numpy as np
 import os

doc/tutorials/06-active_matter/EXERCISES/rectification_simulation.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 from math import cos, pi, sin
 import numpy as np
 import os

doc/tutorials/06-active_matter/SOLUTIONS/enhanced_diffusion.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 import numpy as np
 import os
 import sys

doc/tutorials/06-active_matter/SOLUTIONS/flow_field.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 import numpy as np
 import os
 import sys

doc/tutorials/06-active_matter/SOLUTIONS/rectification_geometry.py

@@ -23,8 +23,6 @@
 #                                                                              #
 ##########################################################################
 
-from __future__ import print_function
-
 from math import cos, pi, sin
 import numpy as np
 import os