Add Python docstrings (NanoComp#1240)

* First pass on tool to extract docstrings and add them to the main documentation.

* Add templates used for building the Python API docs

* Update docs README with new info

* Add link to new API docs. The old is still there for now.

* First pass on adding docstrings to the python code.

* Clarify the note in the README about when to rebuild

* Fix typo

* Refactor doc generator tool to pull items into the master doc by name, rather than as a collection of all classes found by the scan

* Fix typo

* Use spaces inside the tag markers. It's more readable.

* Reorder methods to ensure that __init__ comes first

* Look only at items that are in the class, not all inherited members.
This prevents a lot of noise in the documentation.

* Move remaining class and method docs to docstrings

* Fix typo in Simulation tag

* Regenerate document

* Copy recent changes from the non-template version of the document

* Only load the template files once, and other light refactoring

* Add doc generation support for functions

* Continue moving documentation to doctrings

* Add and export a GDSII_prisms function, for consistency with the other GDSII functions.

* Fix some copy/paste mistakes

* Expose the DtfFOO classes so their docstrings can be linked to

* Move and link dostrings for GDSII, Run and Step, and Output sections

* Move remaining documentation snippets to docstrings in the source.

* Revert Python_User_Interface.md to avoid merge conflicts later.

* Word-wrap and other formatting tweaks to help the docstrings be more readable in the code.

* More docstring formatting

* Typo fix

* Add coding declaration to the top of the Python files.

Python2 defaults to decoding source files to ASCII, so with non-ASCII characters in docstrings or other strings will result in a SyntaxError.

* Add a .css file for custom configuration of styles without touching the existing theme files

* Insert some <div class="..."> tags in the templates

* Add styles

* Ignore doc/site and an additional item in tests

* Backslashes in docstrings need to be escaped

* More template and style tweaks.

* Remove the now generated Python_User_Interface.md file

* Add support for building the documentation via the Makefile

* rewrap paragraph

* Set PYTHONPATH and use $(PYTHON)

* Convert README to Markdown

* `make dist` now depends on building the docs, and the docs are included in the dist archive

* More info and clarification in the doc/README.md file

* Include generate_py_api.py in the distribution

* Fix building docs when configured in a separate build folder

* Install the docs-related packages on Travis

* Python 2.7 mkdocs has broken dependencies, skip building the docs on Python 2.7 builds

* Python2 does not have f-strings

* Use Python2-style super()

* Also skip running the generator on Python2, for now.

* Move generate_py_api script to the doc folder,
enable it to be run from anywhere and still find the templates, etc.

* Restore the generated API doc

* cleanup

* Revert including the docs build in the `make dist` and clean up the doc targets

* Minor tweaks in doc/README.md

Co-authored-by: Robin Dunn <>

.gitignore

@@ -45,6 +45,7 @@ gen-ctl-io
 ctl-io.c
 ctl-io.h
 *.coverage
+doc/site
 scheme/ctl-io.cpp
 scheme/ctl-io.i
 scheme/geom.cpp
@@ -88,6 +89,7 @@ tests/cyl-ellipsoid-ll
 tests/cylindrical
 tests/dft-fields
 tests/flux
+tests/gdsII-3d
 tests/h5test
 tests/harmonics
 tests/integrate

.travis.yml

@@ -24,7 +24,7 @@ addons:
 ##################################################
 env:
   global:
-  - CPPFLAGS=-I${HOME}/local/include 
+  - CPPFLAGS=-I${HOME}/local/include
   - LDFLAGS=-L${HOME}/local/lib
   - HDF5_BASE_CPPFLAGS=-I/usr/include/hdf5
   - HDF5_SERIAL_CPPFLAGS=${HDF5_BASE_CPPFLAGS}/serial
@@ -59,6 +59,7 @@ matrix:
       - CPPFLAGS="${HDF5_SERIAL_CPPFLAGS} ${CPPFLAGS}"
       - LDFLAGS="${HDF5_SERIAL_LDFLAGS} ${LDFLAGS}"
       - BUILD_WITHOUT_MPB="1"
+      - SKIP_BUILD_DOCS="1"
     addons:
       apt:
         packages:
@@ -74,6 +75,7 @@ matrix:
       - CXX=mpic++
       - CPPFLAGS="${HDF5_SERIAL_CPPFLAGS} ${CPPFLAGS}"
       - LDFLAGS="${HDF5_SERIAL_LDFLAGS} ${LDFLAGS}"
+      - SKIP_BUILD_DOCS="1"
     addons:
       apt:
         packages:
@@ -90,6 +92,7 @@ matrix:
       - CXX=mpic++
       - CPPFLAGS="${HDF5_PARALLEL_CPPFLAGS} ${CPPFLAGS}"
       - LDFLAGS="${HDF5_PARALLEL_LDFLAGS} ${LDFLAGS}"
+      - SKIP_BUILD_DOCS="1"
     virtualenv:
       system_site_packages: true
     addons:
@@ -113,7 +116,7 @@ matrix:
           - *common_deps
           - libhdf5-serial-dev
     install:
-      - pip install numpy mpi4py scipy h5py coverage==4.5.4 coveralls matplotlib
+      - pip install numpy mpi4py scipy h5py coverage==4.5.4 coveralls matplotlib mkdocs==0.17.5 python-markdown-math mkdocs-material
   - python: "3.7"
     env:
       - MPICONF="--with-mpi"
@@ -128,7 +131,7 @@ matrix:
           - *common_deps
           - libhdf5-serial-dev
     install:
-      - pip install numpy mpi4py scipy h5py matplotlib
+      - pip install numpy mpi4py scipy h5py matplotlib mkdocs==0.17.5 python-markdown-math mkdocs-material
   - python: "3.5"
     env:
       - MPICONF="--with-mpi"
@@ -147,7 +150,7 @@ matrix:
           - libhdf5-openmpi-dev
           - python3-h5py
     install:
-      - pip install numpy mpi4py scipy matplotlib ipython
+      - pip install numpy mpi4py scipy matplotlib ipython mkdocs==0.17.5 python-markdown-math mkdocs-material
 
 
 ##################################################

Makefile.am

@@ -17,7 +17,7 @@ endif
   SUBDIRS += python
 endif
 
-EXTRA_DIST = NEWS.md LICENSE COPYRIGHT m4 meep-pkgconfig.in
+EXTRA_DIST = NEWS.md LICENSE COPYRIGHT m4 meep-pkgconfig.in doc
 
 pkgconfigdir = $(libdir)/pkgconfig
 pkgconfig_DATA = meep.pc
@@ -28,4 +28,31 @@ meep.pc: meep-pkgconfig
 libmeep:
 	cd src && $(MAKE)
 
+# Generate the Python API Document
+# PYTHONPATH is set to use the in-place build of meep
+python_api_doc: all
+	if test '$(top_srcdir)' != . ; then \
+		cp -rf $(top_srcdir)/doc .; \
+		chmod -R +rw doc; \
+	fi
+	if test -z $$SKIP_BUILD_DOCS; then \
+		PYTHONPATH=./python $(PYTHON) doc/generate_py_api.py; \
+	fi
+
+# Build the HTML version of the docs
+docs: python_api_doc
+	if test -z $$SKIP_BUILD_DOCS; then \
+		$(PYTHON) -m mkdocs build -f $(top_srcdir)/mkdocs.yml; \
+	fi
+
+# Build a tarball containing just the html docs
+docs-dist: docs
+	tar czvf meep-documentation-$(PACKAGE_VERSION).tgz doc/site
+
+clean-local:
+	rm -f meep-documentation-$(PACKAGE_VERSION).tgz
+	if test '$(top_srcdir)' != . ; then \
+		rm -rf doc; \
+	fi
+
 DISTCLEANFILES = meep.pc

doc/README.md

@@ -0,0 +1,62 @@
+This is the documentation tree for MEEP.
+
+markdown (.md) files are in the `doc/docs` folder.
+
+To build and visualize the HTML documentation locally using the
+mkdocs package (useful for verifying changes on your local machine
+before committing), first install `mkdocs` (version 0.17.5) as
+well as two auxiliary packages via e.g.:
+
+```
+% pip3 install --user mkdocs==0.17.5 python-markdown-math mkdocs-material
+```
+
+To (re)generate the Python API documentation (extracted from the docstrings)
+run the following command in the project root folder. Note that this requires
+that the Python extension for the meep library has been built:
+
+```
+% PYTHONPATH=./python python doc/generate_py_api.py
+```
+
+Rerun this after making any changes to the docstrings in the source and
+rebuilding the project, in order to update the documentation.
+
+Next, run the following command from the top-level MEEP repository tree:
+
+```
+% mkdocs serve
+```
+
+Finally, open the following address in a browser window:
+
+http://127.0.0.1:8000
+
+This launches a web server on your local machine plus a filesystem hook for
+rebuilding the documentation tree automatically whenever any .md file is
+modified. This enables viewing the HTML documentation in real time as the
+source files are edited.
+
+To build the HTML version of the documentation, run:
+
+```
+% mkdocs build
+```
+
+The results will be put into the `./doc/site` folder.
+
+Alternatively, the project `Makefile` can also be used to build the
+documentation and to build a distribution archive containing the
+documentation. To do so, run one of the following commands from the
+top-level MEEP repository tree:
+
+```
+# Only regenerate the Python API document, useful if you're using `mkdocs serve`
+% make python_api_doc
+
+# Regenerate the API doc and run `mkdocs build`
+% make docs
+
+# Bundle the HTML version of the document files into a tarball archive
+% make docs-dist
+```

doc/_api_snippets/class_template.md

@@ -0,0 +1,20 @@
+---
+<a id="{class_name}"></a>
+
+### {class_name}
+
+```python
+class {class_name}({base_classes}):
+```
+
+<div class="class_docstring" markdown="1">
+
+{docstring}
+
+</div>
+
+<div class="class_members" markdown="1">
+
+{method_docs}
+
+</div>

doc/_api_snippets/function_template.md

@@ -0,0 +1,14 @@
+
+<a id="{function_name}"></a>
+
+**meep.{function_name_escaped} function**
+
+```python
+def {function_name}{parameters}:
+```
+
+<div class="function_docstring" markdown="1">
+
+{docstring}
+
+</div>

doc/_api_snippets/method_template.md

@@ -0,0 +1,14 @@
+
+<a id="{class_name}.{method_name}"></a>
+
+**{class_name}.{method_name_escaped}**
+
+```python
+def {method_name}{parameters}:
+```
+
+<div class="method_docstring" markdown="1">
+
+{docstring}
+
+</div>

doc/docs/Python_Developer_Information.md

@@ -73,7 +73,7 @@ Pure Python implementations of the geometric objects defined in `libctl`. The us
 
 ### `simulation.py`
 
-Holds the `Simulation` class, which is the primary abstraction of the high-level interface. Minimally, a simulation script amounts to passing the desired keyword arguments to the `Simulation` contructor and calling the `run` method on the resulting instance. The various step functions are also included in this file. When adding extra functions or class to this file, an import statement should also be added to the `%pythoncode` block at the bottom of `python/meep.i`.
+Holds the `Simulation` class, which is the primary abstraction of the high-level interface. Minimally, a simulation script amounts to passing the desired keyword arguments to the `Simulation` constructor and calling the `run` method on the resulting instance. The various step functions are also included in this file. When adding extra functions or class to this file, an import statement should also be added to the `%pythoncode` block at the bottom of `python/meep.i`.
 
 ### `source.py`