How to resolve flaky tests resulting from using a single GMT session

[weiji14]

Description of the problem

There's been instances of flaky tests in PyGMT's test suite reported in #1217 (comment). This likely stems from the fact that PyGMT uses a single GMT session (initiated during import pygmt) instead of separate GMT sessions for each figure (see #327 (comment)).

@meghanrjones asked about whether we should stick with using a single GMT session in #1217 (comment), or use independent sessions per figure

I understand the original logic behind a single GMT session for all tests in #327 (comment). Still, I don't expect that users will be attempting to use the entire PyGMT library in a single session, which is the goal of the test suite. So I think it would be worth revisiting this decision. Could it be possible to periodically test the examples/tutorials against baseline images to ensure that producing multiple plots in a single session is consistent and have the unit tests each use individual sessions?

Full code that generated the error

Flaky tests are hard to reproduce (that is their definition actually), but in PyGMT's case, can be found e.g. when a single test passing on pytest pygmt/tests/test_somemodule.py fails when ran using make test, or vice versa.

E.g. as reported by @meghanrjones in #1217 (comment)

edit: I have not yet been able to figure out a solution. The two makecpt tests fail if there is a docstring example that imports pygmt and instantiates a figure (e.g., extract_region() in pygmt/clib/session.py and pygmt/src/grdfilter.py) and is tested before pygmt/tests/test_makecpt.py.

Related issues affected by having a single GMT session:

• Support multiprocessing without importing gmt in each process #217
• Allow for "makecpt" style colormap functionality in grdimage and colobar #372 (comment)
• changing map annotation font size #733
• Strange behaviour when using Session.call_module(). #1582

System information

Please paste the output of python -c "import pygmt; pygmt.show_versions()":

PyGMT information:
  version: v0.3.2.dev117+g7466dc31
System information:
  python: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 16:07:37)  [GCC 9.3.0]
  executable: ~/username/miniconda3/envs/pygmt/bin/python
  machine: Linux-5.4.0-72-generic-x86_64-with-debian-bullseye-sid
Dependency information:
  numpy: 1.17.1
  pandas: 1.2.3
  xarray: 0.17.0
  netCDF4: 1.5.6
  packaging: 20.9
  ghostscript: 9.53.3
  gmt: 6.2.0rc1
GMT library information:
  binary dir: ~/username/miniconda3/envs/pygmt/bin
  cores: 6
  grid layout: rows
  library path: ~/username/miniconda3/envs/pygmt/lib/libgmt.so
  padding: 2
  plugin dir: ~/username/miniconda3/envs/pygmt/lib/gmt/plugins
  share dir: ~/username/miniconda3/envs/pygmt/share/gmt
  version: 6.2.0rc1

[weiji14]

Ok, the flakiness appears to have been an upstream GMT issue that was fixed in GenericMappingTools/gmt#3344. There are some tests that are wrong but currently passing (i.e. false positives) identified in #1217 (comment) and #1217 (comment) that need to be updated once we bump to GMT 6.2.0rc2.

• pygmt/tests/test_subplot.py Update fig.basemap, fig.subplot and fig.wiggle baseline images for GMT 6.2.0rc2 #1291
• pygmt/tests/test_text.py Update fig.text baseline images for GMT 6.2.0rc2 #1292
• pygmt/tests/test_wiggle.py Update fig.basemap, fig.subplot and fig.wiggle baseline images for GMT 6.2.0rc2 #1291

[maxrjones]

The past few flaky tests revealing GMT bugs have convinced me of the usefulness of the current structure, even though it would be nice to have the option to run tests in parallel.

[maxrjones]

We seem to be semi-regularly getting failures on windows-latest - Python 3.7 / NumPy 1.18 with :
..\tests\test_sph2grd.py::test_sph2grd_outgrid FAILED [ 87%]
..\tests\test_sph2grd.py::test_sph2grd_no_outgrid FAILED [ 87%]
due to issues with the remote file.

[weiji14]

We seem to be semi-regularly getting failures on windows-latest - Python 3.7 / NumPy 1.18 with : ..\tests\test_sph2grd.py::test_sph2grd_outgrid FAILED [ 87%] ..\tests\test_sph2grd.py::test_sph2grd_no_outgrid FAILED [ 87%] due to issues with the remote file.

Yes this has been popping up recently, but I don't think this is related to flakiness in a single GMT session since the error is Error: [ERROR]: Libcurl Error: Timeout was reached, so maybe open a separate issue for this.

[seisman]

https://forum.generic-mapping-tools.org/t/memory-temporary-storage-issues/5256 This post is a good example showing that using a single GMT session sometimes causes issues.