Testing

[KGergo88]

The following topics need to be done:

Tool evaluations

• unittest - Python standard library unit testing framework
  • Link: https://docs.python.org/3/library/unittest.html
  • Decision: This will be our primary tool for the unit testing
• pytest - 3rd party testing framework for python
  • Link: https://docs.pytest.org/en/latest
  • Decision: will not use it because of bad experiences with other projects
• Coverage.py - Tool to create code coverage reports for the tests
  • Compatibility: Works with unittest and pytest
  • Link: https://coverage.readthedocs.io/en/v4.5.x/
  • Decision: Can be used for checking the unit test coverage
• Hypothesis
  • Compatibility: Works with unittest and pytest
  • Link: https://hypothesis.readthedocs.io/en/latest/index.html
  • Decision: Can be used for some tests parallel to unittest

Project structure

• The issue needs to be developed on the ISSUE_#3_Testing branch
• Define the project structure that has a folder for the tests and below that subfolders for unit tests, functional tests and other test relevant tools/data/mapfiles/configurations/scripts
• Create a script to generate code coverage reports for the unit tests

Unit testing

• Test first the most important functions and classes:
  • memoryMap.py::emma_libs.memoryMap.caluclateObjectsInSections
  • memoryManager.py::resolveDuplicateContainmentOverlap()
  • memoryManager.py::importData
  • memoryEntry.py::MemEntry
• Develop full tests for the following files:
  • Files TBD...
• Further points TBD...

Functional testing

• Call Emma with various wrong command line arguments
• Call Emma-vis with various wrong command line arguments
• Call Emma-Delta with various wrong command line arguments
• Create functional tests for the Emma with test_project (this is a precondition for Memory Manager Redesign #5)
• Test the dependencies of arguments
• Make tests with mapfiles that have a wrong structure/syntax
• Make tests with configurations that are ill formed:
  • globalConfig.json without any configID defined in it
  • A config with an address spaces file that has an ignored memory entries that do not exist in it
• Categorisation
  • If a category does not match, we shall get a warning
• Call Emma with no map files at all and check for the error message (exit code -10)
• TBD...

Corrections needed based on the first review of unit tests

• Test naming (was corrected in 641a7fb): b7bfe55#r34194228
• Constant naming (was corrected in e70596d): b7bfe55#r34194285