Initial unit testing framework and tests #7026

jcsteh · 2017-03-29T04:09:15Z

This adds an initial unit testing framework, including necessary changes for SCons and AppVeyor. It also adds some tests for the cursorManager module, which serve both as useful tests and as an example of how this can be used.

This is far from perfect. In particular, there is some pretty ugly bootstrapping code, and because NVDA wasn't originally designed with unit testing in mind, tests aren't isolated from side effects. However, we need to start somewhere and I think we can improve these things over time. Most of the ugliness is in one file and we have a base TestCase so we can provide base setUp and tearDown methods in order to better isolate tests.

Unit tests and infrastructure are located in the tests/unit directory. See the docstring in the tests/unit/__init__.py file for details. You can run tests using "scons tests". See the "Running Tests" section of readme.md for details.

…l__ when unit tests finish (and when NVDA exits). These have existed for a long time, but they aren't normally noticed when exiting NVDA, since we normally don't see stderr when exiting. To fix this, nvwave and tones now register an atexit callback to clean up their WavePlayer instances before modules get cleaned up.

…comInterfaces files to be rebuilt unnecessarily. This is not a new problem, but it's particularly annoying for automated testing because running the tests often causes files to be rebuilt unnecessarily. 1. The ISimpleDOM and MathPlayer interfaces have multiple idl files. The targets for the idl file have an implicit dependency on midl.exe, but its position in the dependencies is different in the run before the idl files are copied versus subsequent runs. I think this is an SCons bug/quirk, probably because before the copy, the idl file must be scanned after the midl tool is considered. Explicitly specifying these extra idls as dependencies doesn't help. The only workaround I've found is to ignore midl.exe as a dependency when checking whether the targets should be rebuilt. 2. comInterfaces generates .pyc as well as .py files. We now tell SCons about the .pyc files as well so they get considered appropriately. Previously, the .py file could be removed, so SCons would rebuild the .py file. However, the .pyc still existed, so it was just loaded without generating a .py file. Subsequently, SCons would try to rebuild the .py file on every build.

feerrenrut · 2017-03-31T04:49:21Z

.gitignore

@@ -39,3 +39,4 @@ uninstaller/UAC.nsh
 *.pyc
 *.pyo
 *.dmp
+tests/unit/nvda.ini


This is probably fine for now. But in the future this might need to pulled into the working directory for the test (even better if it was in memory rather than moving actual files around) during the setup for the test. If all tests use the same config file, there is a chance that some test will modify the settings and we could get some pretty nasty test interactions.

Totally agree that ideally, we want to keep this in memory and isolate it per test. If it eases your concern any in the short-term, it's worth noting that nothing in NVDA ever really saves the configuration except the save configuration command or auto save on NVDA exit, neither of which should ever be reasonably called from a unit test.

feerrenrut · 2017-03-31T06:08:22Z

appveyor.yml

+ # We don't need launcher to run tests, so run the tests before launcher.
+ - py scons.py tests %sconsArgs%
+ - 'echo scons output targets: %sconsOutTargets%'
+ - py scons.py %sconsOutTargets% %sconsArgs%


Each of these lines ending with %sconsArgs% (lines: 76, 78, 80) has different line ending ^M

I don't know why GitHub is flagging this. All of these files have Windows line endings all the way through (as is generally our convention)... and looking at the raw content (outside of an editor), there's nothing special about these lines.

feerrenrut · 2017-03-31T06:33:06Z

nvdaHelper/ISimpleDOM_sconscript

@@ -24,5 +24,10 @@ tlbFile,headerFile,iidSourceFile,proxySourceFile,dlldataSourceFile=env.TypeLibra
 	source=idlFile,
 	MIDLFLAGS=['/c_ext','/I',Dir('.')],
 )
+# hack: Ignore midl.exe when deciding to rebuild, as its position in the dependencies


Do we want to open an issue somewhere to keep track of this? It might be helpful for someone who wants to investigate, and see what has been tried. Having the issue number in the comments really helps people track down where the information is.

feerrenrut · 2017-03-31T06:35:22Z

source/nvwave.py

+
+# When exiting, ensure fileWavePlayer is deleted before modules get cleaned up.
+# Otherwise, WavePlayer.__del__ will fail with an exception.
+@atexit.register


There is a windows line ending here too.

feerrenrut · 2017-03-31T06:35:41Z

source/tones.py

@@ -20,6 +21,13 @@
 	log.warning("Failed to initialize audio for tones")
 	player = None

+# When exiting, ensure player is deleted before modules get cleaned up.
+# Otherwise, WavePlayer.__del__ will fail with an exception.
+@atexit.register


There is also a windows line ending here.

feerrenrut · 2017-03-31T06:53:47Z

tests/unit/__init__.py

+
+import unittest
+
+class TestCase(unittest.TestCase):


As discussed, I dont think we need this for now. We will need some way of managing setup/tear down of dependencies, but perhaps we should address this later.

feerrenrut · 2017-03-31T06:57:28Z

tests/unit/__init__.py

+# We don't want logging.
+import logging
+from logHandler import log
+log.addHandler(logging.NullHandler())


As discussed, I think it could be helpful to be able to get the log information out of some tests. The log info could make debugging the unit tests easier. Could you open (and p4) an issue to keep track of the different ways we talked about to achieve this? I think for now there is low urgency on actually implementing this, however at the point when someone is trying to debug a unit test this could become more important.

feerrenrut · 2017-03-31T06:59:06Z

tests/unit/test_cursorManager.py

+		self.assertEqual(cm.selectionOffsets, (1, 1)) # Caret at "b"
+
+	def test_prevChar(self):
+		cm = CursorManager(text="abc", selection=(1, 1)) # Caret at "b"


Rather than the comment, the (1,1) could be held in a variable named caretAtB

feerrenrut · 2017-03-31T07:00:45Z

tests/unit/test_cursorManager.py

+		self.assertEqual(cm.selectionOffsets, (0, 0)) # Caret at "a", no selection
+
+	def test_selForwardThenUnselThenSelBackward(self):
+		"""Test selecting forward, then unselecting and selecting backward.


The docstring could be more explicit about the dependencies between tests / behaviour.

jcsteh · 2017-04-03T10:23:34Z

As discussed with @michaelDCurran and @feerrenrut, we'll incubate this one for a day or so just to make sure there's no serious breakage in the build or the like. However, we want to get it into master sooner rather than later so we can actually start basing branches off it and getting unit tests. Also, there's no notable code change here for NVDA itself, so this should have no effect for users.

This adds an initial unit testing framework, including necessary changes for SCons and AppVeyor. It also adds some tests for the cursorManager module, which serve both as useful tests and as an example of how this can be used. Unit tests and infrastructure are located in the tests/unit directory. See the docstring in the tests/unit/__init__.py file for details. You can run tests using "scons tests". See the "Running Tests" section of readme.md for details.

jcsteh · 2017-04-12T04:06:05Z

Oops. I accidentally squash merged this when I meant to do a normal merge. While we normally squash merge, there is context in these commits which is quite important to figure out why I made some seemingly unrelated changes.

I force pushed to fix this. Just so we have this documented:

The old merge was 423463e. The original What's New commit was 8d0fe3d.
I undid both of these commits, merged manually, cherry-picked the What's New entry and force pushed.
The new merge commit is 948abb1. The new What's New entry commit is 8120e75.
I double checked there were no commits between my original What's New commit and my force push.
I confirmed that the tree between the old head and the new head is the same:
```
$ git diff 8d0fe3df..8120e75c
$
```

jcsteh added 5 commits March 29, 2017 14:02

Add initial unit testing framework.

7e21223

Unit tests and infrastructure are located in the tests/unit directory. See the docstring in the tests/unit/__init__.py file for details. You can run tests using "scons tests". See the "Running Tests" section of readme.md for details.

Add some tests for the cursorManager module.

baad666

AppVeyor: Run automated tests.

a8f946e

jcsteh requested a review from feerrenrut March 29, 2017 04:09

feerrenrut approved these changes Mar 31, 2017

View reviewed changes

This was referenced Mar 31, 2017

scons: Fix changing order of dependencies when building an interface with multiple IDL files #7036

Open

Provide ability to see log output when debugging unit tests #7045

Open

Address code review comments.

bddfab6

jcsteh force-pushed the unitTests branch from 3d04cfb to bddfab6 Compare April 3, 2017 03:47

jcsteh added a commit that referenced this pull request Apr 3, 2017

Iinitial unit testing framework and tests. Incubates #7026.

1fbf43a

nvaccessAuto added the incubating label Apr 3, 2017

nvaccessAuto assigned jcsteh Apr 3, 2017

jcsteh merged commit 423463e into master Apr 12, 2017

nvaccessAuto removed the incubating label Apr 12, 2017

nvaccessAuto added this to the 2017.2 milestone Apr 12, 2017

jcsteh deleted the unitTests branch April 12, 2017 03:42

jcsteh added a commit that referenced this pull request Apr 12, 2017

Update What's New for PR #7026.

8d0fe3d

jcsteh added a commit that referenced this pull request Apr 12, 2017

Update What's New for PR #7026.

8120e75

derekriemer mentioned this pull request Jul 3, 2017

Automated system testing framework #708

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial unit testing framework and tests #7026

Initial unit testing framework and tests #7026

jcsteh commented Mar 29, 2017

feerrenrut Mar 31, 2017

jcsteh Mar 31, 2017

feerrenrut Mar 31, 2017

jcsteh Mar 31, 2017

feerrenrut Mar 31, 2017

feerrenrut Mar 31, 2017

feerrenrut Mar 31, 2017

feerrenrut Mar 31, 2017

feerrenrut Mar 31, 2017

feerrenrut Mar 31, 2017

feerrenrut Mar 31, 2017

jcsteh commented Apr 3, 2017

jcsteh commented Apr 12, 2017

Initial unit testing framework and tests #7026

Initial unit testing framework and tests #7026

Conversation

jcsteh commented Mar 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcsteh commented Apr 3, 2017

jcsteh commented Apr 12, 2017