For all items in MUST / SHOULD / COULD / WOULD BE NICE, the following legends are defined.
| Legend | Means |
|---|---|
| [d] | Done - implemented |
| [o] | Ongoing - currently working on |
| [t] | To do - waiting to be started |
A test function, defined by the function name ending in ‘_TEST’ contains a variable assignment to the variable ‘expect’ in the UT namespace (#.UT.expect) which defines the expected return value of the function.
∇ Z ← one_plus_one_TEST #.UT.expect ← 2 Z ← 1 * 1 ∇
Test functions are executed using UT.run with the test function name as the argument
#.UT.run 'one_plus_one_TEST'
Upon execution of the test function (one_plus_one_TEST), the returned value (in Z) is checked against the expecte value (in UT_expected).
If the returned values does not match the expected value, the test is considered as failed, appropriate text is displayed, clearly outlining the Expected value and the Actual value, along with the name of the test.
FAILED: one_plus_one_TEST Expected 2 Got 1
If the returned value of the function matches the expected value, the test is considered as passed and a passed message is printed
UT.run 'one_plus_one_TEST' Passed
It shall be possible to execute multiple unit tests from an array of unit tests by name
Tests ← 'a_TEST' 'b_TEST' 'c_TEST' 'd_TEST' #.UT.run Tests
The result of the execution of tests shall display the result of each test as if executd by #.UT.run There shall also be an agregated result displayed on the screen at the end of the execution.
Test execution result ⍋ Passed: 20 ⍒ Failed: 3
A crashing test must not cause subsequent tests not to be executed. Crashing test cases are reported in the final result as ‘Crashed’ when executing a list of testcases.
⍋ Passed: 4 ⍟ Crashed: 2 ⍒ Failed: 1
On single test case execution, the test will be reported as failed with the following output, where ${DIAGNOSTIC_MESSAGE} is replaced with the ⎕DM of the crash
CRASHED: one_plus_one_TEST
Expected
2
Got
${DIAGNOSTIC_MESSAGE}
It shall be possible to epxress that the test will result in an exception and to set what Diagnostic Message is to be expected
∇ exception_generating_TEST
#.UT.exception ← 'DOMAIN_ERROR'
Z ← 1 ÷ 0
∇
Upon execution of such a test, the test is considered successfull if the test function generates an exception which has a ⎕DM matching the #.UT.exception text.
On failure, a message is displayed, clearly indicating the expected and actual diagnostic messages.
FAILED: failing_error_TEST
Expected
┌→───────────┐
│DOMAIN ERROR│
└────────────┘
Got
┌→─────────┐
│RANK ERROR│
└──────────┘
On success, the output is ‘Passed’.
Having multiple Single line tests in a file, it must be possible to execute them all in one go.
UT.run '/path/to/File.dyalog'
During the execution of the test cases, every passed/failed/crashed test is displayed as in the single test/multiple test execution.
At the end of the execution, an aggregated result is printed to the screen. The amount of Passed/Crashed/Failed testcases displayed.
/path/to/File.dyalog tests ⍋ Passed: 10 ⍟ Crashed: 3 ⍒ Failed: 0
Executed tests include all _TEST functions (functions whose name ends in _TEST). Such as ‘this_TEST’.
Arrays containing testcases will not be executed.
It shall be possible to generate a coverage report of selected functions as a result of unit test execution.
Coverage report generation is requested by creating an instance of the UTcover class
Conf ← ⎕NEW UTcover
Setting the cover page generation output directory property
Conf.pages ← '/home/APL/coverage'
And setting the array of functions to cover property, the functions to cover are targeted through the corresponding namespace and function. In the example below, we are targeting the plus_function in the Example namespace.
Conf.cover ← ⊂ '#.Example.plus_function'
This instance is then given as left argument to the ordinary UT.run function.
Conf ← ⎕NEW UTcover Conf.pages ← '/home/APL/coverage' Conf.cover ← ⊂ '#.Example.plus_function' Conf UT.run 'one_plus_one_TEST'
The resulting HTML page will be generated to a file in the path defined by the user through the parameter ‘UT.pages’. The name of the page will be determined by the type of Tests being executed (the right argument of UT.run).
| Coverage on | Page Output in | Page Name |
|---|---|---|
| Conf #.UT.run ‘a_TEST’ | Conf.pages | a_TEST_coverage.html |
| Conf #.UT.run ‘a_TEST’ ‘b_TEST’ | Conf.pages | list_coverage.html |
| Conf #.UT.run ‘/t/File.dyalog’ | Conf.pages | File_coverage.html |
Thus, the example
Conf ← ⎕NEW UTcover Conf.pages ← '/home/APL/coverage' Conf.cover ← ⊂ '#.Example.timetable_selector' Conf UT.run 'out_of_bound_TEST' Passed
Will generate a coverage result for the function timetable_selector, the result coverage page is written to the file ‘/home/APL/coverage/out_of_bound_TEST.html’
The Coverage result page will have a timestamp indicating when the page was generated.
The timestam will have the format
YYYY-MM-DD HH:mm:SS
and be located at the bottom of the page, together with the text
Page generated:
So, an example page will have the following text at the bottom of the page.
Page generated: 2013-2-19 | 19:11:36
The reason to put this info at the bottom of the page, is that the most interesting part is the coverage.
When executing #.UT.run ${PATH} if ${PATH} is a directory, then find all APLUnit test files and execute them in order.
UT.run '/path/to/directory/'
APLUnit test files are SALT scripts whone name ends in ‘_tests.dyalog’.
Currently this only works for *nix as it used the OS level test -d
When setting the coverage configuration, it should be possible to only passing the namespace to the cover property of the Cover conf object.
Conf ← ⎕NEW UTcover Conf.pages ← '/home/APL/coverage' Conf.cover ← ⊂ '#.Example'
Running the Unit Tests from the Example_tests.dyalog file should now generate a full coverage report of all functions defined in #.Example
When writing Unit Tests, it shall be possible to express a non-equality for the result. This is done by exchanging the expect with nexpect.
∇ Z ← dyadic_iota_will_not_give_array_TEST Z ← 1 2 ⍳ 1 #.UT.nexpect ← 1 ∇
The non equality shall behave the same as the equality, except that the condition for success is the actual argument NOT being equal to the result.
When executing multiple tests with #.UT.run, the resulting output on the screen should be suppressed
.....
Passed
Passed
Passed
-----------------------------------------
../numbername_tests.dyalog tests
⍋ Passed: 45
⍟ Crashed: 0
⍒ Failed: 0
Without the currently following
#.UT.[UTresult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[U
Tresult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTr
esult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTres
ult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTresult] #.UT.[UTresul
....
Instead of using a specific Coverage configuration as the left argument to UT.run, it should accept an unordered keyed list (proplist) carrying the configuration parameters. This allows extension to be performed easier.
This change will break backwards compatibility with the currently existing tests. It is important that the merge point labels this clearly.
C ← ⍬
C,← ⊂('cover_out' '/home/APL/coverage/')
C,← ⊂('cover_target' ('#.Demo.this_TEST' '#.Demo.something_TEST'))
C,← ⊂('skip' ⊂'#.Demo.count_zero_comments_from_no_input_TEST')
C UT.run 'out_of_bound_TEST'
It shall be possible to quickly reload all dyalog SALT scripts from within the Dyalog environment without too much ceremony. For this to work, a global configuration parameter is set, pointing to the root path of the Dyalog Application.
#.UT.appdir ← './path/to/app'
There is an implicit assumption here, that all Dyalog applications following this style has two directories, one ‘src/’ and one ‘test/’, found under ‘./path/to/app’
./path/to/app/src/ ./path/to/app/test/
within these directories are one or more dyalog SALT scripts which can be loaded using ⎕SE.SALT.Load. All tests are supposed to be under test/ and all implementation source is assumed to be under src/
Upon having APLUnit already loaded in the system, running the usual test commands will always start by first loading all the Dyalog salt scripts into the current namespace first, and then proceeding as usual.
The path can be relative and absolute, just as passed to SALT. To disable this behavior, ⍬ can be assigned.
#.UT.appdir ← ⍬