OK Writing and Running Tests

[khalilsarwari]

Addressing #63

This PR allows for OK to read and collect test files that users create independently.

Vision: An easy to use write/run testing setup with minimal interference from OK

The src files specified in the config.ok are crucial in deciding what gets tracked for coverage and what files are imported into the testing environment automatically.

The following run the tests in 'mytests.rst', unless another file is specified after -t:

Will return overall test coverage percent

    python3 ok -t
    python3 ok -t alttestfile.rst
    python3 ok -t -v --suite 1 --case 1

Will return overall test coverage percentage, as well as suggested lines to test

python3 ok -t -cov 

The hope now is to be able to include this so we could have students write their own tests for Ants. I know @papajohn was interested in having students be able to write tests, so I've requested his review here.

Example .rst student test file:

[khalilsarwari]

Looking forward to some feedback! (specifically regarding the PR or otherwise)

[epai]

@khalilsarwari Hmm, it looks like some tests are failing. Can you check and make sure no tests fail when you run nosetests tests?

[epai]

Ah, I think I see the reason. Make sure you push your support_files so your testing_test test cases pass.

[epai]

Tests must pass

Tests now pass, will re-evaluate

[papajohn]

That's very cool! Three things that we should consider:

1. We should also support students adding doctests directly in their regular source files. I'm not sure what happens in OK if they do this now. (Does OK always run all doctests, or only when asked?)

2. It would be nice to allow semantic names to suites and cases, instead of numbering. If the case numbering format is fixed, that's something to get wrong.

Lists:

    >>> master = [1, 2, 3]
    
    Pop:
        >>> master.pop()
        3
        >>> master.pop()
        2

    Extend:
        >>> master.extend([4, 5]) is None
        True
    
    ...

3. Could we change all of our staff-written OK tests to this format? (Perhaps in a different PR.)

[khalilsarwari]

After confirming with @epai :

1. OK runs some of its tests from the doctests that come with the source file(s), as well as some independent tests. All the doctests are run by default, and deleting some lines/spaces can make OK tests not pass. The idea of running these doctests from the source file(s) may have been a result of the difficulty in writing OK tests in the ok_test format.
   If this difficulty were not a significant issue, a potential solution would be to make these doctests independent by converting what is necessary to ok_tests, thus enabling students to modify and create doctests in the source files. (Another potential solution could be 3., redoing ok_tests as a whole.)

2. Support for semantic labels has now been added, and the overall format is actually pretty flexible, which I should clarify now. Suite 1 is fine, so is Suite Lists or Suite lists. So far, the only required format is Suite [/d/w]+ and Case [/d/w]+which I think is enough to parse with maximum freedom (comments, spacing etc).

3. We could! We would just have to add support for WWPD and Scheme questions, as well as support for other functionality such as locking, hiding, and scoring. I can look into it after this PR.

[papajohn]

Cool. Let's keep support for OK running doctests. Not everyone is going to learn how to write a .rst file. I'm out of town this week, so I'll let you all finish up reviewing and merging.

…

On Mon, Aug 14, 2017 at 11:57 PM, Khalil Sarwari ***@***.***> wrote: After confirming with @epai <https://github.com/epai> : 1. OK runs some of its tests from the doctests that come with the source file(s), as well as some independent tests. All the doctests are run by default, and deleting some lines/spaces can make OK tests not pass. The idea of running these doctests from the source file(s) may have been a result of the difficulty in writing OK tests in the ok_test format. If this difficulty were not a significant issue, a potential solution would be to make these doctests independent by converting what is necessary to ok_tests, thus enabling students to modify and create doctests in the source files. (Another potential solution could be 3., redoing ok_tests as a whole.) 2. Support for semantic labels has now been added, and the overall format is actually pretty flexible, which I should clarify now. Suite 1 is fine, so is Suite Lists or Suite lists. So far, the only required format is Suite [/d/w]+ and Case [/d/w]+which I think is enough to parse with maximum freedom (comments, spacing etc). 3. We could! We would just have to add support for WWPD and Scheme questions, as well as support for other functionality such as locking, hiding, and scoring. I can look into it after this PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#313 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATikTHXKR1P8_KfUtjVuedDHYuMZyg6ks5sYUF2gaJpZM4OY6Iz> .

[epai]

Is there a reason why failed and attempted variables are created instead of just using failed and rattempted?

[khalilsarwari]

Ah I didn't catch that!