Add regression tests for pay leave for parents #1986
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is mainly for discussion at the moment so I'm going to add the "Don't merge" label. |
|
Other than my comments, this looks good to me. 👍 |
chrisroos
force-pushed
the
add-regression-tests-for-pay-leave-for-parents
branch
from
6d0710a to
180e50d
Compare
|
I've updated this branch, rebased on master and force pushed. I've added a number of additional responses to increase the code coverage of this Smart Answer. I'm confident that we're now reaching all the conditional branches in the outcomes/snippets. I'm not sure about the changes in the last commit (titled, "Only run pay-leave-for-parents tests when required") but I'm hoping it's good enough for now. |
|
Other than my most recent comment, this looks good to me. |
Generated using:
$ rails r script/generate-questions-and-responses-for-smartdown-flow.rb \
pay-leave-for-parents
I've arbitrarily chosen salary amounts of £10,000/year. I came up with the due dates up by running the regression tests multiple times until I was confident that I was covering all code branches in the outcomes/snippets. I did this by adding marker text in the various conditionals, running the regression tests and then comparing the marker text in the test artefacts to the marker text in the outcomes/snippets. This set of due dates allows me to reach each path. This approach helped me find some unused code paths which I've removed in PR #2006.
Generated using:
$ rails r script/generate-responses-and-expected-results-for-smartdown-flow.rb \
pay-leave-for-parents
chrisroos
force-pushed
the
add-regression-tests-for-pay-leave-for-parents
branch
from
180e50d to
39807a6
Compare
Generated using:
$ rails r script/generate-checksums-for-smart-answer.rb \
pay-leave-for-parents
Generated using:
$ RUN_REGRESSION_TESTS=pay-leave-for-parents \
ruby test/regression/smart_answers_regression_test.rb
These tests are in place ready for the conversion of this Smart Answer from Smartdown to Ruby. We don't need them to run as part of the main regression test suite until the Smart Answer has been converted. This isn't the best implementation but I think it's probably OK given that I'll revert it as soon as this Smart Answer has been converted to Ruby.
chrisroos
force-pushed
the
add-regression-tests-for-pay-leave-for-parents
branch
from
39807a6 to
a2994ae
Compare
|
I've rebased on master and force pushed. Merging to master now. |
chrisroos
added a commit
that referenced
this pull request
Oct 15, 2015
…eave-for-parents Add regression tests for pay leave for parents
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Now that pay-leave-for-parents has been converted to single-question-per-page we're able to generate regression tests. These tests should give us confidence that we haven't broken anything when we convert from Smartdown to Ruby Smart Answers.
I'm particularly interested in feedback on the first (currently marked as "TEMP") commit. We need to preload the Smartdown flow to avoid the regression tests taking about 20 seconds each. However, we don't always want to preload as that causes the non-Smartdown tests to take longer than necessary. I'm kinda happy with the hacky approach but I think the bit that's missing is resetting the
preload_flowsoption in some kind of teardown hook. I haven't looked at whether that might be possible yet.