-
Notifications
You must be signed in to change notification settings - Fork 685
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add unit tests of XCCDF 1.2 elements #9617
Conversation
I believe that the 2 similar blocks of code are OK because they work with different items: rules and groups. |
I am not a big fan of tests that contain an overwhelming amount of low-level asserts. Those tests may be challenging to maintain, because when an assert starts to fail, what to do? Modify the test, modify test data, or look into the source code? In some cases when the situation is unclear and time pressure is present, a wrong action may be taken. |
It can cause tracebacks when we call a method on None object. This doesn't happen during normal usage, but it can happen during unit tests.
This adds simple unit tests for _to_xml_element() methods of Rule, Profile, Value, and Group class in ssg.build_yaml module.
14f634d
to
faf7299
Compare
I have resolved the conflicts and rebased the PR on the top of the latest upstream master branch. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want this kind of path in the repo? I agree with @matejak regarding the number low level asserts.
I have removed the hard coded definition location paths. |
We will use xmldiff in unit tests of to_xml_element methods because it allows us to store expected XML outputs as separate files and also it allows us to detect all differences from the expected output immediately.
614edea
to
50a029a
Compare
The conditional import will make skip tests that require xmldiff and lxml because xmldiff isn't present on all systems where we would like to run our tests because xmldiff can be at this moment installed only using pip, it isn't packaged in most of Linux distributions.
We have discussed this to @matejak and we have found out it's also possible to use xmldiff with this type of tests it allows us to store expected XML outputs as separate files and also it allows us to detect all differences from the expected output immediately. I have add conditional imports to make the tests skip that require xmldiff and lxml when they aren't available because xmldiff isn't present on all systems where we would like to run our tests because xmldiff can be at this moment installed only using pip, it isn't packaged in most of Linux distributions. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this approach better. I think we should document the need for the xmldiff
python package in the building CaC doc.
Can you add a new line of the files like tests/unit/ssg-module/data/accounts_tmout.xml
or will that cause issues with the XML diff?
I think we can waive the code climate duplication errors.
I have add documentation text.
Unfortunately, xmldiff is sensitive on whitespace before and after tags. I don't like this behavior of xmldiff, but it seems that we will either have to live with that or invent some sort of whitespace normalization that would be executed before the test data would be passed to xmldiff. @matejak has pointed to me https://xmldiff.readthedocs.io/en/stable/api.html#using-formatters but we have found that it's related to the diff output instead of input. Then I did an experiment where I add a newline after a tag to the test data and whatever argument I put into this |
But now, an unexpected twist: we have found that this isn't true for the So we can save the actual outputs produced by the SUT to temporary files and then compare the temporary files with the test data files using the |
Originally, we thought xmldiff is sensitive on whitespace before and after tags. But now we have found that this isn't true for the diff_files method. When using the diff_trees it is sensitive on whitespace and when using diff_files method it isn't sensitive on whitespace. So we can save the actual outputs produced by the SUT to temporary files and then compare the temporary files with the test data files using the diff_files.
I have se diff_files instead of diff_trees |
Code Climate has analyzed commit 4b98c1a and detected 2 issues on this pull request. Here's the issue category breakdown:
The test coverage on the diff in this pull request is 0.0% (50% is the threshold). This pull request will bring the total coverage in the repository to 41.9% (0.6% change). View more on Code Climate. |
Description:
This PR adds simple unit tests for
_to_xml_element()
methods of Rule, Profile, Value, and Group class in ssg.build_yaml module.The unit test fixtures will use example data created from real-world resolved rules, profile, value, and group that were taken from build executed on current master.
Rationale:
Increases testing of build scripts.