Implement an 'audit' reporter that will terminate Chef Client runs on profile failures

[sbabcoc]

Description

This pull request implements an audit enforcer reporter to replace the fail_if_any_audits_failed feature that was removed a while ago.

Issues Resolved

[List any existing issues this PR resolves]

Check List

• All tests pass. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/TESTING.MD
• New functionality includes testing.
• New functionality has been documented in the README if applicable
• All commits have been signed for the Developer Certificate of Origin. See https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD

[sbabcoc]

Looks like the Travis failures were caused by a configuration issue that's been resolved.

[sbabcoc]

Given that the purpose of this reporter is to throw an error, I don't know how to implement a test for it.

[alexpop]

Thank you Scott for bringing this option back into the audit cookbook.

I pushed a commit to this branch:
https://github.com/chef-cookbooks/audit/commits/pr-380

Can you cherry-pick that into your fork please? It adds unit tests for AuditEnforcer and takes care of lint errors.

[sbabcoc]

Merged unit tests and linting fixes from @alexpop

[sbabcoc]

I reviewed the changes made by @alexpop, and they look great!

[alexpop]

Much appreciated Scott!

[clintoncwolfe]

As far as I can see, this will make a reporter that fails. I'm not sure how much usefulness that has, since it is my understanding that the chef client run has already completed at this point, and you would only have an error in the reporter, not a node that failed to converge; I may be mistaken.

Of more concern to me is that this is to replicate a feature that was withdrawn. I'm trying to get context on why fail_if_any_audit_failed was removed, and whether those reasons are any different today.

Finally, the TravisCI failure is a linting failure - we need those addressed before we can accept. Pinning the chefstyle gem in the Gemfile may be advisable. Thanks!

[sbabcoc]

The pull request that added the fail_if_any_audits_failed feature is here: #25
The pull request that removed this feature is here: #113

Here's the specific comment:

jeremymv2 on Oct 19, 2016 Contributor
Since we're moving away from a resource converge I don't think 'fail_if_any_audits_failed' makes sense any more. The original intent was to fail during converge in a pipeline phase such as kitchen converge. Now, with the report handler, we'd rely on kitchen-inspec to reference the profiles and kitchen verify to provide a pass|fail status to the CI framework.

The intent of my PR is to fail actual chef-client runs when non-compliance is detected. The comment attached to the removal of the fail_if_any_audits_failed feature refers to kitchen, which is a completely different context.

[sbabcoc]

As for usefulness, this enables me to terminate a pipeline if a non-compliant node is encountered. The current behavior is to passively report the failure and continue to the next stage, which will then choke because of the previously detected issue.

[sbabcoc]

Sorry, it's really easy to close pull requests accidentally in the mobile view of GitHub.

[sbabcoc]

@alexpop has been incredibly helpful on this PR! Many thanks!
He discovered that the audit enforced raises the error, but the audit cookbook is eating the failure. This happens in the run_report_safely method, which catches exceptions and only re-throws them if the fail_if_not_present attribute is specified. I'll push revisions that will also re-throw for audit enforcer.

[clintoncwolfe]

I see, it took me a bit to get when this would fire - thanks for this!