Cannot report meta-profiles to Chef Compliance

[chris-rock]

The current version of the audit cookbook does not support to report meta-profiles as defined here: inspec/inspec#1283 to Chef Compliance. This is caused by the json-min reporter.

[chris-rock]

Hi @alexpop this is my hack to run meta-profiles: b281654 The issue is that the json-min is including information about the different profiles, but we just report as one profile. We need to find a way that enables us to not reverse engineer the profile information (as we do right now in audit cookbook). Therefore we have two options:

• extend json-min information in InSpec
• Change the reporting API endpoint in Chef Compliance (to consume the same JSON as we get from a plain inspec exec call)
• generate json-min from json reporter and reuse the additional information to fulfill the need for the API

Lets first gather some information and add this to our ticket:

• Reporting format of audit cookbook to Chef Compliance API
• Reporting format of inspec exec in Chef Compliance (plain json-min)

The main issue is the additional information about profiles: https://github.com/chef-cookbooks/audit/blob/b281654e282639b6542d06435d820c921803239d/libraries/collector_classes.rb#L244

[username-is-already-taken2]

I hit this problem today with what we are trying to achieve with profile chaining so would like to subcribe to any updates regarding this

[chris-rock]

The current problem is located in our use with json-min. This format is optimize to minimize the amout of data transfered between client and server. Here is one example:

$ inspec exec examples/meta-profile examples/profile --format json-min | jq .
{
  "version": "1.7.0",
  "controls": [
    {
      "id": "ssh-01",
      "status": "passed",
      "code_desc": "File /etc/ssh/ssh_config should exist",
      "profile_id": "ssh-hardening"
    },
    ...
    {
      "id": "gordon-1.0",
      "status": "passed",
      "code_desc": "File  content should match nil",
      "profile_id": "profile"
    },
  ],
  "statistics": {
    "duration": 0.196187
  }
}

Problem:
If we execute more then one profile, we cannot distinguish between the source profiles.

This problem does not occur with the json format:

$ inspec exec examples/meta-profile examples/profile --format json | jq .
{
  "version": "1.7.0",
  "profiles": [
    {
      "name": "meta-profile",
      "title": "Meta Compliance Profile",
      "maintainer": "InSpec Authors",
      "copyright": "InSpec Authors",
      "copyright_email": "support@chef.io",
      "license": "Apache 2",
      "summary": "InSpec Profile that is only consuming dependencies",
      "version": "0.2.0",
      "depends": [
        {
          "name": "hardening/ssh-hardening"
        },
        {
          "git": "https://github.com/dev-sec/ssl-benchmark.git"
        },
        {
          "name": "windows-patch-benchmark",
          "git": "https://github.com/chris-rock/windows-patch-benchmark.git"
        }
      ],
      "supports": [],
      "controls": [
        {
          "title": "client: Check ssh_config owner, group and permissions.",
          "desc": "The ssh_config should owned by root, only be writable by owner and readable to all.",
          "impact": 1,
          "refs": [],
          "tags": {},
          "code": "",
          "source_location": {
            "ref": "tests-ssh-hardening-master/controls/ssh_spec.rb",
            "line": 25
          },
          "id": "ssh-01",
          "results": [
            {
              "status": "passed",
              "code_desc": "File /etc/ssh/ssh_config should exist",
              "run_time": 0.001083,
              "start_time": "2016-12-02 18:54:23 +0100"
            },
            ...
          ]
        },
      ],
      "groups": [
         ...
      ],
      "attributes": []
    },
    {
      "name": "profile",
      "title": "InSpec Example Profile",
      "maintainer": "Chef Software, Inc.",
      "copyright": "Chef Software, Inc.",
      "copyright_email": "support@chef.io",
      "license": "Apache 2 license",
      "summary": "Demonstrates the use of InSpec Compliance Profile",
      "version": "1.0.0",
      "supports": [
        {
          "os-family": "unix"
        }
      ],
      "controls": [
        {
          "title": "Create /tmp directory",
          "desc": "An optional description...",
          "impact": 0.7,
          "refs": [
            {
              "url": "http://...",
              "ref": "Document A-12"
            }
          ],
          "tags": {
            "data": "temp data",
            "security": null
          },
          "code": "control \"tmp-1.0\" do                        # A unique ID for this control\n  impact 0.7                                # The criticality, if this control fails.\n  title \"Create /tmp directory\"             # A human-readable title\n  desc \"An optional description...\"         # Describe why this is needed\n  tag data: \"temp data\"                     # A tag allows you to associate key information\n  tag \"security\"                            # to the test\n  ref \"Document A-12\", url: 'http://...'    # Additional references\n\n  describe file('/tmp') do                  # The actual test\n    it { should be_directory }\n  end\nend\n",
          "source_location": {
            "ref": "examples/profile/controls/example.rb",
            "line": 8
          },
          "id": "tmp-1.0",
          "results": [
            {
              "status": "passed",
              "code_desc": "File /tmp should be directory",
              "run_time": 0.0001,
              "start_time": "2016-12-02 18:54:24 +0100"
            }
          ]
        },
      ],
      "groups": [
         ...
      ],
      "attributes": []
    }
  ],
  "other_checks": [],
  "statistics": {
    "duration": 0.17154
  }
}

Possible solutions:

• we use UUIDs instead of "profile_id": "ssh-hardening". This requires a change in InSpec report generation and inspec json, keeps the output format identical, but may still break existing applications.
• generate json min from full json, to know all the details, we'll derive everything from full json. This has the advantage that it does not introduce a change and works with all reporter. It does not solve the root problem.
• use new json-min structure to support multiple profiles in output report as shown following. It has the disadvantage of introducing a breaking change.

$ inspec exec examples/meta-profile examples/profile --format json-min | jq .
{
  "version": "1.7.0",
  "profiles": [
    {
      "id": "lockfile_id of profile",
      "name": "meta-profile",
      "controls": [
        {
          "id": "ssh-01",
          "status": "passed",
          "code_desc": "File /etc/ssh/ssh_config should exist",
          "profile_id": "ssh-hardening"
        },
      ],
    },
    {
      "id": "lockfile_id of profile"
      "name": "profile",
      "controls": [
        {
          "id": "gordon-1.0",
          "status": "passed",
          "code_desc": "File  content should match nil",
          "profile_id": "profile"
        },
      ],
    }
  ],
  "other_checks": [],
  "statistics": {
    "duration": 0.17154
  }
}

[chris-rock]

As discussed with @arlimus we go forward with the intermediate solution:

generate json min from full json, to know all the details, we'll derive everything from full json.

[chris-rock]

This should be fixed with 3.1. Please re-open if this continues to be an issue.