Skip to content
This repository has been archived by the owner on Jan 9, 2023. It is now read-only.

Problem generating report for some rules #204

Closed
cipherboy opened this issue Jul 2, 2021 · 3 comments · Fixed by #216
Closed

Problem generating report for some rules #204

cipherboy opened this issue Jul 2, 2021 · 3 comments · Fixed by #216

Comments

@cipherboy
Copy link
Member

cipherboy commented Jul 2, 2021 •
edited
Loading

With OpenSCAP v1.2.16 from the Ubuntu distribution, on SSG-built content, we get something like the following:

$ arf-to-graph ~/ssg-ubuntu2004-ds-1-arf.xml file_permissions_audit_configuration 
Traceback (most recent call last):
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/xml_parser.py", line 45, in __init__
    self.scan_definitions = _XmlParserScanDefinitions(
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/_xml_parser_oval_scan_definitions.py", line 9, in __init__
    self.test_info_parser = _XmlParserTestInfo(report_data)
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/_xml_parser_test_info.py", line 20, in __init__
    self.system_data = self._get_system_data()
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/_xml_parser_test_info.py", line 39, in _get_system_data
    for item in data:
TypeError: 'NoneType' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/command_line.py", line 30, in arf_to_graph
    main(ArfToHtml(args))
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/command_line_client/arf_to_html.py", line 8, in __init__
    super().__init__(args)
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/command_line_client/client_arf_input.py", line 10, in __init__
    self.arf_xml_parser = XmlParser(self.source_filename)
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/xml_parser.py", line 48, in __init__
    raise ValueError(
ValueError: This file "/home/cipherboy/ssg-ubuntu2004-ds-1-arf.xml" is not arf report file or there are no results

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cipherboy/.local/bin/arf-to-graph", line 33, in <module>
    sys.exit(load_entry_point('oval-graph', 'console_scripts', 'arf-to-graph')())
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/command_line.py", line 32, in arf_to_graph
    print_detail_traceback_if_verbose(args)
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/command_line.py", line 24, in print_detail_traceback_if_verbose
    if any(arg in args for arg in ("-v", "--verbose")):
  File "/home/cipherboy/GitHub/OpenSCAP/oval-graph/oval_graph/command_line.py", line 24, in <genexpr>
    if any(arg in args for arg in ("-v", "--verbose")):
TypeError: argument of type 'NoneType' is not iterable

In _XmlParserTestInfo I see:

    def _get_system_data(self):
        data = self.report_data.find(
            ('.//XMLSchema:oval_results/XMLSchema:results/'
             'XMLSchema:system/oval-characteristics:oval_system_characteristics'
             '/oval-characteristics:system_data'), ns)
        out = {}
        for item in data:
            out[item.attrib.get('id')] = item
        return out

If I change it as so:

    def _get_system_data(self):
        data = self.report_data.find(
            ('.//XMLSchema:oval_results/XMLSchema:results/'
             'XMLSchema:system/oval-characteristics:oval_system_characteristics'
             '/oval-characteristics:system_data'), ns)
        out = {}
        if not data:
            return out
        for item in data:
            out[item.attrib.get('id')] = item
        return out

it works, but I'm wondering if this is the right result. Perhaps the element tree is corrupted/modified during loading?

Oddly, if I print the children of oval-characteristics:oval_system_characteristics I see:

        x = self.report_data.find(
            ('.//XMLSchema:oval_results/XMLSchema:results/'
             'XMLSchema:system/oval-characteristics:oval_system_characteristics'), ns)
        print(x)
        for element in x.findall(".//"):
            print("\t", element)
        print(ElementTree.tostring(x, encoding='utf8'))
<Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}oval_system_characteristics at 0x7fb36088cfc0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}generator at 0x7fb36088ce80>
	 <Element {http://oval.mitre.org/XMLSchema/oval-common-5}product_name at 0x7fb36088cdc0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-common-5}product_version at 0x7fb36088ce40>
	 <Element {http://oval.mitre.org/XMLSchema/oval-common-5}schema_version at 0x7fb36088cf00>
	 <Element {http://oval.mitre.org/XMLSchema/oval-common-5}timestamp at 0x7fb36088cf80>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}system_info at 0x7fb360891040>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}os_name at 0x7fb360891080>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}os_version at 0x7fb3608910c0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}architecture at 0x7fb360891100>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}primary_host_name at 0x7fb360891140>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interfaces at 0x7fb36088cf40>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface at 0x7fb360891180>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface_name at 0x7fb3608911c0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}ip_address at 0x7fb360891200>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}mac_address at 0x7fb360891240>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface at 0x7fb360891280>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface_name at 0x7fb3608912c0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}ip_address at 0x7fb360891300>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}mac_address at 0x7fb360891340>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface at 0x7fb360891380>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface_name at 0x7fb3608913c0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}ip_address at 0x7fb360891400>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}mac_address at 0x7fb360891440>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface at 0x7fb360891480>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}interface_name at 0x7fb3608914c0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}ip_address at 0x7fb360891500>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}mac_address at 0x7fb360891540>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}collected_objects at 0x7fb360891580>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}object at 0x7fb36088cac0>
	 <Element {http://oval.mitre.org/XMLSchema/oval-system-characteristics-5}object at 0x7fb36088ca00>
Rule "xccdf_org.ssgproject.content_rule_file_permissions_audit_configuration" done!
Result is saved:"/tmp/graph-of-xccdf_org.ssgproject.content_rule_file_permissions_audit_configuration-02_07_2021-08_36_51.html"

And it just stops parsing the objects and stuff under it collected_objects before getting to system_data!

Snippet of XML below:

Subset of ARF file 
...

  <arf:reports>

...

    <arf:report id="oval1">
      <arf:content>
        <oval_results xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:oval="http://oval.mitre.org/XMLSchema/oval-common-5" xmlns="http://oval.mitre.org/XMLSchema/oval-results-5" xsi:schemaLocation="http://oval.mitre.org/XMLSchema/oval-results-5 oval-results-schema.xsd http://oval.mitre.org/XMLSchema/oval-common-5 oval-common-schema.xsd">

...

          <results>
            <system>

...

              <oval_system_characteristics xmlns:oval="http://oval.mitre.org/XMLSchema/oval-common-5" xmlns:unix-sys="http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#unix" xmlns:ind-sys="http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#independent" xmlns:lin-sys="http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#linux" xmlns:win-sys="http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#windows" xmlns="http://oval.mitre.org/XMLSchema/oval-system-characteristics-5" xsi:schemaLocation="http://oval.mitre.org/XMLSchema/oval-system-characteristics-5 oval-system-characteristics-schema.xsd http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#independent independent-system-characteristics-schema.xsd http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#unix unix-system-characteristics-schema.xsd http://oval.mitre.org/XMLSchema/oval-system-characteristics-5#linux linux-system-characteristics-schema.xsd http://oval.mitre.org/XMLSchema/oval-common-5 oval-common-schema.xsd">
                <generator>
                  <oval:product_name>combine_ovals.py from SCAP Security Guide</oval:product_name>
                  <oval:product_version>ssg: [0, 1, 57], python: 3.9.5</oval:product_version>
                  <oval:schema_version>5.11</oval:schema_version>
                  <oval:timestamp>2021-07-02T12:20:01</oval:timestamp>
                </generator>
                <system_info>
                  <os_name>Linux</os_name>
                  <os_version>#82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021</os_version>
                  <architecture>x86_64</architecture>
                  <primary_host_name>ubuntu-2004-base</primary_host_name>
                  <interfaces>
                    <interface>
                      <interface_name>lo</interface_name>
                      <ip_address>127.0.0.1</ip_address>
                      <mac_address>00:00:00:00:00:00</mac_address>
                    </interface>
                    <interface>
                      <interface_name>enp1s0</interface_name>
                      <ip_address>192.168.122.145</ip_address>
                      <mac_address>52:54:00:95:12:ED</mac_address>
                    </interface>
                    <interface>
                      <interface_name>lo</interface_name>
                      <ip_address>::1</ip_address>
                      <mac_address>00:00:00:00:00:00</mac_address>
                    </interface>
                    <interface>
                      <interface_name>enp1s0</interface_name>
                      <ip_address>fe80::5054:ff:fe95:12ed</ip_address>
                      <mac_address>52:54:00:95:12:ED</mac_address>
                    </interface>
                  </interfaces>
                </system_info>
                <collected_objects>
                  <object id="oval:ssg-obj_lsb:obj:1" version="1" flag="complete">
                    <reference item_ref="1345581"/>
                  </object>
                  <object id="oval:ssg-obj_ubuntu:obj:1" version="1" flag="complete">
                    <reference item_ref="1345631"/>
                  </object>
                  <object id="oval:ssg-obj_ubuntu_focal:obj:1" version="1" flag="complete">
                    <reference item_ref="1345632"/>
                  </object>
                  <object id="oval:ssg-object_installed_env_is_a_docker_container:obj:1" version="1" flag="does not exist"/>
                  <object id="oval:ssg-object_installed_env_is_a_podman_container:obj:1" version="1" flag="does not exist"/>
                  <object id="oval:ssg-object_unix_family:obj:1" version="1" flag="complete">
                    <reference item_ref="1345531"/>
                  </object>
                </collected_objects>
                <system_data>
                  <ind-sys:textfilecontent_item id="1345632" status="exists">
                    <ind-sys:filepath>/etc/lsb-release</ind-sys:filepath>

 ...

            </system>
          </results>

XMLLint runs fine on the output (and doesn't detect any dropped closing tags or anything), so I'm really confused.

The report generates fine once the if not data is added though:

Screenshot from 2021-07-02 08-27-25
@Honny1
Copy link
Member

Honny1 commented Jul 7, 2021

Hello @cipherboy,
I was unable to replicate the issue. Can you please provide an ARF report? I hope with the report, I can find out more about why part of the XML file is not processed.

@cipherboy
Copy link
Member Author

cipherboy commented Jul 8, 2021 •
edited
Loading

@Honny1 Do you have any way of minimizing the ARF? I've pasted part of it above (see "Subset of ARF file"), but the file is still several hundred KB when gziped and is from our private CaC branch.

I'm wondering if it is perhaps a parsing issue caused by the older OpenSCAP?

@Honny1
Copy link
Member

Honny1 commented Jul 8, 2021 •
edited
Loading

@cipherboy I don't have any way of minimizing the ARF. You can put here the ARF file as a TXT file. Like in issue. Hundreds of KB are fine for Github.

I tried to generate graphs from ARF generated from content for Ubuntu 20.04 with OpenSCAP 1.2.7 and 1.2.6 and everything was fine. It is very strange that the system_data could not be found, even though they are in the report.

@Honny1 Honny1 mentioned this issue Nov 16, 2021
Merged
@Honny1 Honny1 mentioned this issue Nov 25, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix issue with missing system data in the arf report Honny1/oval-graph
2 participants
@cipherboy @Honny1