Add Merge function type and sort enumeration #177

vanderpol · 2024-10-10T17:02:27Z

Adding a merge function which joins together multiple elements into a single variable, with an optional delimiter, and optional sorting. Had to add a new sort enumeration which may need to be discussed.

solind · 2024-10-10T19:20:42Z

oval-schemas/oval-definitions-schema.xsd

@@ -938,6 +939,16 @@
            <xsd:group ref="oval-def:ComponentGroup"/>
        </xsd:sequence>
    </xsd:complexType>
+    <xsd:complexType name="MergeFunctionType">
+        <xsd:annotation>
+            <xsd:documentation>The merge function takes one or more components and merges them together into a single string, with an optional delimiter.  For example if data from one registry multi-size value contains values of "abc" and "def".  The merge function will resolve to a local_variable with the value of 'abcdef'.  If an optional delimiter of ',' was used, the merge function would resolve to a local_variable with the value of 'abc,'def'</xsd:documentation>


Edited slightly:

The merge function takes one or more components and merges them together into a single string value, optionally including a delimiter string. For example, if data from one registry multi-size value contains values of "abc" and "def", the merge function operated on those values would resolve to a string with the value of 'abcdef'. If an optional delimiter of ',' was used, the merge function would resolve to a value of 'abc,'def'.

solind · 2024-10-10T19:22:44Z

oval-schemas/oval-definitions-schema.xsd

+            <xsd:documentation>The SortEnumeration simple type defines basic sorting operations.  Currently 'none', 'alphabetical' and 'alphanumerical' are defined..</xsd:documentation>
+        </xsd:annotation>
+        <xsd:restriction base="xsd:string">
+            <xsd:enumeration value="none">


I wouldn't call this "none", I would call this "document_order". After all, two interpreters using the same system-characteristics file as input should both generate identical results.

I wasn't entirely sure what your "document_order" meant, as long as we explain it as the order of the items/values in OVAL system characteristics, I would agree.

Document order means... this is an XML document. The order is the order in which the associated elements appear in the document.

In my brain, this all happens in memory structures, long before we end up writing it to XML, so that was my disconnect.

solind · 2024-10-10T19:30:35Z

oval-schemas/oval-definitions-schema.xsd

+					<xsd:documentation>No sorting is performed, the values will remain in the order they came in as.</xsd:documentation>
+                </xsd:annotation>
+			</xsd:enumeration>	
+            <xsd:enumeration value="alphabetical">


I'd call this "lexical", which I believe is more standard terminology, and typically stands in contrast with "natural" ordering.

For example, given the strings {"a1, "a2", "a12", "1", "2", "12"}, in ascending order*, lexical sorting yields:

"1", "12", "2", "a1", "a2", "a12"

And natural sorting yields:

"1", "2", "12", "a1", "a12", "a2"

this reminds me, there is a difference between the sort type and the order (ascending vs. descending)

I'm fine with that, will update

solind · 2024-10-10T19:31:04Z

oval-schemas/oval-definitions-schema.xsd

+                    <xsd:documentation>Sort alphabetical, useful for pure string lists</xsd:documentation>
+                </xsd:annotation>
+			</xsd:enumeration>
+			<xsd:enumeration value="alphanumerical">


How is this different from "natural"?

solind

I think maybe you need a SortEnumeration (lexical, natural, numeric) and an OrderEnumeration (ascending/descending)

Where numeric would generate an error if it encounters a string value that cannot be converted to a base-10 number (float).

vanderpol · 2024-10-10T19:38:22Z

I think maybe you need a SortEnumeration (lexical, natural, numeric) and an OrderEnumeration (ascending/descending)

Where numeric would generate an error if it encounters a string value that cannot be converted to a base-10 number (float).

I wasn't sure about a numeric sort, as accidental string data could cause all kinds of errors, but an alphanumeric (or natural), would handle them both. I'm not 100% certain on the need for alphanumeric vs natural, likely just natural. I can easily add, and content authors beware.

Agree on OrderEnumeration, will add.

solind · 2024-10-10T19:54:06Z

As for numerical vs. natural, it's just a thought. You could effectively achieve the same thing using a variable datatype, like you did. 😀

vanderpol · 2024-10-11T11:04:19Z

So for the OrderEnumeration, I'm pondering the interaction between that and "document" sort? I guess we can just document that the OrderEnumeration has no influence on the 'document' sort, as 'document' sort, isn't sorting at all? Default OrderEnumeration should be set to "ascending".

vanderpol · 2024-10-11T12:47:40Z

Here's some updated sample results, I've added a recursive Registry test, a numeric sort test using variables, and I threw in a WMI57 test to show it works with records.
SCC-5.10.1_Beta1_2024-10-11_084515_OVAL-Results_merge_test_content.xml.txt

solind · 2024-10-11T15:13:02Z

So for the OrderEnumeration, I'm pondering the interaction between that and "document" sort? I guess we can just document that the OrderEnumeration has no influence on the 'document' sort, as 'document' sort, isn't sorting at all? Default OrderEnumeration should be set to "ascending".

OrderEnumeration values would be "ascending" and "descending". I guess in the case of a "document_order" sort value, this would be ignored.

ETA: which I guess is what you're proposing, so we agree! Also yes to the default.

solind · 2024-10-11T15:26:59Z

Here's some updated sample results, I've added a recursive Registry test, a numeric sort test using variables, and I threw in a WMI57 test to show it works with records. SCC-5.10.1_Beta1_2024-10-11_084515_OVAL-Results_merge_test_content.xml.txt

Naturally, a little digression here... I noticed this item in your results:

<win-sc:registry_item id="10" status="exists">
  <win-sc:hive>HKEY_LOCAL_MACHINE</win-sc:hive>
  <win-sc:key>SYSTEM\CurrentControlSet\Control\Session Manager</win-sc:key>
  <win-sc:name>ExcludeFromKnownDlls</win-sc:name>
  <win-sc:last_write_time datatype="int">133730601810000000</win-sc:last_write_time>
  <win-sc:type>reg_multi_sz</win-sc:type>
  <win-sc:value></win-sc:value>
  <win-sc:value></win-sc:value>
  <win-sc:windows_view>64_bit</win-sc:windows_view>
</win-sc:registry_item>

I see two empty values, which in XML parlance means strings of zero length. I assume this happened because a REG_SZ is terminated by a pair of NULL (0x00) bytes. IIRC, Joval would interpret this as a non-valued key, and represent that by creating a single item value like this:

  <win-sc:value status="does not exist"/>

This is important, because empty strings are still string values that exist.

vanderpol · 2024-10-11T15:34:02Z

Very astute observation in the results, and given the lack of multi-sz content in the wild we may have overlooked this. Below is an export from my registry, so just to confirm you would say that this should be a DNE on the value for ExcludeFromKnownDlls?

In looking back in our code revision history, a former developer decided at least 12 years ago that we should intentionally mark this scenario as exists. Not saying it's 'right', but it was intentionally done, if misguided. Shows how frequently multi-sz keys are used in content more than anything I'd guess.

Windows Registry Editor Version 5.00;

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager]
"AutoChkTimeout"=dword:00000008
"BootExecute"=hex(7):61,00,75,00,74,00,6f,00,63,00,68,00,65,00,63,00,6b,00,20,
00,61,00,75,00,74,00,6f,00,63,00,68,00,6b,00,20,00,2a,00,00,00,00,00
"BootShell"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,62,
00,6f,00,6f,00,74,00,69,00,6d,00,2e,00,65,00,78,00,65,00,00,00
"CriticalSectionTimeout"=dword:00278d00
"ExcludeFromKnownDlls"=hex(7):00,00

solind · 2024-10-11T18:41:27Z

so just to confirm you would say that this should be a DNE on the value for ExcludeFromKnownDlls?

Yes, I would. I believe we even had a discussion about this on the old OVAL mailing list.

vanderpol · 2024-10-11T19:36:27Z

@solind any other thoughts on the PR? I feel like (after a lot of improvements based on your feedback, which I greatly appreciate) that we are pretty much good to go on this?

solind · 2024-10-11T21:31:08Z

oval-schemas/oval-definitions-schema.xsd

@@ -938,6 +939,17 @@
            <xsd:group ref="oval-def:ComponentGroup"/>
        </xsd:sequence>
    </xsd:complexType>
+    <xsd:complexType name="MergeFunctionType">
+        <xsd:annotation>
+            <xsd:documentation>The merge function takes one or more components and merges them together into a single string value, optionally including a delimiter string. For example, if data from one registry multi-size value contains values of "abc" and "def", the merge function operated on those values would resolve to a string with the value of 'abcdef'. If an optional delimiter of ',' was used, the merge function would resolve to a value of 'abc,'def'.</xsd:documentation>


REG_MULTI_SZ doesn't actually mean "multi-size", it means multiple null-terminated strings.

I swear I know what I'm doing from time to time... probably would have been faster to have you write this spec from the start.

solind · 2024-10-11T21:36:16Z

oval-schemas/oval-definitions-schema.xsd

+			</xsd:enumeration>
+			<xsd:enumeration value="natural">
+			    <xsd:annotation>
+                    <xsd:documentation>Sort like PHP's natsort https://www.php.net/manual/en/function.natsort.php</xsd:documentation>


Natural sort order is a pretty established computing concept, with implementations in most programming languages and on most operating systems. So, I think this might be a better external URL to reference: https://en.wikipedia.org/wiki/Natural_sort_order

Sounds good.

solind

My only comments are documentation-related. LMK if you agree with them.

vanderpol · 2024-10-11T23:13:34Z

Fixed documentation per your suggestions. Hoping this is now done.

solind

Looks good to me!

vanderpol added 3 commits September 20, 2024 13:59

Added MergeFunctionType to resolve OVAL-Community#135

188496c

Added "merge" to function group, fixed quote typo in documentation

088b989

Updated documentation for merge function and added sortEnumeration

c108f00

vanderpol requested a review from solind October 10, 2024 17:02

solind reviewed Oct 10, 2024

View reviewed changes

solind requested changes Oct 10, 2024

View reviewed changes

Added OrderEnumeration and updated SortEnumeration based on feedback

a2e7ccb

vanderpol requested a review from solind October 11, 2024 12:49

solind reviewed Oct 11, 2024

View reviewed changes

Minor fixes to documentation

45157d6

vanderpol requested a review from solind October 11, 2024 23:12

solind approved these changes Oct 12, 2024

View reviewed changes

vanderpol merged commit 83ef99f into OVAL-Community:develop Oct 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Merge function type and sort enumeration #177

Add Merge function type and sort enumeration #177

vanderpol commented Oct 10, 2024

solind Oct 10, 2024

solind Oct 10, 2024

vanderpol Oct 10, 2024

solind Oct 10, 2024

vanderpol Oct 11, 2024

solind Oct 10, 2024

vanderpol Oct 10, 2024

solind Oct 10, 2024

solind left a comment

vanderpol commented Oct 10, 2024

solind commented Oct 10, 2024

vanderpol commented Oct 11, 2024

vanderpol commented Oct 11, 2024

solind commented Oct 11, 2024 •

edited

Loading

solind commented Oct 11, 2024

vanderpol commented Oct 11, 2024 •

edited

Loading

solind commented Oct 11, 2024

vanderpol commented Oct 11, 2024

solind Oct 11, 2024

vanderpol Oct 11, 2024

solind Oct 11, 2024 •

edited

Loading

vanderpol Oct 11, 2024

solind left a comment

vanderpol commented Oct 11, 2024

solind left a comment

Add Merge function type and sort enumeration #177

Add Merge function type and sort enumeration #177

Conversation

vanderpol commented Oct 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

solind left a comment

Choose a reason for hiding this comment

vanderpol commented Oct 10, 2024

solind commented Oct 10, 2024

vanderpol commented Oct 11, 2024

vanderpol commented Oct 11, 2024

solind commented Oct 11, 2024 • edited Loading

solind commented Oct 11, 2024

vanderpol commented Oct 11, 2024 • edited Loading

solind commented Oct 11, 2024

vanderpol commented Oct 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

solind Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

solind left a comment

Choose a reason for hiding this comment

vanderpol commented Oct 11, 2024

solind left a comment

Choose a reason for hiding this comment

solind commented Oct 11, 2024 •

edited

Loading

vanderpol commented Oct 11, 2024 •

edited

Loading

solind Oct 11, 2024 •

edited

Loading