Merge pull request #16 from sarnold/rename-cleanup

Rename/oscal cleanup

README.rst

@@ -1,49 +1,66 @@
-================================
- ymltoxml (and more YAML tools)
-================================
+=======================================
+ yaml-tools (and more for CSV and XML)
+=======================================
 
 |ci| |wheels| |release| |badge| |coverage|
 
 |pre| |cov| |pylint|
 
 |tag| |license| |python|
 
-Python command line tools to convert files between XML_ and YAML_,
-preserving attributes and comments (with minor corrections).  The default
-file encoding for both types is UTF-8 without a BOM. Now includes more
+Python command line tools for working with YAML and similar structured
+text data, eg, round-trip conversion between XML_ and YAML_, preserving
+attributes and comments (with minor corrections).  The default file
+encoding for all types is UTF-8 without a BOM. Now includes more
 console entry points to grep or sort interesting YAML files (eg, lists
-of rules found in the `SCAP Security Guide`_) and support for more input
-file types to ingest SSG and other upstream data, eg, NIST oscal-content_.
+of rules found in the `SCAP Security Guide`_) and support for more
+input file types to ingest SSG and other upstream data, eg, NIST
+oscal-content_.
 
 .. _SCAP Security Guide: https://github.com/ComplianceAsCode/content
 .. _oscal-content: https://github.com/usnistgov/oscal-content.git
 
 Quick Start
 ===========
 
-Available console commands and scripts:
+Available modules, console commands, and scripts:
 
 * ``ymltoxml`` - YAML / XML round-trip conversion and cleanup
 * ``yasort`` - sort large lists in YAML files
 * ``yagrep`` - grep for keys/values in YAML files
 * ``oscal`` (*WIP*) - ingest NIST 800-53 content in multiple formats
-* ``analyze_control_ids.py`` (*experimental*) - analyze control ID sets
+
+Experimental "demo" scripts:
+* ``analyze_control_ids.py`` - analyze control ID sets with fuzzy match
+* ``analyze_ssg_controls.py`` - analyze NIST controls from SSG content
+
+For the above "demo" scripts, check the top of the source file for any knobs
+adjustable via environment variables, eg:
+
+.. code-block:: python
+
+  FILE = os.getenv(
+      "ID_FILE",
+      default="tests/data/OE-expanded-profile-all-ids.txt",
+  )
+  SSG_PATH = os.getenv("SSG_PATH", default="ext/content/controls")
+  DEBUG = int(os.getenv("DEBUG", default=0))
 
 
 Install with pip
 ----------------
 
 This package is *not* yet published on PyPI, thus use one of the following
-to install ymltoxml on any platform. Install from the main branch::
+to install yaml-tools on any platform. Install from the main branch::
 
-  $ https://github.com/sarnold/ymltoxml/archive/refs/heads/main.tar.gz
+  $ https://github.com/sarnold/yaml-tools/archive/refs/heads/main.tar.gz
 
 or use this command to install a specific release version::
 
-  $ pip install https://github.com/sarnold/ymltoxml/releases/download/0.3.0/ymltoxml-0.3.0.tar.gz
+  $ pip install https://github.com/sarnold/yaml-tools/releases/download/0.4.0/yaml_tools-0.4.0.tar.gz
 
-The full package provides the ``ymltoxml.py`` executable as well as
-a reference configuration file with defaults for all values.
+The ``yaml_tools`` package provides the modules shown above as well as
+module-specific reference configuration files with defaults for all values.
 
 If you'd rather work from the source repository, it supports the common
 idiom to install it on your system in a virtual env after cloning::
@@ -116,7 +133,7 @@ configuration file, do::
 yagrep
 ------
 
-A new helper script is now included for searching keys or values in
+A new module is now included for searching keys and values in
 YAML files. The ``yagrep`` script also has its own built-in config
 file, which can be copied and edited as shown above. In this case the
 script is intended to feel more-or-less like ``grep`` so the default
@@ -219,10 +236,10 @@ Default yasort.yaml:
 Features and limitations
 ------------------------
 
-We mainly test on mavlink XML message definitions and NIST/SSG YAML files,
-so round-trip conversion *may not* work at all on arbitrarily complex XML
-files with namespaces, etc.  The current round-trip is not exact, due to
-the following:
+We mainly test ymltoxml on mavlink XML message definitions and NIST/SSG
+YAML files, so round-trip conversion *may not* work at all on
+arbitrarily complex XML files with namespaces, etc.  The current
+round-trip is not exact, due to the following:
 
 * missing encoding is added to version tag
 * leading/trailing whitespace in text elements and comments is not preserved
@@ -302,8 +319,8 @@ all the dependencies and run the specified commands, eg:
 
 ::
 
-  $ git clone https://github.com/sarnold/ymltoxml
-  $ cd ymltoxml/
+  $ git clone https://github.com/sarnold/yaml-tools.git
+  $ cd yaml-tools/
   $ tox -e py
 
 The above will run the tests using your (default) system Python;
@@ -345,9 +362,11 @@ file and reproduced below::
   new: test: added a bunch of test around user usability of feature X.
   fix: typo in spelling my name in comment. !minor
 
-See the following docs page (or generate-changelog_ on Github) for more
+See the following docs page (or generate-changelog.rst_ on Github) for more
 details.
 
+.. _generate-changelog.rst: https://github.com/sarnold/yaml-tools/blob/main/docs/source/dev/generate-changelog.rst
+
 This repo is also pre-commit_ enabled for various linting and format
 checks.  The checks run automatically on commit and will fail the
 commit (if not clean) with some checks performing simple file corrections.
@@ -380,8 +399,8 @@ something like::
 
 then install it into the repo you just cloned::
 
-  $ git clone https://github.com/sarnold/ymltoxml
-  $ cd ymltoxml/
+  $ git clone https://github.com/sarnold/yaml_tools
+  $ cd yaml_tools/
   $ pre-commit install
 
 It's usually a good idea to update the hooks to the latest version::
@@ -393,40 +412,40 @@ It's usually a good idea to update the hooks to the latest version::
 .. _pre-commit: http://pre-commit.com/
 
 
-.. |ci| image:: https://github.com/sarnold/ymltoxml/actions/workflows/ci.yml/badge.svg
-    :target: https://github.com/sarnold/ymltoxml/actions/workflows/ci.yml
+.. |ci| image:: https://github.com/sarnold/yaml-tools/actions/workflows/ci.yml/badge.svg
+    :target: https://github.com/sarnold/yaml-tools/actions/workflows/ci.yml
     :alt: CI Status
 
-.. |wheels| image:: https://github.com/sarnold/ymltoxml/actions/workflows/wheels.yml/badge.svg
-    :target: https://github.com/sarnold/ymltoxml/actions/workflows/wheels.yml
+.. |wheels| image:: https://github.com/sarnold/yaml-tools/actions/workflows/wheels.yml/badge.svg
+    :target: https://github.com/sarnold/yaml-tools/actions/workflows/wheels.yml
     :alt: Wheel Status
 
-.. |coverage| image:: https://github.com/sarnold/ymltoxml/actions/workflows/coverage.yml/badge.svg
-    :target: https://github.com/sarnold/ymltoxml/actions/workflows/coverage.yml
+.. |coverage| image:: https://github.com/sarnold/yaml-tools/actions/workflows/coverage.yml/badge.svg
+    :target: https://github.com/sarnold/yaml-tools/actions/workflows/coverage.yml
     :alt: Coverage workflow
 
-.. |badge| image:: https://github.com/sarnold/ymltoxml/actions/workflows/pylint.yml/badge.svg
-    :target: https://github.com/sarnold/ymltoxml/actions/workflows/pylint.yml
+.. |badge| image:: https://github.com/sarnold/yaml-tools/actions/workflows/pylint.yml/badge.svg
+    :target: https://github.com/sarnold/yaml-tools/actions/workflows/pylint.yml
     :alt: Pylint Status
 
-.. |release| image:: https://github.com/sarnold/ymltoxml/actions/workflows/release.yml/badge.svg
-    :target: https://github.com/sarnold/ymltoxml/actions/workflows/release.yml
+.. |release| image:: https://github.com/sarnold/yaml-tools/actions/workflows/release.yml/badge.svg
+    :target: https://github.com/sarnold/yaml-tools/actions/workflows/release.yml
     :alt: Release Status
 
-.. |cov| image:: https://raw.githubusercontent.com/sarnold/ymltoxml/badges/main/test-coverage.svg
-    :target: https://github.com/sarnold/ymltoxml/
+.. |cov| image:: https://raw.githubusercontent.com/sarnold/yaml-tools/badges/main/test-coverage.svg
+    :target: https://github.com/sarnold/yaml-tools/
     :alt: Test coverage
 
-.. |pylint| image:: https://raw.githubusercontent.com/sarnold/ymltoxml/badges/main/pylint-score.svg
-    :target: https://github.com/sarnold/ymltoxml/actions/workflows/pylint.yml
+.. |pylint| image:: https://raw.githubusercontent.com/sarnold/yaml-tools/badges/main/pylint-score.svg
+    :target: https://github.com/sarnold/yaml-tools/actions/workflows/pylint.yml
     :alt: Pylint score
 
-.. |license| image:: https://img.shields.io/github/license/sarnold/ymltoxml
-    :target: https://github.com/sarnold/ymltoxml/blob/master/LICENSE
+.. |license| image:: https://img.shields.io/github/license/sarnold/yaml-tools
+    :target: https://github.com/sarnold/yaml-tools/blob/master/LICENSE
     :alt: License
 
-.. |tag| image:: https://img.shields.io/github/v/tag/sarnold/ymltoxml?color=green&include_prereleases&label=latest%20release
-    :target: https://github.com/sarnold/ymltoxml/releases
+.. |tag| image:: https://img.shields.io/github/v/tag/sarnold/yaml-tools?color=green&include_prereleases&label=latest%20release
+    :target: https://github.com/sarnold/yaml-tools/releases
     :alt: GitHub tag
 
 .. |python| image:: https://img.shields.io/badge/python-3.8+-blue.svg

docs/source/conf.py

@@ -29,12 +29,12 @@
 
 # -- Project information -----------------------------------------------------
 
-project = 'ymltoxml'
-copyright = '2023, Stephen L Arnold'
+project = 'yaml_tools'
+copyright = '2024, Stephen L Arnold'
 author = 'Stephen Arnold'
 
 # The full version, including alpha/beta/rc tags
-release = version('ymltoxml')
+release = version('yaml_tools')
 # The short X.Y version.
 version = '.'.join(release.split('.')[:2])
 
@@ -58,7 +58,7 @@
     'recommonmark',
 ]
 
-apidoc_module_dir = '../../src/ymltoxml/'
+apidoc_module_dir = '../../src/yaml_tools/'
 apidoc_output_dir = 'api'
 apidoc_excluded_paths = ['test', 'scripts']
 apidoc_separate_modules = True
@@ -79,7 +79,7 @@
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
-description = 'Console tool for bidirectional transformation of YAML and XML files.'
+description = 'Console tools for working with YAML and other structured content.'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -134,7 +134,7 @@
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'ymltoxmldoc'
+htmlhelp_basename = 'yaml_toolsdoc'
 
 
 # -- Options for LaTeX output ---------------------------------------------
@@ -161,7 +161,7 @@
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'ymltoxml.tex', 'ymltoxml Documentation',
+    (master_doc, 'yaml_tools.tex', 'yaml_tools Documentation',
      [author], 'manual'),
 ]
 
@@ -171,7 +171,7 @@
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
 man_pages = [
-    (master_doc, 'ymltoxml', 'ymltoxml Documentation',
+    (master_doc, 'yaml_tools', 'yaml_tools Documentation',
      [author], 1)
 ]
 
@@ -182,8 +182,8 @@
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'ymltoxml', 'ymltoxml Documentation',
-     [author], 'ymltoxml', description,
+    (master_doc, 'yaml_tools', 'yaml_tools Documentation',
+     [author], 'yaml_tools', description,
      'Miscellaneous'),
 ]
 

docs/source/index.rst

@@ -1,5 +1,5 @@
-Welcome to the ymltoxml documentation!
-======================================
+Welcome to the yaml-tools documentation!
+========================================
 
 .. git_commit_detail::
     :branch:

scripts/analyze_control_ids.py

@@ -10,7 +10,11 @@
 
 from fuzzy_match import match as fmatch
 
-from ymltoxml.utils import get_filelist, text_file_reader
+from yaml_tools.utils import (
+    get_filelist,
+    get_profile_type,
+    text_file_reader,
+)
 
 id_count: typing.Counter[str] = Counter()
 result_queue = deque()
@@ -39,19 +43,6 @@ def get_profile_sets(dirpath='tests/data', filepattern='*.txt', debug=False):
     :return: tuple of lists: (profile_sets, PROFILE_NAMES)
     """
 
-    def get_profile_type(filename, debug=False):
-        """
-        Get oscal profile type from filename, where profile type is one of the
-        exported profile names, ie, HIGH, MODERATE, LOW, or PRIVACY.
-        """
-        match = None
-
-        if any((match := substring) in filename for substring in PROFILE_NAMES):
-            if debug:
-                print(f'Found profile type: {match}')
-
-        return match
-
     h_set = set()
     m_set = set()
     l_set = set()
@@ -128,7 +119,9 @@ def get_profile_type(filename, debug=False):
         if len(match_list) > 0:
             result_queue.append((ctl_id, match_list))
 
-    print(f"\nFuzzy ID match shows {len(result_queue)} possible controls match HIGH set")
+    print(
+        f"\nFuzzy ID match shows {len(result_queue)} possible controls match HIGH set"
+    )
 
     if DEBUG:
         for match_res in result_queue:

scripts/analyze_ssg_controls.py

@@ -10,21 +10,22 @@
 from pathlib import Path
 
 from diskcache import Deque
-
 from nested_lookup import nested_lookup
-from ymltoxml.templates import xform_id
-from ymltoxml.utils import FileTypeError, get_filelist, text_file_reader
 
-id_count: typing.Counter[str] = Counter()
-id_queue = Deque(get_cachedir(dir_name='id_queue'))
-ctl_queue = Deque(get_cachedir(dir_name='ctl_queue'))
+from yaml_tools.templates import xform_id
+from yaml_tools.utils import (
+    FileTypeError,
+    get_filelist,
+    text_file_reader,
+)
 
 FILE = os.getenv('ID_FILE', default='tests/data/OE-expanded-profile-all-ids.txt')
 SSG_PATH = os.getenv('SSG_PATH', default='ext/content/controls')
 DEBUG = int(os.getenv('DEBUG', default=0))
 
 OPTIONS = {'file_encoding': 'utf-8'}
 FILE_GLOB = 'nist_*.yml'
+CONTROL_SETS = []
 CONTROL_FILES = [
     'nist_ocp4.yml',
     'nist_rhidm.yml',
@@ -54,6 +55,10 @@ def set_unique(sequence):
     print(f'Input file {FILE} not found!')
     sys.exit(1)
 
+
+id_count: typing.Counter[str] = Counter()
+id_queue = Deque(get_cachedir(dir_name='id_queue'))
+ctl_queue = Deque(get_cachedir(dir_name='ctl_queue'))
 input_ids = Path(FILE).read_text(encoding='utf-8').splitlines()
 
 if input_ids[0].islower():
@@ -103,6 +108,7 @@ def set_unique(sequence):
     pname, id_list = id_queue.popleft()
     print(f"\n{pname} control IDs -> {len(id_list)}")
     id_set = set(id_list)
+    CONTROL_SETS.append((pname, id_set))
 
     print(f"Input set is in {pname} set: {id_set > in_set}")
     common_set = sorted(id_set & in_set)
@@ -111,3 +117,14 @@ def set_unique(sequence):
     print(f"Num input controls not in {pname} set -> {len(not_in_set)}")
     if DEBUG:
         print(f"Input controls not in {pname} set: {not_in_set}")
+
+if DEBUG:
+    print(
+        f"\n{CONTROL_SETS[0][0]} == {CONTROL_SETS[1][0]} {CONTROL_SETS[0][1] == CONTROL_SETS[1][1]}"
+    )
+    print(
+        f"{CONTROL_SETS[0][0]} == {CONTROL_SETS[2][0]} {CONTROL_SETS[0][1] == CONTROL_SETS[2][1]}"
+    )
+    print(
+        f"{CONTROL_SETS[0][0]} == {CONTROL_SETS[3][0]} {CONTROL_SETS[0][1] == CONTROL_SETS[3][1]}"
+    )