application-inventory: use core-kit version for packages sourced from the bottlerocket-core-kit

[ginglis13]

Description of changes:

In order for package version comparisons to be valid in the OOTB and
kits style builds of Bottlerocket, application inventory is generated to
list the version of a package by where it was sourced from. This will be
the kit version.

This is a first iteration on this approach and must be extended
in future work to apply to any external kit and not just the
bottlerocket-core-kit, although that is the only external kit for now.

This change will help resolve cases where downstream consumers of application inventory and security updates do not respect the Epoch field.

Testing done:

Local checkout of bottlerocket-os/bottlerocket#4060, built Bottlerocket and verified:

1. The version of packages matches the version of the core kit
2. The packages listed in app inventory are limited to those defined for the variant (i.e., does not include all packages in the core-kit)

Given that the version of the core kit I was using matched Bottlerocket's version, for testing purposes I overrode CORE_KIT_VERSION to "2.0" to observe changes:

{
  "Content": [
    {
      "Name": "acpid",
      "Publisher": "Bottlerocket",
      "Version": "2.0",
      "Release": "7a7b5dd3",
      "InstalledTime": "2024-06-18T21:52:21Z",
      "ApplicationType": "Unspecified",
      "Architecture": "aarch64",
      "Url": "http://sourceforge.net/projects/acpid2/",
      "Summary": "ACPI event daemon"
    }

and a package that is not in core kit retains Bottlerocket's version:

    {
      "Name": "settings-defaults",
      "Publisher": "Bottlerocket",
      "Version": "1.21.0",
      "Release": "7a7b5dd3",
      "InstalledTime": "2024-06-18T21:52:21Z",
      "ApplicationType": "Unspecified",
      "Architecture": "aarch64",
      "Url": "https://github.com/bottlerocket-os/bottlerocket",
      "Summary": "Settings defaults"
    }

Testing on a Bottlerocket 1.0.0

To test that these changes work as expected, I created an aws-dev variant, with a release-version of 1.0.0 in both Twoliter.toml and Release.toml. SSM reports inventory as expected:

And the generated app inventory is as expected:

{
  "Content": [
    {
      "Name": "acpid",
      "Publisher": "Bottlerocket",
      "Version": "2.0.0",
      "Release": "7a7b5dd3-dirty",
      "InstalledTime": "2024-06-19T04:56:22Z",
      "ApplicationType": "Unspecified",
      "Architecture": "aarch64",
      "Url": "http://sourceforge.net/projects/acpid2/",
      "Summary": "ACPI event daemon"
    },
...
  {
      "Name": "settings-defaults",
      "Publisher": "Bottlerocket",
      "Version": "1.0.0",
      "Release": "7a7b5dd3-dirty",
      "InstalledTime": "2024-06-19T04:56:22Z",
      "ApplicationType": "Unspecified",
      "Architecture": "aarch64",
      "Url": "https://github.com/bottlerocket-os/bottlerocket",
      "Summary": "Settings defaults"
    },

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[ginglis13]

^ use the correct core kit RPM repo path, based on core kit name + vendor

[ginglis13]

^ rename installed_core_kit_rpms to all_core_kit_rpms

[ginglis13]

^ improve the comment about writing the inventory to local build output dir in addition to the image itself, and remove some unnecessary jq processing

[ginglis13]

^ fix bug in setting INVENTORY_DATA_KIT, add more comments detailing the approach

[ginglis13]

^ # shellcheck disable=SC2312 where appropriate

[bcressey]

Here's a script with a few potential simplifications:

#!/bin/bash

set -euxo pipefail

EXTERNAL_KIT_PATH="/home/fedora/brdev/build/external-kits"
EXTERNAL_KIT_METADATA="${EXTERNAL_KIT_PATH}/external-kit-metadata.json"

CORE_KIT_VERSION="$(jq --raw-output '.kit[]|select(.name == "bottlerocket-core-kit")|.version' "${EXTERNAL_KIT_METADATA}")"
CORE_KIT_VENDOR="$(jq --raw-output '.kit[]|select(.name == "bottlerocket-core-kit")|.vendor' "${EXTERNAL_KIT_METADATA}")"
CORE_KIT_PATH="${EXTERNAL_KIT_PATH}/bcressey/bottlerocket-core-kit/x86_64"

if [[ -n "${CORE_KIT_VERSION}" && -n "${CORE_KIT_VENDOR}" ]]; then
  # List packages in the core kit repo.
  CORE_KIT_PKGS="$(\
    dnf \
      --repofrompath "core-kit,file://${CORE_KIT_PATH}" \
      --repo=core-kit repoquery \
      --queryformat "%{NAME}\n")"

  # Convert the list of core kit packages to a JSON array.
  CORE_KIT_LIST="$(\
    jq \
      --null-input \
      --compact-output \
      '$ARGS.positional // []' \
      --args ${CORE_KIT_PKGS[@]})"

  # Convert the JSON array to a map of prefixed to unprefixed package names.
  jq \
    --compact-output \
    'map({ (.|tostring): (.|sub("^bottlerocket-";""))}) | add' <<<"${CORE_KIT_LIST}" \
    > replacements.json

  # For any core kit packages in the inventory, replace the version with the
  # core kit's version, and replace the name with the unprefixed name.
  jq \
    --argfile replace replacements.json \
    --arg CORE_KIT_VERSION "${CORE_KIT_VERSION}" \
    '(.Content[] | select(.Name | $replace[.] != null) | .Version) = $CORE_KIT_VERSION |
     .Content[].Name |= (if $replace[.] then $replace[.] else . end)' \
    app1.json
fi

The repo query seems a bit more complex than required; we can borrow a trick from rpm2kit and turn a bash array into a JSON array.

The final jq transformations are essentially the same but made more efficient by generating a map of prefixed to unprefixed names, then using that to target replacements.

[ginglis13]

^ retain the %_cross_os prefix in package names installed in app inventory. Take @bcressey's suggestions for more efficient jq approach.

[ginglis13]

^ fix some bash lints, move # Verify we successfully inventoried some RPMs. to after the core kit inventory manipulation

[cbgbt]

Nice work.