Add bundle_replacements parameter to regenerate_bundle

.bandit

@@ -1,2 +1,2 @@
 [bandit]
-exclude: /tests,/docker,/htmlcov,/.pytest-cache,/.git*,__pycache__,/.tox,.eggs,*.egg,/env*,/iib-data,/venv*,/.vscode,/.pyre
+exclude: /tests,/docker,/htmlcov,/.pytest-cache,/.git*,__pycache__,/.tox,.eggs,*.egg,/env*,/iib-data,/venv,/venv*,/.vscode,/.pyre

.flake8

@@ -0,0 +1,6 @@
+[flake8]
+exclude =
+    .tox
+    build
+    dist
+    venv

README.md

@@ -15,6 +15,7 @@ Podman 1.9.2+ is required by IIB.
 * [General Documentation](https://iib.readthedocs.io/en/latest/)
 * [Python Module Documentation](https://iib.readthedocs.io/en/latest/module_documentation/index.html)
 
+# Development
 ## Coding Standards
 
 The codebase conforms to the style enforced by `flake8` with the following exceptions:
@@ -57,7 +58,7 @@ The testing environment is managed by [tox](https://tox.readthedocs.io/en/latest
 If you'd like to run a specific unit test, you can do the following:
 
 ```bash
-tox -e py37 tests/test_web/test_api_v1.py::test_add_bundle_invalid_param
+tox -e py38 tests/test_web/test_api_v1.py::test_add_bundle_invalid_param
 ```
 
 ## Development Environment
@@ -121,14 +122,49 @@ adding a new package. To upgrade a package, use the `-P` argument of the `pip-co
 To update `requirements-test.txt`, run
 `pip-compile --generate-hashes requirements-test.in -o requirements-test.txt`.
 
+Both `requirements.txt` and `requirements-test.txt` can be updated using the tox command
+
+```bash
+$ tox -e pip-compile
+```
+
+You can also use this to upgrade specific packages and specific package versions
+
+```bash
+$ tox -e pip-compile -- --upgrade-package <package-name><==package-version>
+```
+
 When installing the dependencies in a production environment, run
 `pip install --require-hashes -r requirements.txt`. Alternatively, you may use
 `pip-sync requirements.txt`, which will make sure your virtualenv only has the packages listed in
 `requirements.txt`.
 
+
 To ensure the pinned dependencies are not vulnerable, this project uses
 [safety](https://github.com/pyupio/safety), which runs on every pull-request.
 
+## Database Migrations
+
+When the models are modified, the database schema also needs to be updated which includes leveraging
+`alembic` to generate a database migration. In order to simplify this process, a `tox` command can be
+used
+
+```bash
+$ tox -e migrate-db <simple message describing the change.>
+```
+
+Before running this command, however, you should `SQLALCHEMY_DATABASE_URI` to point to `'sqlite:///'`
+in `iib/web/config.py`. This parameter also needs to be added to the `Config` class specification.
+
+**NOTE:** This tox command was written as a convenience function that might not work in all situations.
+If alembic cannot auto-create the version file for the migration with `migrate`, you can just create it
+and then manually populate the steps with
+
+```bash
+$ flask db revision -m <simple message describing the change.>
+```
+
+# Configuring IIB deployments
 ## Registry Authentication
 
 IIB does not handle authentication with container registries directly. If authentication is needed,
@@ -310,6 +346,10 @@ The custom configuration options for the Celery workers are listed below:
     question to their digests. If this customization is not specified in the config for an
     organization, the pinning will not be done. If no organization is specified, IIB will try and
     pin the pull specs in the CSV files to their digests.
+  * The `perform_bundle_replacements` customization type is a dictionary with no additional
+    arguments. It enables bundle replacements to be passed to the bundle regeneration APIs with
+    the `bundle_replacements` parameter. If the customization type is not set, and bundle
+    replacements specified will be ignored.
 
   Here is an example that ties this all together:
 
@@ -370,6 +410,7 @@ must be set along with `iib_aws_s3_bucket_name` config variable:
 
 More info on these environment variables can be found in the [AWS User Guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html)
 
+# Additional information on specific IIB functionality
 ## Regenerating Bundle Images
 
 In addition to building operator index images, IIB can also be used to regenerate operator bundle
@@ -415,9 +456,10 @@ them to the index image. If a Greenwave configuration is setup for your queue, I
 Greenwave to check if your bundle image builds have passed the tests in the Greenwave policy you
 have defined. The IIB request submitted to that queue will succeed only if the policy is satisfied.
 
-## Read the Docs Documentation
+# Documentation
+This package has documentaiton on [Read the Docs](https://iib.readthedocs.io)
 
-### Build the Docs
+## Build the Docs
 
 To build and serve the docs, run the following commands:
 
@@ -426,7 +468,7 @@ tox -e docs
 google-chrome .tox/docs_out/index.html
 ```
 
-### Expanding the Docs
+## Expanding the Docs
 
 To document a new Python module, find the `rst` file of the corresponding Python package that
 contains the module. Once found, add a section under "Submodules" in alphabetical order such as:

iib/web/api_v1.py

@@ -525,6 +525,13 @@ def patch_request(request_id):
             for bundle in value:
                 if not isinstance(bundle, str):
                     raise ValidationError(exc_msg)
+        elif key == 'bundle_replacements':
+            exc_msg = f'The "{key}" key must be a dictionary object mapping from strings to strings'
+            if not isinstance(value, dict):
+                raise ValidationError(exc_msg)
+            for k, v in value.items():
+                if not isinstance(v, str) or not isinstance(k, str):
+                    raise ValidationError(exc_msg)
         elif not value or not isinstance(value, str):
             raise ValidationError(f'The value for "{key}" must be a non-empty string')
 
@@ -667,6 +674,7 @@ def regenerate_bundle():
         payload.get('organization'),
         request.id,
         payload.get('registry_auths'),
+        payload.get('bundle_replacements'),
     ]
     safe_args = _get_safe_args(args, payload)
 
@@ -731,6 +739,7 @@ def regenerate_bundle_batch():
                 build_request.get('organization'),
                 request.id,
                 build_request.get('registry_auths'),
+                build_request.get('bundle_replacements'),
             ]
             safe_args = _get_safe_args(args, build_request)
             error_callback = failed_request_callback.s(request.id)

iib/web/migrations/versions/a0eadb516360_update_regenerate_bundle_request_.py

@@ -0,0 +1,26 @@
+"""Update regenerate_bundle_request endpoint.
+
+Revision ID: a0eadb516360
+Revises: 625fba6081be
+Create Date: 2022-09-06 15:00:55.115536
+
+"""
+from alembic import op
+import sqlalchemy as sa
+
+
+# revision identifiers, used by Alembic.
+revision = 'a0eadb516360'
+down_revision = '625fba6081be'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    with op.batch_alter_table('request_regenerate_bundle', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('bundle_replacements', sa.String(), nullable=True))
+
+
+def downgrade():
+    with op.batch_alter_table('request_regenerate_bundle', schema=None) as batch_op:
+        batch_op.drop_column('bundle_replacements')

iib/web/models.py

@@ -1178,12 +1178,34 @@ class RequestRegenerateBundle(Request):
     )
     # The name of the organization the bundle should be regenerated for
     organization = db.Column(db.String, nullable=True)
+    # The mapping of bundle replacements to apply to the regeneration request
+    _bundle_replacements = db.Column('bundle_replacements', db.Text, nullable=True)
 
     __mapper_args__ = {
         'polymorphic_identity': RequestTypeMapping.__members__['regenerate_bundle'].value
     }
     build_tags = None
 
+    @property
+    def bundle_replacements(self):
+        """Return the Python representation of the JSON bundle_replacements."""
+        return json.loads(self._bundle_replacements) if self._bundle_replacements else None
+
+    @bundle_replacements.setter
+    def bundle_replacements(self, bundle_replacements):
+        """
+        Set the bundle_replacements column to the input bundle_replacements as a JSON string.
+
+        If ``None`` is provided, it will be simply set to ``None`` and not be converted to JSON.
+
+        :param dict bundle_replacements: the dictionary of the bundle_replacements or ``None``
+        """
+        self._bundle_replacements = (
+            json.dumps(bundle_replacements, sort_keys=True)
+            if bundle_replacements is not None
+            else None
+        )
+
     @classmethod
     def from_json(cls, kwargs, batch=None):
         """
@@ -1199,8 +1221,17 @@ def from_json(cls, kwargs, batch=None):
         validate_request_params(
             request_kwargs,
             required_params={'from_bundle_image'},
-            optional_params={'organization', 'registry_auths'},
+            optional_params={'bundle_replacements', 'organization', 'registry_auths'},
         )
+        # Validate bundle_replacements is correctly provided
+        bundle_replacements = request_kwargs.get('bundle_replacements', None)
+        if bundle_replacements is not None:
+            if not isinstance(bundle_replacements, dict):
+                raise ValidationError('The value of "bundle_replacements" must be a JSON object')
+
+            for key, value in bundle_replacements.items():
+                if not isinstance(value, str) or not isinstance(key, str):
+                    raise ValidationError(f'The key and value of "{key}" must be a string')
 
         # Validate organization is correctly provided
         organization = request_kwargs.get('organization')
@@ -1271,6 +1302,7 @@ def get_mutable_keys(self):
         rv = super().get_mutable_keys()
         rv.add('bundle_image')
         rv.add('from_bundle_image_resolved')
+        rv.add('bundle_replacements')
         return rv
 
 

iib/web/static/api_v1.yaml

@@ -1075,6 +1075,16 @@ components:
             auths:
               registry.example.io:
                 auth: "my-secret-token-base-64-encoded"
+        bundle_replacements:
+          type: object
+          description: >
+            A dictionary of bundle image pullspecs to replace in the CSV. The dictionary
+            should contain a mapping of the original pullspec to the target pullspec. Since
+            replacement happens according to the location of the `perform_bundle_replacements`
+            customization type, the original pullspecs need to be accurate for where the
+            customization is run.
+          example:
+            quay.io/foo/bar@sha256:123456: quay.io/foo/bar@sha256:654321
         related_bundles:
           type: object
           properties:

iib/workers/config.py

@@ -104,6 +104,7 @@ class DevelopmentConfig(Config):
         'company-marketplace': [
             IIBOrganizationCustomizations({'type': 'resolve_image_pullspecs'}),
             IIBOrganizationCustomizations({'type': 'related_bundles'}),
+            IIBOrganizationCustomizations({'type': 'perform_bundle_replacements'}),
             CSVAnnotations(
                 {
                     'type': 'csv_annotations',

iib/workers/tasks/build_regenerate_bundle.py

@@ -55,15 +55,18 @@ def handle_regenerate_bundle_request(
     organization: str,
     request_id: int,
     registry_auths: Optional[Dict[str, Any]] = None,
+    bundle_replacements: Optional[Dict[str, str]] = None,
 ) -> None:
     """
     Coordinate the work needed to regenerate the operator bundle image.
 
     :param str from_bundle_image: the pull specification of the bundle image to be regenerated.
     :param str organization: the name of the organization the bundle should be regenerated for.
-    :param int request_id: the ID of the IIB build request
+    :param int request_id: the ID of the IIB build request.
     :param dict registry_auths: Provide the dockerconfig.json for authentication to private
       registries, defaults to ``None``.
+    :param dict bundle_replacements: Dictionary mapping from original bundle pullspecs to rebuilt
+      bundle pullspecs.
     :raises IIBError: if the regenerate bundle image build fails.
     """
     _cleanup()
@@ -104,7 +107,12 @@ def handle_regenerate_bundle_request(
             metadata_path = os.path.join(temp_dir, 'metadata')
             _copy_files_from_image(from_bundle_image_resolved, '/metadata', metadata_path)
             new_labels = _adjust_operator_bundle(
-                manifests_path, metadata_path, request_id, organization, pinned_by_iib
+                manifests_path,
+                metadata_path,
+                request_id,
+                organization=organization,
+                pinned_by_iib=pinned_by_iib,
+                bundle_replacements=bundle_replacements,
             )
 
             with open(os.path.join(temp_dir, 'Dockerfile'), 'w') as dockerfile:
@@ -199,6 +207,7 @@ def _adjust_operator_bundle(
     organization: Optional[str] = None,
     pinned_by_iib: bool = False,
     recursive_related_bundles: bool = False,
+    bundle_replacements: Optional[Dict[str, str]] = {},
 ) -> Dict[str, str]:
     """
     Apply modifications to the operator manifests at the given location.
@@ -222,6 +231,8 @@ def _adjust_operator_bundle(
         IIB to perform image pinning of related images.
     :param bool recursive_related_bundles: whether or not the call is from a
         recursive_related_bundles request.
+    :param dict bundle_replacements: mapping between original pullspecs and rebuilt bundles,
+        allowing the updating of digests if any bundles have been rebuilt.
     :raises IIBError: if the operator manifest has invalid entries
     :return: a dictionary of labels to set on the bundle
     :rtype: dict
@@ -241,6 +252,7 @@ def _adjust_operator_bundle(
             {'type': 'related_bundles'},
             {'type': 'package_name_suffix'},
             {'type': 'registry_replacements'},
+            {'type': 'perform_bundle_replacements'},
             {'type': 'image_name_from_labels'},
             {'type': 'csv_annotations'},
             {'type': 'enclose_repo'},
@@ -322,6 +334,14 @@ def _adjust_operator_bundle(
             log.info('Resolving image pull specs')
             bundle_metadata = _get_bundle_metadata(operator_manifest, pinned_by_iib)
             _resolve_image_pull_specs(bundle_metadata, labels, pinned_by_iib)
+        elif customization_type == 'perform_bundle_replacements':
+            log.info('Performing bundle replacements')
+            bundle_metadata = _get_bundle_metadata(operator_manifest, pinned_by_iib)
+            replacement_pullspecs = {}
+            for old, new in bundle_replacements.items():
+                if _is_bundle_image(old):
+                    replacement_pullspecs[ImageName.parse(old)] = ImageName.parse(new)
+            _replace_csv_pullspecs(bundle_metadata, replacement_pullspecs)
 
     return labels
 
@@ -594,13 +614,24 @@ def get_related_bundle_images(bundle_metadata: BundleMetadata) -> List[str]:
     related_bundle_images = []
     for related_pullspec_obj in bundle_metadata['found_pullspecs']:
         related_pullspec = related_pullspec_obj.to_str()
-        if yaml.load(
-            get_image_label(related_pullspec, 'com.redhat.delivery.operator.bundle') or 'false'
-        ):
+        if _is_bundle_image(related_pullspec):
             related_bundle_images.append(related_pullspec)
     return related_bundle_images
 
 
+def _is_bundle_image(image_pullspec: str) -> bool:
+    """
+    Determine whether a specific image pullspec is for a bundle image.
+
+    :param str image_pullspec: the string of the image pullspec to test
+    :rtype: bool
+    :return: whether the image is considered a bundle image
+    """
+    return yaml.load(
+        get_image_label(image_pullspec, 'com.redhat.delivery.operator.bundle') or 'false'
+    )
+
+
 def write_related_bundles_file(
     related_bundle_images: List[str], request_id: int, local_directory: str, s3_file_identifier: str
 ) -> None:

iib/workers/tasks/iib_static_types.py

@@ -39,6 +39,7 @@ class PrebuildInfo(TypedDict):
     binary_image: str
     binary_image_resolved: str
     bundle_mapping: NotRequired[Dict[str, List[str]]]
+    bundle_replacements: NotRequired[Dict[str, str]]
     distribution_scope: str
     extra: NotRequired[str]
     from_index_resolved: NotRequired[str]
@@ -67,6 +68,7 @@ class UpdateRequestPayload(TypedDict, total=False):
     binary_image_resolved: NotRequired[str]
     bundle_image: NotRequired[str]
     bundle_mapping: NotRequired[Dict[str, List[str]]]
+    bundle_replacements: NotRequired[Dict[str, str]]
     distribution_scope: NotRequired[str]
     from_bundle_image_resolved: NotRequired[str]
     from_index_resolved: NotRequired[str]