Migrate index generation process to File-Based Catalog #2054

orenc1 · 2022-08-16T15:16:48Z

SQLite-Based catlog format, which we were using so far, is being deprecated in favour of the new format of file-based catalog.
With this change, the 'opm index add --bundles' command no longer works, and we need to change the workflow to work with json files inside the catalog image.
References:

Signed-off-by: Oren Cohen ocohen@redhat.com

Reviewer Checklist

Reviewers are supposed to review the PR for every aspect below one by one. To check an item means the PR is either "OK" or "Not Applicable" in terms of that item. All items are supposed to be checked before merging a PR.

Release note:

migrate to FBC in index image workflows.

coveralls · 2022-08-16T15:23:39Z

Pull Request Test Coverage Report for Build 2897762102

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 84.969%

Totals
Change from base Build 2888057182:	0.0%
Covered Lines:	4460
Relevant Lines:	5249

💛 - Coveralls

hco-bot · 2022-08-16T17:17:36Z

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

kubevirt-bot · 2022-08-16T17:17:39Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-08-16T17:39:04Z

hco-e2e-image-index-azure, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-gcp

kubevirt-bot · 2022-08-16T17:39:07Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-gcp

In response to this:

hco-e2e-image-index-azure, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa

Looks very good.

Added some comments.

nunnatsa · 2022-08-17T12:19:40Z

hack/build-index-image.sh

+  fi
+
+  ${OPM} render ${BUNDLE_IMAGE_NAME} > bundle.json
+  CSV_NAME=$(jq .name bundle.json | tr -d '"')


you can use the -r flag instead of tr -d:

Suggested change

CSV_NAME=$(jq .name bundle.json | tr -d '"')

CSV_NAME=$(jq -r .name bundle.json)

nunnatsa · 2022-08-17T12:21:58Z

hack/build-index-image.sh

-  ${OPM} index add --bundles "${BUNDLE_IMAGE_NAME}" ${INDEX_IMAGE_PARAM} --tag "${INDEX_IMAGE_NAME}" -u podman --mode semver
+
+  # File-Based Catalog handling
+  ${OPM} migrate ${INDEX_IMAGE_NAME} fbc-catalog || true


I think we should avoid writing to the root directory of the project. What do you think about output to _out/fbc-catalog?

If you do that, please add an env var for this path, in order to use it later in the script.

I think it would be better to change the working directory in this part to ${PROJECT_ROOT}/_out to avoid adding the _out/ prefix for every file reference.

nunnatsa · 2022-08-17T12:23:50Z

hack/build-index-image.sh

+    oc image extract ${INDEX_IMAGE_NAME} --file /configs/catalog.json
+  else
+    # The migration took place
+    mv fbc-catalog/community-kubevirt-hyperconverged/catalog.json catalog.json


Please move to _out/catalog.json

nunnatsa · 2022-08-17T12:24:26Z

hack/build-index-image.sh

+    mv fbc-catalog/community-kubevirt-hyperconverged/catalog.json catalog.json
+  fi
+
+  ${OPM} render ${BUNDLE_IMAGE_NAME} > bundle.json


please write to _out/bundle.json

nunnatsa · 2022-08-17T12:25:31Z

hack/build-index-image.sh

-  ${OPM} index add --bundles "${BUNDLE_IMAGE_NAME}" ${INDEX_IMAGE_PARAM} --tag "${INDEX_IMAGE_NAME}" -u podman --mode semver
+
+  # File-Based Catalog handling
+  ${OPM} migrate ${INDEX_IMAGE_NAME} fbc-catalog || true


Please wrap with double quotes.

Suggested change

${OPM} migrate ${INDEX_IMAGE_NAME} fbc-catalog || true

${OPM} migrate "${INDEX_IMAGE_NAME}" _out/fbc-catalog || true

nunnatsa · 2022-08-17T12:26:13Z

hack/build-index-image.sh

+  if [ ! -d fbc-catalog ]
+  then
+    # The index image is already in file-based format. Extracting its catalog file
+    oc image extract ${INDEX_IMAGE_NAME} --file /configs/catalog.json


Please wrap with double quotes.

Suggested change

oc image extract ${INDEX_IMAGE_NAME} --file /configs/catalog.json

oc image extract "${INDEX_IMAGE_NAME}" --file /configs/catalog.json

nunnatsa · 2022-08-17T12:27:28Z

hack/build-index-image.sh

+    mv fbc-catalog/community-kubevirt-hyperconverged/catalog.json catalog.json
+  fi
+
+  ${OPM} render ${BUNDLE_IMAGE_NAME} > bundle.json


Suggested change

${OPM} render ${BUNDLE_IMAGE_NAME} > bundle.json

${OPM} render "${BUNDLE_IMAGE_NAME}" > _out/bundle.json

nunnatsa · 2022-08-17T12:46:58Z

hack/build-index-image.sh

+    select(.schema=="olm.bundle" and (.name | contains($CHANNEL))))' \
+    updated_fbc.json.tmp > updated_fbc.json
+
+  mkdir -p fbc-catalog


maybe _out/fbc-catalog?

nunnatsa · 2022-08-17T12:47:47Z

hack/build-index-image.sh

+  remove_tmp_files
+}
+
+function remove_tmp_files() {


not needed if everything in _out

hco-bot · 2022-08-17T17:22:19Z

hco-e2e-upgrade-index-azure lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-aws

kubevirt-bot · 2022-08-17T17:22:24Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-index-aws

In response to this:

hco-e2e-upgrade-index-azure lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-08-17T17:44:20Z

hco-e2e-image-index-azure, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-gcp
hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

kubevirt-bot · 2022-08-17T17:44:25Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-gcp, ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-image-index-azure, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-gcp
hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-08-17T19:16:36Z

hco-e2e-kv-smoke-gcp lane succeeded.
/override ci/prow/hco-e2e-kv-smoke-azure

kubevirt-bot · 2022-08-17T19:16:39Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-kv-smoke-azure

In response to this:

hco-e2e-kv-smoke-gcp lane succeeded.
/override ci/prow/hco-e2e-kv-smoke-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2022-08-21T05:00:21Z

hack/build-index-image.sh

-  ${OPM} index add --bundles "${BUNDLE_IMAGE_NAME}" ${INDEX_IMAGE_PARAM} --tag "${INDEX_IMAGE_NAME}" -u podman --mode semver
+
+  # File-Based Catalog handling
+  cd "${OUT_DIR}"


This will work. But I'm not sure I'm happy with cd and cd back. What do you think about placing all of this in a new function, then call it like this?

(cd "${OUT_DIR}" && my_new_function)

valid point.
updated, thanks.

SQLite-Based catlog format, which we were using so far, is being deprecated in favour of the new format of file-based catalog. With this change, the 'opm index add --bundles' command no longer works, and we need to change the workflow to work with json files inside the catalog image. References: - https://olm.operatorframework.io/docs/reference/file-based-catalogs/ - k8s-operatorhub/community-operators#505 Signed-off-by: Oren Cohen <ocohen@redhat.com>

sonarcloud · 2022-08-21T08:17:57Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

nunnatsa · 2022-08-21T08:19:26Z

/lgtm

hco-bot · 2022-08-21T10:13:50Z

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

kubevirt-bot · 2022-08-21T10:13:52Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

hco-bot · 2022-08-21T10:34:56Z

hco-e2e-image-index-gcp, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-azure

kubevirt-bot · 2022-08-21T10:34:59Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-azure

In response to this:

hco-e2e-image-index-gcp, hco-e2e-image-index-aws lanes succeeded.
/override ci/prow/hco-e2e-image-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-08-21T10:46:40Z

@orenc1: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/hco-e2e-upgrade-prev-index-azure	`05c1fb7`	link	true	`/test hco-e2e-upgrade-prev-index-azure`
ci/prow/hco-e2e-image-index-azure	`05c1fb7`	link	true	`/test hco-e2e-image-index-azure`
ci/prow/hco-e2e-upgrade-prev-index-sno-azure	`05c1fb7`	link	false	`/test hco-e2e-upgrade-prev-index-sno-azure`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

hco-bot · 2022-08-21T10:55:16Z

hco-e2e-upgrade-prev-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-sno-azure

kubevirt-bot · 2022-08-21T10:55:19Z

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-prev-index-sno-azure

In response to this:

hco-e2e-upgrade-prev-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nunnatsa · 2022-08-21T11:13:36Z

/approve

kubevirt-bot · 2022-08-21T11:14:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nunnatsa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nunnatsa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. size/M labels Aug 16, 2022

kubevirt-bot requested review from sradco and tiraboschi August 16, 2022 15:17

nunnatsa reviewed Aug 17, 2022

View reviewed changes

orenc1 force-pushed the migrate_unstable_index_to_fbc branch from 3c10dee to f06e5c8 Compare August 17, 2022 14:42

nunnatsa reviewed Aug 21, 2022

View reviewed changes

orenc1 force-pushed the migrate_unstable_index_to_fbc branch from f06e5c8 to 05c1fb7 Compare August 21, 2022 08:16

kubevirt-bot assigned nunnatsa Aug 21, 2022

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 21, 2022

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 21, 2022

kubevirt-bot merged commit d689d03 into kubevirt:main Aug 21, 2022

orenc1 deleted the migrate_unstable_index_to_fbc branch August 21, 2022 12:19

	CSV_NAME=$(jq .name bundle.json \| tr -d '"')
	CSV_NAME=$(jq -r .name bundle.json)

	${OPM} migrate ${INDEX_IMAGE_NAME} fbc-catalog \|\| true
	${OPM} migrate "${INDEX_IMAGE_NAME}" _out/fbc-catalog \|\| true

	oc image extract ${INDEX_IMAGE_NAME} --file /configs/catalog.json
	oc image extract "${INDEX_IMAGE_NAME}" --file /configs/catalog.json

	${OPM} render ${BUNDLE_IMAGE_NAME} > bundle.json
	${OPM} render "${BUNDLE_IMAGE_NAME}" > _out/bundle.json

Migrate index generation process to File-Based Catalog #2054

Migrate index generation process to File-Based Catalog #2054

Conversation

orenc1 commented Aug 16, 2022

coveralls commented Aug 16, 2022 • edited Loading

Pull Request Test Coverage Report for Build 2897762102

💛 - Coveralls

hco-bot commented Aug 16, 2022

kubevirt-bot commented Aug 16, 2022

hco-bot commented Aug 16, 2022

kubevirt-bot commented Aug 16, 2022

nunnatsa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hco-bot commented Aug 17, 2022

kubevirt-bot commented Aug 17, 2022

hco-bot commented Aug 17, 2022

kubevirt-bot commented Aug 17, 2022

hco-bot commented Aug 17, 2022

kubevirt-bot commented Aug 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Aug 21, 2022

nunnatsa commented Aug 21, 2022

hco-bot commented Aug 21, 2022

kubevirt-bot commented Aug 21, 2022

hco-bot commented Aug 21, 2022

kubevirt-bot commented Aug 21, 2022

openshift-ci bot commented Aug 21, 2022 • edited Loading

hco-bot commented Aug 21, 2022

kubevirt-bot commented Aug 21, 2022

nunnatsa commented Aug 21, 2022

kubevirt-bot commented Aug 21, 2022

coveralls commented Aug 16, 2022 •

edited

Loading

openshift-ci bot commented Aug 21, 2022 •

edited

Loading