Mass pruning tool for unused RHCOS AMIs in rh-dev #3373

jupierce · 2022-12-08T04:08:12Z

No description provided.

mike-nguyen · 2022-12-12T19:05:47Z

This is great! Is there any way to get graph data for 4.5 and prior?

jupierce · 2022-12-12T19:53:03Z

@mike-nguyen - Not that the data is gone, but thankfully I ran this before the release controller team archived older releases. You can see that the <= 4.5 releases are mentioned in the analysis I posted: https://drive.google.com/file/d/19BVPz7qn-K_ADO8WZKSQBgXHB-81-3ND/view?usp=share_link . I'll build off this cache if we run this. I'll tag AMIs that are preserved so we don't have to do any fancy work to remember this fact in the future.

jlebon · 2022-12-12T20:33:49Z

Would be good to add a top-level docstring documenting the approach of this tool. IIUC, it's crawling through the update graph and then collating the AMIs to keep by looking at each node and its associated rhcos.json metadata in the corresponding openshift/installer repo commit.

I think it's a valid approach, but I think a better approach long-term is to prune at the cosa build level since it's not just AMIs that we want to prune. The code here could build a list of cosa build IDs we want to keep, and a GC process could prune all the cosa builds (and the referenced remote resources within, which include AMIs) which aren't on that list and older than X. The larger issue for that is tracked at coreos/fedora-coreos-tracker#99 and parts of the functionality exists in https://github.com/coreos/coreos-assembler/blob/main/src/cmd-remote-prune.

As is, this code will invalidate AMIs referenced in the meta.json of their associated cosa builds which isn't a huge concern, but isn't great either. I wouldn't consider it a blocker to do it this way.

mike-nguyen · 2022-12-12T20:49:28Z

@mike-nguyen - Not that the data is gone, but thankfully I ran this before the release controller team archived older releases. You can see that the <= 4.5 releases are mentioned in the analysis I posted: https://drive.google.com/file/d/19BVPz7qn-K_ADO8WZKSQBgXHB-81-3ND/view?usp=share_link . I'll build off this cache if we run this. I'll tag AMIs that are preserved so we don't have to do any fancy work to remember this fact in the future.

I looked at the file in my browser and didn't realize it only loaded partially until I scrolled down more.

sosiouxme

seems legit

sosiouxme · 2022-12-13T18:44:56Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+                # No machine-os version information in the image. In old releases, use image info for the
+                # machine-os-content image.
+                print('  Querying for machine-os-content info')
+                moc_pullspec = list(filter(lambda tag: tag['name'] == 'machine-os-content', release_info['references']['spec']['tags']))[0]['from']['name']


pretty soon we will cease to have a machine-os-content. a refinement here would look up the container configured as primary in group.yml.

We could add that, but the method itself is a fallback. In current builds, the rhcos version is in the payload metadata and doesn't require inspecting the rhcos image: https://github.com/openshift/aos-cd-jobs/pull/3373/files#diff-a3f55c4ee0db478ae07381ad1394f41bc69ea3b33c5f606ace1e946afd082c27R87

sosiouxme · 2022-12-13T18:47:03Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+                # machine-os-content image.
+                print('  Querying for machine-os-content info')
+                moc_pullspec = list(filter(lambda tag: tag['name'] == 'machine-os-content', release_info['references']['spec']['tags']))[0]['from']['name']
+                if 'registry.svc' in moc_pullspec:


hmm. that wouldn't ever happen in a release image we promoted would it? doesn't hurt to have this as failsafe but seems unnecessary. likewise below

I added this because some ancient CI payload did. Nothing ART promoted.

sosiouxme · 2022-12-13T19:09:12Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+                        if region_name not in installer_ami_ids:
+                            installer_ami_ids[region_name] = {}
+                            installer_ami_ids[region_name]['images'] = []
+                        ami_id = region_entry['image']
+                        installer_ami_ids[region_name]['images'].append(ami_id)


it's not important but i just prefer brevity

Suggested change

if region_name not in installer_ami_ids:

installer_ami_ids[region_name] = {}

installer_ami_ids[region_name]['images'] = []

ami_id = region_entry['image']

installer_ami_ids[region_name]['images'].append(ami_id)

ami_id = region_entry['image']

installer_ami_ids.setdefault(region_name, {}).setdefault('images', []).append(ami_id)

sosiouxme · 2022-12-13T19:16:00Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+                cache_entry = yaml.dump({
+                    payload_key: rhcos_in_use[payload_key]
+                })
+                # Write, line by line, to the cache file


minor note that yaml.dump includes a line break.

for a moment i was confused thinking each entry would be one line. perhaps better to say:

Suggested change

# Write, line by line, to the cache file

# Write to the cache file separated by a blank line

sosiouxme · 2022-12-13T19:44:49Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+                if payload_key_references['payload_version'] in ami_info:
+                    print(f'Will not clean up RHCOS AMI {image_id} ({image_name}) since it is referenced by a release controller release payload {payload_key}')
+                    matches_payload_keys.append(payload_key)


do we actually care about release payload references to AMIs? we only use AMIs at install time, right? and i'm pretty sure we don't reference AMIs anywhere besides build metadata and the installer so don't we only need to keep things that the installer refers to?

the difference isn't going to be huge though if you want to be cautious here and keep both.

the difference isn't going to be huge though if you want to be cautious here and keep both.
I don't think we do need them. So, yes, this is just a cautionary step.

sosiouxme · 2022-12-13T19:47:43Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+            if matches_payload_keys:
+                will_preserve.add(f'{aws_region}:{image_id}')
+
+            if not matches_payload_keys:


Suggested change

if not matches_payload_keys:

else:

? :-)

sosiouxme · 2022-12-14T16:19:21Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+                if '.ci' in payload_version or '.nightly' in payload_version:
+                    continue


Actually... we don't want to preserve everything in a nightly for the indefinite future, but we do want to preserve it for long enough to become a promoted release. So should there also just be a blanket exemption for AMIs until they're old enough (say, a month)? Presumably garbage collection config would cover that, perhaps that's why it's not built in here.

Right - If we ran this again the future, older nightlies would have been garbage collected from the release controller. As written, this tool would not try to preserve them.

mike-nguyen · 2022-12-15T01:36:39Z

I was just working on some boot image bumps and realized that this tool could mark images for deletion that are in openshift/installer pull requests that have not merged. I don't think that should affect this PR but just something to be aware of when this is setup to run.

With the new RHCOS pipeline, we won't be replicating to all regions unless we run the cloud-replicate job (which will only be used when creating boot images) so that should cut down on the number of snapshots on AWS.

- AMIs less than 30 days old will not be considered for pruning. - Recycle bin enabling in all regions - Tags are now being applied to images to indicate whether they should be garbage collected or preserved.

jupierce · 2022-12-15T14:01:54Z

@mike-nguyen . In the latest push, I've excluded images created in the last 30 days.

RHCOS suggested that an exhaustive scan of installer commits was safer than isolating AMIs associated with active releases.

jlebon · 2022-12-16T15:37:11Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+
+    git_depth = 0
+    while True:
+        installer_commit: Commit = f'upstream/master~{git_depth}'


This is going down master only. I think we need to go down from all the release branches too, no?

Great call. I wasn't considering post-cut changes. Updated.

jlebon · 2022-12-16T15:42:05Z

hacks/prune-rhcos-snapshots/prune-rhcos-snapshots.py

+        # or older format like https://github.com/openshift/installer/blob/bfdba2d19e3fa1105bab93aa4cce8bb7f44b4b2c/data/data/rhcos.json
+        amis_in_file = list(re.findall('ami-[a-z0-9]+', str(gitshow_process.stdout)))
+        amis_in_use.update(amis_in_file)
+        git_depth += 1


Instead of visiting every commit on the branch, it'd be more efficient to ask git for only the commits which touch the files we care about and then only visit those. Something like:

git log --format=%H <branch> -- data/data/coreos/rhcos.json data/data/rhcos.json

I like it. Updated.

jupierce · 2022-12-16T17:54:23Z

That latest push scans all branches and finds 2159 AMIs from the installer repo.

jlebon

Approach looks sane to me! One thing I double-checked was whether we've used rhcos.json even from 4.1.0, which we did.

mike-nguyen · 2022-12-22T14:09:45Z

LGTM to run

jupierce force-pushed the clean-up-snapshots branch 2 times, most recently from db3ce8f to a081589 Compare December 8, 2022 18:23

Mass pruning tool for unused RHCOS AMIs in rh-dev

cc930b9

jupierce force-pushed the clean-up-snapshots branch 2 times, most recently from cc930b9 to d12a701 Compare December 8, 2022 22:29

sosiouxme approved these changes Dec 13, 2022

View reviewed changes

sosiouxme reviewed Dec 14, 2022

View reviewed changes

Enable safety checks and tagging

b257519

- AMIs less than 30 days old will not be considered for pruning. - Recycle bin enabling in all regions - Tags are now being applied to images to indicate whether they should be garbage collected or preserved.

jupierce force-pushed the clean-up-snapshots branch from d12a701 to b257519 Compare December 15, 2022 13:59

Simplify to simply scanning installer commits for AMIs

47bf15e

RHCOS suggested that an exhaustive scan of installer commits was safer than isolating AMIs associated with active releases.

jupierce force-pushed the clean-up-snapshots branch from e991e49 to 47bf15e Compare December 15, 2022 23:40

jlebon reviewed Dec 16, 2022

View reviewed changes

Check all branches and use git log instead

5242187

jlebon approved these changes Dec 20, 2022

View reviewed changes

jupierce merged commit d029e71 into openshift-eng:master Dec 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mass pruning tool for unused RHCOS AMIs in rh-dev #3373

Mass pruning tool for unused RHCOS AMIs in rh-dev #3373

jupierce commented Dec 8, 2022

mike-nguyen commented Dec 12, 2022

jupierce commented Dec 12, 2022

jlebon commented Dec 12, 2022

mike-nguyen commented Dec 12, 2022

sosiouxme left a comment

sosiouxme Dec 13, 2022

jupierce Dec 14, 2022

sosiouxme Dec 13, 2022

jupierce Dec 14, 2022

sosiouxme Dec 13, 2022

sosiouxme Dec 13, 2022

sosiouxme Dec 13, 2022

jupierce Dec 14, 2022

sosiouxme Dec 13, 2022

sosiouxme Dec 14, 2022

jupierce Dec 14, 2022

mike-nguyen commented Dec 15, 2022

jupierce commented Dec 15, 2022

jlebon Dec 16, 2022

jupierce Dec 16, 2022

jlebon Dec 16, 2022

jupierce Dec 16, 2022

jupierce commented Dec 16, 2022

jlebon left a comment

mike-nguyen commented Dec 22, 2022

	# Write, line by line, to the cache file
	# Write to the cache file separated by a blank line

		if '.ci' in payload_version or '.nightly' in payload_version:
		continue

Mass pruning tool for unused RHCOS AMIs in rh-dev #3373

Mass pruning tool for unused RHCOS AMIs in rh-dev #3373

Conversation

jupierce commented Dec 8, 2022

mike-nguyen commented Dec 12, 2022

jupierce commented Dec 12, 2022

jlebon commented Dec 12, 2022

mike-nguyen commented Dec 12, 2022

sosiouxme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mike-nguyen commented Dec 15, 2022

jupierce commented Dec 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jupierce commented Dec 16, 2022

jlebon left a comment

Choose a reason for hiding this comment

mike-nguyen commented Dec 22, 2022