Automation for updating templates #1347

AkihiroSuda · 2023-02-02T01:52:55Z

afbjorklund · 2023-02-02T07:06:23Z

Maybe also some means of sharing them, with some kind of FROM system ?

Inherit system images and share them between examples #824

AkihiroSuda · 2023-02-02T07:51:56Z

Maybe also some means of sharing them, with some kind of FROM system ?

Inherit system images and share them between examples #824

Yes, but that is a separate issue

jlm0x017 · 2023-02-09T04:34:25Z

Do you already have the link to the updated image? Or is that step needed as well?
I expect you grab the checksum provided at the source, rather than generating it yourself, but please do confirm.

AkihiroSuda · 2023-02-10T00:57:59Z

Do you already have the link to the updated image?

No, e.g., we have to detect the latest version 20230124-1270 from https://cloud.debian.org/images/cloud/bullseye/ , but I'm not sure what is the robust way to do this.

w3m | grep might be enough, but seriously we should also consider adopting some machine learning stuff.

lobshunter · 2023-04-21T03:07:01Z

What about a naive bash/python script to replace placeholder strings in yaml? Something like:

ubuntu_image = `w3m | grep -e "xxx"`
sed -i "s/UBUNTU_IMAGE/$ubuntu_image/g" ubuntu.yaml

The robustness solely relies on consistent file naming of upstreams(counting on hyrum's law). Sadly I didn't find any public tool to retrive latest released images.

AkihiroSuda · 2023-04-21T05:30:31Z

I'm now planning to use yq

balajiv113 · 2023-05-29T12:26:48Z

How about using libosinfo (osinfo-db os) ??

I could see their db has info of different os variants
https://gitlab.com/libosinfo/osinfo-db/-/tree/main/data/os

We might need to write a python wrapper on top of this library (the tool osinfo-db is not giving out info on image download URL's by architecture that is present in the xml file)

afbjorklund · 2023-05-29T12:32:09Z

yq can read xml too

yq -p xml -P

afbjorklund · 2023-05-29T12:34:08Z

The library and database are licensed under the terms of the GNU LGPL version 2 or later.

https://libosinfo.org/

AkihiroSuda · 2023-05-29T12:38:44Z

osinfo-db

Doesn't seem to contain permalinks: https://gitlab.com/libosinfo/osinfo-db/-/blob/ea8a7974a1f7189953c80fa9b1478b1ff8a75f8e/data/os/ubuntu.com/ubuntu-23.04.xml.in

    <image arch="x86_64" format="qcow2" cloud-init="true">
      <url>https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img</url>
    </image>

balajiv113 · 2023-05-29T12:39:06Z

yq can read xml too

True, but if we can use API it would be great. Else with yq we might need to read all xml files under each folder that we are interested in. With API i think it will be more managable

GNU LGPL version 2 or later

I thought since we are going to use this more of a build tool (Mostly a github actions workflow) this should not be a problem.

afbjorklund · 2023-05-29T12:40:06Z

It was mostly referring to the "and database", most of the tools actually seem to be GPL v2 (and to require glib)

balajiv113 · 2023-05-29T12:58:57Z

AkihiroSuda · 2023-07-31T16:20:57Z

I guess we can consider using GPT

afbjorklund · 2024-06-28T08:20:06Z

For Ubuntu, this is implemented in the ironically named "simple streams" (it's 14M):

sudo apt install simplestreams ubuntu-keyring

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson release='noble' ftype='disk1.img' | jq -r '.[] | [.item_url,.arch,.sha256]'

https://philroche.net/2018/02/12/ubuntu-cloud-images-and-how-to-find-the-most-recent-cloud-image-part-1-of-3/

The JSON+GPG file is: http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson

There is also a highlevel command:

sudo snap install image-status

$ image-status cloud-release
focal    amd64  20240626  disk1.img
jammy    amd64  20240627  disk1.img
mantic   amd64  20240619  disk1.img
noble    amd64  20240622  disk1.img

Where "disk1.img" is the old spelling of QCOW.

AkihiroSuda · 2024-10-08T16:14:56Z

Thanks @norio-nomura 🎉

Add hack/update-template-ubuntu.sh #2702

AkihiroSuda · 2024-10-10T01:26:22Z

For Debian, probably we can parse this JSON: https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.json

sha512 isn't encoded in hex though: "cloud.debian.org/digest": "sha512:2oTWCdfsVkXa4d9QPqcgN7KoMUAdG0LOLn7CqEC2mfB8qK6mMIU6PVQwg5Jowr0ze+RdiUmCZMNqm14ShyxZ7g"

Probably it should be just grepped from https://cloud.debian.org/images/cloud/bookworm/latest/SHA512SUMS instead.

AkihiroSuda · 2024-10-10T01:37:25Z

For ArchLinux, curl -fsSL https://gitlab.archlinux.org/api/v4/projects/archlinux%2Farch-boxes/packages | jq '.[-1].version' can be used for retrieving the latest version

norio-nomura · 2024-10-10T02:29:56Z

sha512 isn't encoded in hex though: "cloud.debian.org/digest": "sha512:2oTWCdfsVkXa4d9QPqcgN7KoMUAdG0LOLn7CqEC2mfB8qK6mMIU6PVQwg5Jowr0ze+RdiUmCZMNqm14ShyxZ7g"

It appears that this is a base64-encoded hash binary with the trailing "==" removed.
The following steps will convert it into hex-encoded format.

$ debian_sha512="2oTWCdfsVkXa4d9QPqcgN7KoMUAdG0LOLn7CqEC2mfB8qK6mMIU6PVQwg5Jowr0ze+RdiUmCZMNqm14ShyxZ7g"
$ echo "${debian_sha512}=="|base64 -d|xxd -p -c -
da84d609d7ec5645dae1df503ea72037b2a831401d1b42ce2e7ec2a840b699f07ca8aea630853a3d5430839268c2bd337be45d89498264c36a9b5e12872c59ee

AkihiroSuda · 2024-11-01T09:31:52Z

Marking as completed, huge thanks to @norio-nomura 🎉

norio-nomura · 2024-11-01T09:35:30Z

👍🏻
I’m still researching the OpenSUSE releases. Once it looks feasible, I’ll create an update-template-opensuse.sh script as well.

⚠️ **CAUTION: this is a major update, indicating a breaking change!** ⚠️ This MR contains the following updates: | Package | Update | Change | |---|---|---| | [lima-vm/lima](https://github.com/lima-vm/lima) | major | `v0.23.2` -> `v1.0.2` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>lima-vm/lima (lima-vm/lima)</summary> ### [`v1.0.2`](https://github.com/lima-vm/lima/releases/tag/v1.0.2) [Compare Source](lima-vm/lima@v1.0.1...v1.0.2) #### Changes - DNS: - Fixed the host resolver regression in v1.0.0 [#2939](lima-vm/lima#2939) ([#2964](lima-vm/lima#2964)) - `limactl create`: - Fixed races during parallel downloads ([#2903](lima-vm/lima#2903), thanks to [@nirs](https://github.com/nirs)) - Optimized qcow2-to-raw conversion for vz mode ([#2933](lima-vm/lima#2933), thanks to [@nirs](https://github.com/nirs)) - `limactl start-at-login`: - Fixed the support for Linux hosts (systemd) ([#2943](lima-vm/lima#2943), thanks to [@kachick](https://github.com/kachick)) - nerdctl: - Updated to [v2.0.1](https://github.com/containerd/nerdctl/releases/tag/v2.0.1) ([#2966](lima-vm/lima#2966)) - Templates: - Updated to the latest revisions ([#2936](lima-vm/lima#2936) [#2953](lima-vm/lima#2953), thanks to [@tcooper](https://github.com/tcooper)) - Web site: - Added an example of running Lima on GitHub Actions to run commands on non-Ubuntu ([#2954](lima-vm/lima#2954)): https://lima-vm.io/docs/examples/gha/ - Project: - Invite Nir Soffer ([@nirs](https://github.com/nirs)) as a Reviewer ([#2916](lima-vm/lima#2916), thanks to [@jandubois](https://github.com/jandubois)) Full changes: https://github.com/lima-vm/lima/milestone/51?closed=1 Thanks to [@SpiffyEight77](https://github.com/SpiffyEight77) [@alexandear](https://github.com/alexandear) [@jandubois](https://github.com/jandubois) [@kachick](https://github.com/kachick) [@nirs](https://github.com/nirs) [@norio-nomura](https://github.com/norio-nomura) [@tamird](https://github.com/tamird) [@tcooper](https://github.com/tcooper) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/12134682585 The sha256sum of the SHA256SUMS file itself is `02ef78494c498ca4180915ba78d5e2fc471ed401f63dfb2b5864c3711f3c0fb2` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) ### [`v1.0.1`](https://github.com/lima-vm/lima/releases/tag/v1.0.1) [Compare Source](lima-vm/lima@v1.0.0...v1.0.1) Reverted the default port forwarder from gRPC to SSH for the stability reason ([#2864](lima-vm/lima#2864)). This reversion fixes several regressions related to `docker run -p` in Lima v1.0.0 ([#2859](lima-vm/lima#2859)). Although the gRPC forwarder is faster and has an advanced feature (UDP support), it turned out to be still immature. Set `LIMA_SSH_PORT_FORWARDER=false` to opt-in to the gRPC forwarder. See <https://lima-vm.io/docs/config/port/>. Full changes: https://github.com/lima-vm/lima/milestone/50?closed=1 Thanks to [@alexandear](https://github.com/alexandear) [@jandubois](https://github.com/jandubois) [@norio-nomura](https://github.com/norio-nomura) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/11735352652 The sha256sum of the SHA256SUMS file itself is `f5c12d003e25dc46291803a8acae9e9d325a45eca0c1f9f40bd6852ec8ed9be1` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) ### [`v1.0.0`](https://github.com/lima-vm/lima/releases/tag/v1.0.0) [Compare Source](lima-vm/lima@v0.23.2...v1.0.0) With the support from 110+ contributors in 3+ years, the Lima project has finally reached v1.0. 🎉 This release introduces several breaking changes, such as switching the default machine driver from QEMU to VZ for better filesystem performance. The `limactl` CLI is designed to print hints when the user hits those breaking changes. e.g., `limactl create template://experimental/vz` now fails with a hint that suggests using `limactl create --vm-type=vz template://default` instead. 🔴 = Major breaking changes 🟡 = Minor breaking changes - VZ: - Graduate VZ machine driver from experimental ([#2758](lima-vm/lima#2758)) - 🔴 Use VZ by default for new instances on macOS >= 13.5 ([#1951](lima-vm/lima#1951)) - Support nested virtualization on M3 ([#2530](lima-vm/lima#2530), thanks to [@abiosoft](https://github.com/abiosoft)) - Optimize qcow2-to-raw image conversion (lima-vm/go-qcow2reader@v0.1.2...v0.4.0 , thanks to [@nirs](https://github.com/nirs)) - Support specifying a custom kernel ([#2562](lima-vm/lima#2562), thanks to [@norio-nomura](https://github.com/norio-nomura)) - QEMU: - Graduate 9p mount driver from experimental ([#2758](lima-vm/lima#2758)) - 🔴 Use 9p by default for most templates ([#1953](lima-vm/lima#1953), [#2822](lima-vm/lima#2822)) - riscv64: switch from u-boot to EDK2 ([#2592](lima-vm/lima#2592)) - Network: - Graduate user-v2 network driver from experimental ([#2758](lima-vm/lima#2758)) - Support UDP port forwarding ([#2411](lima-vm/lima#2411), thanks to [@balajiv113](https://github.com/balajiv113)) - 🔴 Strictly require `socket_vmnet` binary to be owned by root ([#2734](lima-vm/lima#2734)) - SSH: - 🟡 Disable `ssh.loadDotSSHPubKeys` by default ([#2706](lima-vm/lima#2706)) - YAML: - Support generating jsonschema ([#2306](lima-vm/lima#2306), thanks to [@afbjorklund](https://github.com/afbjorklund)) - Support specifying `param` for provisioning scripts ([#2570](lima-vm/lima#2570), thanks to [@jandubois](https://github.com/jandubois)) - Support specifying `minimumLimaVersion` and `vmOpts.qemu.minimumVersion` ([#2659](lima-vm/lima#2659), thanks to [@jandubois](https://github.com/jandubois)) - Support template expansion in mounts ([#2588](lima-vm/lima#2588), thanks to [@norio-nomura](https://github.com/norio-nomura)) - `limactl` CLI: - Add `limactl tunnel` command so as to allow the host to join the guest network ([#2710](lima-vm/lima#2710)) - Add `--log-format=json` ([#2584](lima-vm/lima#2584), thanks to [@nirs](https://github.com/nirs)) - `limactl prune`: Add `--keep-referred` ([#2569](lima-vm/lima#2569), thanks to [@norio-nomura](https://github.com/norio-nomura)) - nerdctl: - Updated to [v2.0.0](https://github.com/containerd/nerdctl/releases/tag/v2.0.0) ([#2178](lima-vm/lima#2178)) - rootless: allocate 1G subuids from 524288 (0x80000) for new users ([#2725](lima-vm/lima#2725)) - Templates: - 🔴 `experimental/vz`: Merged into the `default` template ([#2730](lima-vm/lima#2730), [#2736](lima-vm/lima#2736)) - 🟡 `experimental/{riscv64, armv7l}`: Merged into the `default` template ([#2730](lima-vm/lima#2730), [#2736](lima-vm/lima#2736)) - 🔴 `vmnet`: Removed in favor of `limactl create --network=lima:shared template://default` ([#2736](lima-vm/lima#2736)) - 🟡 `experimental/net-user-v2`: Removed in favor of `limactl create --network=lima:user-v2 template://default` ([#2736](lima-vm/lima#2736)) - 🔴 `experimental/9p`: Removed in favor of `limactl create --mount-type=9p template://default` ([#2736](lima-vm/lima#2736)) - 🟡 `experimental/virtiofs-linux`: Removed in favor of `limactl create --mount-type=virtiofs template://default` ([#2736](lima-vm/lima#2736)) - 🔴 `alpine`: Renamed to `alpine-iso` ([#2704](lima-vm/lima#2704)) - 🔴 `alpine-image`: Renamed to `alpine` ([#2704](lima-vm/lima#2704)) - `archlinux`: Demoted from Tier 1 to Tier 2 ([#2717](lima-vm/lima#2717), [#2823](lima-vm/lima#2823)) - `default`, `ubuntu`, ...: Updated to Ubuntu 24.10. The older versions are available as `ubuntu-20.04`, `ubuntu-22.04`, and `ubuntu-24.04` ([#2755](lima-vm/lima#2755), [#2795](lima-vm/lima#2795)) - `fedora`: Updated to Fedora 41 ([#2821](lima-vm/lima#2821), [#2822](lima-vm/lima#2822), thanks to [@subpop](https://github.com/subpop)) - `opensuse`: Renamed to `opensuse-leap`. Still aliased as `opensuse` ([#2612](lima-vm/lima#2612), thanks to [@afbjorklund](https://github.com/afbjorklund)) - `experimental/opensuse-tumbleweed`: Support aarch64 ([#2613](lima-vm/lima#2613), thanks to [@afbjorklund](https://github.com/afbjorklund)) - `hack/update-template.sh` is added for automating updates ([#1347](lima-vm/lima#1347), thanks to [@norio-nomura](https://github.com/norio-nomura)) - Project: - Invite Norio Nomura ([@norio-nomura](https://github.com/norio-nomura)) as a Reviewer ([#2567](lima-vm/lima#2567)) Full changes: https://github.com/lima-vm/lima/milestone/47?closed=1 Thanks to [@AdamKorcz](https://github.com/AdamKorcz) [@Mr-Sunglasses](https://github.com/Mr-Sunglasses) [@SmartManoj](https://github.com/SmartManoj) [@YorikSar](https://github.com/YorikSar) [@abiosoft](https://github.com/abiosoft) [@afbjorklund](https://github.com/afbjorklund) [@alexandear](https://github.com/alexandear) [@balajiv113](https://github.com/balajiv113) [@hasan4791](https://github.com/hasan4791) [@jandubois](https://github.com/jandubois) [@nirs](https://github.com/nirs) [@norio-nomura](https://github.com/norio-nomura) [@pvdvreede](https://github.com/pvdvreede) [@subpop](https://github.com/subpop) [@tsukasaI](https://github.com/tsukasaI) #### Usage ```console [macOS]$ limactl create [macOS]$ limactl start ... INFO[0029] READY. Run `lima` to open the shell. [macOS]$ lima uname Linux ``` *** The binaries were built automatically on GitHub Actions. The build log is available for 90 days: https://github.com/lima-vm/lima/actions/runs/11695321667 The sha256sum of the SHA256SUMS file itself is `4bd200a163111fe78c6f3e6de405113d416053802fe1507597f9a42f89a98c90` . *** Release manager: [@AkihiroSuda](https://github.com/AkihiroSuda) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

AkihiroSuda added help wanted Extra attention is needed expert labels Feb 2, 2023

This comment was marked as outdated.

Sign in to view

This was referenced May 12, 2023

templates: use yq's indentation style #1537

Closed

templates: use yq's indentation style #1389

Open

afbjorklund mentioned this issue May 27, 2023

Provide the size of the disk image, along with the checksum and url #1586

Open

afbjorklund mentioned this issue Apr 28, 2024

templates: update Ubuntu to 24.04 #2300

Merged

afbjorklund mentioned this issue May 29, 2024

Update almalinux 9 sample to 9.4 #2336

Merged

AkihiroSuda mentioned this issue Jun 25, 2024

Adds latest still supported ubuntu lts templates #2433

Closed

afbjorklund mentioned this issue Aug 10, 2024

Create amazonlinux2023 template #2537

Closed

AkihiroSuda pinned this issue Oct 10, 2024

norio-nomura mentioned this issue Oct 14, 2024

Refactor update-template-ubuntu.sh and add update-template-debian.sh #2731

Merged

4 tasks

This was referenced Oct 23, 2024

Add hack/update-template-archlinux.sh #2788

Merged

Add hack/update-template-centos-stream.sh #2794

Merged

AkihiroSuda unpinned this issue Oct 24, 2024

This was referenced Oct 24, 2024

Add hack/update-template-almalinux.sh #2796

Merged

Add hack/update-template-rocky.sh #2806

Merged

Add hack/update-template-alpine.sh #2811

Merged

Add hack/update-template-oraclelinux.sh #2820

Merged

Add hack/update-template-fedora.sh #2826

Merged

AkihiroSuda closed this as completed Nov 1, 2024

norio-nomura mentioned this issue Nov 6, 2024

Add hack/update-template-opensuse.sh #2852

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automation for updating templates #1347

Automation for updating templates #1347

AkihiroSuda commented Feb 2, 2023 •

edited

Loading

afbjorklund commented Feb 2, 2023

AkihiroSuda commented Feb 2, 2023

jlm0x017 commented Feb 9, 2023

AkihiroSuda commented Feb 10, 2023

This comment was marked as outdated.

This comment was marked as outdated.

lobshunter commented Apr 21, 2023

AkihiroSuda commented Apr 21, 2023

balajiv113 commented May 29, 2023

afbjorklund commented May 29, 2023

afbjorklund commented May 29, 2023 •

edited

Loading

AkihiroSuda commented May 29, 2023

balajiv113 commented May 29, 2023

afbjorklund commented May 29, 2023 •

edited

Loading

balajiv113 commented May 29, 2023

AkihiroSuda commented Jul 31, 2023

afbjorklund commented Jun 28, 2024 •

edited

Loading

AkihiroSuda commented Oct 8, 2024

AkihiroSuda commented Oct 10, 2024

AkihiroSuda commented Oct 10, 2024

norio-nomura commented Oct 10, 2024

AkihiroSuda commented Nov 1, 2024

norio-nomura commented Nov 1, 2024

Automation for updating templates #1347

Automation for updating templates #1347

Comments

AkihiroSuda commented Feb 2, 2023 • edited Loading

afbjorklund commented Feb 2, 2023

AkihiroSuda commented Feb 2, 2023

jlm0x017 commented Feb 9, 2023

AkihiroSuda commented Feb 10, 2023

This comment was marked as outdated.

This comment was marked as outdated.

lobshunter commented Apr 21, 2023

AkihiroSuda commented Apr 21, 2023

balajiv113 commented May 29, 2023

afbjorklund commented May 29, 2023

afbjorklund commented May 29, 2023 • edited Loading

AkihiroSuda commented May 29, 2023

balajiv113 commented May 29, 2023

afbjorklund commented May 29, 2023 • edited Loading

balajiv113 commented May 29, 2023

AkihiroSuda commented Jul 31, 2023

afbjorklund commented Jun 28, 2024 • edited Loading

AkihiroSuda commented Oct 8, 2024

AkihiroSuda commented Oct 10, 2024

AkihiroSuda commented Oct 10, 2024

norio-nomura commented Oct 10, 2024

AkihiroSuda commented Nov 1, 2024

norio-nomura commented Nov 1, 2024

AkihiroSuda commented Feb 2, 2023 •

edited

Loading

afbjorklund commented May 29, 2023 •

edited

Loading

afbjorklund commented May 29, 2023 •

edited

Loading

afbjorklund commented Jun 28, 2024 •

edited

Loading