Multi-platform cleanup #58

onyxraven · 2023-05-15T19:20:18Z

Background

Build consistency for multiplatform

builds docker manifest image using buildx bake
builds universal binary for macos
builds windows binaries
updates installer scripts to be os/platform aware
adds apk package
adds sboms

Versioning

Minor - we're updating a few file paths and things.

Additional Requests to Reviewers

Tasks

Specs written
Manual testing

onyxraven · 2023-05-15T19:23:10Z

Dockerfile

@@ -1,8 +1,14 @@
 FROM alpine:latest as build
+ARG TARGETARCH


gets amd64 etc from the --platform arg

onyxraven · 2023-05-15T19:23:57Z

Dockerfile

 COPY sopsinstall.sh /tmp/sopsinstall.sh
-RUN sh /tmp/sopsinstall.sh -b /usr/local/bin
+RUN sh /tmp/sopsinstall.sh -b /usr/local/bin -a $TARGETARCH


decided to continue using a script to get this since sops puts the version number in their github release artifacts, so it still needs to do the tag lookup.

onyxraven · 2023-05-15T19:25:53Z

scripts/deploy-sops

+$DIR/../sopsinstall.sh -o linux -a arm64 "$SOPSDIST"
+$DIR/../sopsinstall.sh -o darwin -a amd64 "$SOPSDIST"
+$DIR/../sopsinstall.sh -o darwin -a arm64 "$SOPSDIST"
+$DIR/../sopsinstall.sh -o windows -a amd64 "$SOPSDIST"


uses the same downloader to grab the archives for upload to our s3.

onyxraven · 2023-05-15T20:43:17Z

.goreleaser.yml

+    # Whether to remove the previous single-arch binaries from the artifact list.
+    # If left as false, your end release might have both several macOS archives:
+    # amd64, arm64 and all.
+    replace: true


locally, it still produces all 3. i'm not sure what the release will look like

onyxraven · 2023-05-15T20:43:33Z

.goreleaser.yml

  # Default is `{{ .ProjectName }}_{{ .Version }}_checksums.txt`.
  name_template: "{{ .ProjectName }}_checksums.txt"

+sboms:
+  - artifacts: binary
+    documents: ["{{ .Binary }}_{{ .Os }}_{{ .Arch }}.sbom"]


match format of the binary

onyxraven · 2023-05-15T20:43:58Z

.goreleaser.yml

    extra_files:
-      - sopsinstall.sh
+      - glob: ./*install.sh


have the deploy upload the installers instead of the extra script

onyxraven · 2023-05-15T20:45:44Z

scripts/deploy-docker

+REGEX="^v([0-9]+)\.([0-9]+)\.([0-9]+)"
+export VERSION=$(jq -r '.version' dist/metadata.json)
+export TAG=$(jq -r '.tag' dist/metadata.json)
+if [[ "${TAG}" =~ ${REGEX} ]]; then


gets the version information from the goreleaser metadata output

joellarsonibotta · 2023-05-15T22:01:40Z

.goreleaser.yml

+    goarm:
+      - ""
+    goamd64:
+      - ""


Is an empty string meant to clear the defaults? If so, you could set them to:

goarm: [] goamd64: []

I think I have to specificaly set them - the default for goamd64 is v1 and it ends up in variant stuff, which is weird. I can try but this is what the docs kinda looked like

joellarsonibotta · 2023-05-15T22:02:08Z

.goreleaser.yml

+    format_overrides:
+      - goos: windows
+        format: zip


Do we use windows?

Ibotta doesnt, but I want to on a windows project.

joellarsonibotta · 2023-05-15T22:05:05Z

Dockerfile

+# grab appropriate sopstool binary from dist
+COPY dist/sopstool_linux_$TARGETARCH/sopstool /usr/local/bin/sopstool


Did you mean to copy the dist file from the host machine?

I see now that the Dockerfile is just importing the binary anyway.

yes - this is the build of the image from the compiled source outputs of goreleaser.

joellarsonibotta · 2023-05-15T22:08:45Z

install.sh

+    http_exec https://raw.githubusercontent.com/Ibotta/sopstool/master/sopsinstall.sh -b "${BINDIR}" "$@" "${SOPS_VERSION}"
  fi

-  http_exec https://raw.githubusercontent.com/Ibotta/sopstool/master/sopstoolinstall.sh -b "$BINDIR" "$SOPSTOOL_VERSION"
+  http_exec https://raw.githubusercontent.com/Ibotta/sopstool/master/sopstoolinstall.sh -b "${BINDIR}" "$@" "${SOPSTOOL_VERSION}"


These github links are using master instead of main

good call, I thought I found them all but this snuck through

onyxraven · 2023-05-16T15:48:36Z

.github/workflows/release.yaml

@@ -29,6 +29,9 @@ jobs:
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

+      - name: Set up Syft
+        uses: anchore/sbom-action/download-syft@v0


this is so goreleaser can create SBOMs for the binaries (we also have docker buildx, which uses the same tool, create SBOMs for the containers)

onyxraven · 2023-05-16T15:48:57Z

.goreleaser.yml

@@ -1,3 +1,5 @@
+report_sizes: true


these get tracked in an output json, this was useful for debugging.

onyxraven · 2023-05-16T16:16:30Z

install.sh

@@ -8,39 +8,69 @@ usage() {

 $this: download binaries for sopstool

-Usage: $this [bindir]
-  [bindir] sets bindir or installation directory, Defaults to ./bin
+Usage: $this [-b bindir] [-o OS] [-a ARCH] [-s SOPS_VERSION] [-t SOPSTOOL_VERSION] [-d] [bindir]


this now uses getopts, but is backward compatible.

onyxraven · 2023-05-16T16:16:56Z

install.sh

-  if ! is_command sops; then
-    http_exec https://raw.githubusercontent.com/Ibotta/sopstool/master/sopsinstall.sh -b "$BINDIR" "$SOPS_VERSION"
+  if [ -n "${TARGET_ARCH}" ]; then
+    set -- "$@" "-a" "${TARGET_ARCH}"


arrays in posix shells lol.

onyxraven · 2023-05-16T16:18:02Z

sopsinstall.sh

+    sort -t "." -n -k1,1 -k2,2 -k3,3 -k4,4 | head -n 1)
+  if [ "${LOWEST_ARCH_VERSION}" != "${MIN_ARCH_VERSION}" ]; then
+    oldarch="${VERSION}"
+  fi


theres quite a bit of 'friendlyness' here that maybe isn't fully necessary - it'll bork at you if you pick arm64 windows, for example. or arm64 before sops was building for it,

onyxraven · 2023-05-16T16:18:44Z

sopsinstall.sh

+  install "${TMPDIR}/${NAME}" "${BINDIR}/${BINARY}"
+  log_info "installed ${BINDIR}/${BINARY}"
+}
+archive_binary() {


all of this is for our download and re-upload, though its kinda nice to have otherwise too. I might consider contribbing this to sops.

onyxraven · 2023-05-16T16:19:10Z

sopsinstall.sh

-    netbsd) return 0 ;;
-    openbsd) return 0 ;;
-    plan9) return 0 ;;
-    solaris) return 0 ;;


I went ahead and removed the unsupported os/arch from these lists as well

onyxraven · 2023-05-16T16:19:25Z

sopsinstall.sh

@@ -257,90 +296,47 @@ github_release() {
  test -z "$version" && return 1
  echo "$version"
 }
-hash_sha256() {


sops doesn't publish checksums :(

onyxraven · 2023-05-16T16:19:52Z

sopstoolinstall.sh

    darwin/amd64) BINARIES="sopstool" ;;
    darwin/arm64) BINARIES="sopstool" ;;
    linux/amd64) BINARIES="sopstool" ;;
    linux/arm64) BINARIES="sopstool" ;;
+    windows/amd64) BINARIES="sopstool.exe" ;;
+    windows/arm64) BINARIES="sopstool.exe" ;;


we can do windows now ;)

.github/CONTRIBUTING.md

Co-authored-by: Rishi Sheth <2817944+physik932@users.noreply.github.com>

elementalvoid

LGTM! These are some much-needed cleanups.

onyxraven added 3 commits May 12, 2023 13:57

not working yet

6057fce

move forward

849d30a

scripts and docs

170e574

onyxraven commented May 15, 2023

View reviewed changes

onyxraven force-pushed the multiplatformdocker branch 6 times, most recently from 123bf4e to c5275ee Compare May 15, 2023 20:39

fix building

a10040a

onyxraven force-pushed the multiplatformdocker branch from c5275ee to a10040a Compare May 15, 2023 20:53

onyxraven changed the title ~~Multiplatformdocker~~ Multi-platform cleanup May 15, 2023

onyxraven marked this pull request as ready for review May 15, 2023 20:58

onyxraven requested review from elementalvoid, ibottamike and physik932 as code owners May 15, 2023 20:58

onyxraven commented May 15, 2023

View reviewed changes

joellarsonibotta reviewed May 15, 2023

View reviewed changes

joellarsonibotta previously approved these changes May 15, 2023

View reviewed changes

master->main

8232353

onyxraven dismissed joellarsonibotta’s stale review via 8232353 May 15, 2023 22:53

onyxraven commented May 16, 2023

View reviewed changes

physik932 reviewed May 16, 2023

View reviewed changes

.github/CONTRIBUTING.md Outdated Show resolved Hide resolved

Update .github/CONTRIBUTING.md - add rtx link

6c7fcd7

Co-authored-by: Rishi Sheth <2817944+physik932@users.noreply.github.com>

onyxraven mentioned this pull request May 16, 2023

Remove installer scripts in favor of popular or upstream install methods #47

Open

elementalvoid approved these changes May 18, 2023

View reviewed changes

kr3cj approved these changes May 18, 2023

View reviewed changes

onyxraven merged commit f8ed143 into main May 18, 2023

onyxraven deleted the multiplatformdocker branch May 18, 2023 17:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-platform cleanup #58

Multi-platform cleanup #58

onyxraven commented May 15, 2023 •

edited

Loading

onyxraven May 15, 2023

onyxraven May 15, 2023

onyxraven May 15, 2023

onyxraven May 15, 2023

onyxraven May 15, 2023

onyxraven May 15, 2023

onyxraven May 15, 2023

joellarsonibotta May 15, 2023

onyxraven May 15, 2023

joellarsonibotta May 15, 2023

onyxraven May 15, 2023

joellarsonibotta May 15, 2023

joellarsonibotta May 15, 2023

onyxraven May 15, 2023

joellarsonibotta May 15, 2023

onyxraven May 15, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

onyxraven May 16, 2023

elementalvoid left a comment

		# grab appropriate sopstool binary from dist
		COPY dist/sopstool_linux_$TARGETARCH/sopstool /usr/local/bin/sopstool

Multi-platform cleanup #58

Multi-platform cleanup #58

Conversation

onyxraven commented May 15, 2023 • edited Loading

Background

Versioning

Additional Requests to Reviewers

Tasks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elementalvoid left a comment

Choose a reason for hiding this comment

onyxraven commented May 15, 2023 •

edited

Loading