added multiarch build #6577

AndrewChubatiuk · 2023-11-06T11:07:17Z

What type of PR is this?

Build multiarch docker images

Description

Added ability to create multiarch images

How is this tested?

Unit tests (pytest, jest)
E2E Tests (Cypress)
[] Manually
N/A

guidopetri · 2023-11-06T11:55:06Z

Hey @AndrewChubatiuk ! Thanks for the PR. We've tried a arm64 build on GHA recently and it runs super long - so long, in fact, that it hits some hidden 6 hour limit for GHA and does not finish. Would you mind testing this on your own repo and potentially making changes there before we consider merging this in?

AndrewChubatiuk · 2023-11-06T11:58:52Z

this change will run arm64 and amd64 image builds on separate machines, so it should improve build speed. okay, will test it in my fork

guidopetri · 2023-11-06T12:02:56Z

Here's an example action that we already ran: https://github.com/getredash/redash/actions/runs/6546702269

I'm not sure how that was running so maybe we just did it wrong. :)

guidopetri · 2023-11-07T02:14:23Z

Dockerfile

@@ -1,7 +1,5 @@
 FROM node:16.20.1-bookworm as frontend-builder

-RUN npm install --global --force yarn@1.22.19


Could you explain the changes you're making to the Dockerfile here?

Removed frontend build from Dockerfile as there are to many places, where same node dependencies are installed. Also there's no sense to build same platform independent frontend in many places

Dockerfile

guidopetri · 2023-11-07T02:15:33Z

client/.babelrc

@@ -4,7 +4,7 @@
      "@babel/preset-env",
      {
        "exclude": ["@babel/plugin-transform-async-to-generator", "@babel/plugin-transform-arrow-functions"],
-        "corejs": "2",
+        "corejs": "3",


I don't understand any of the JS dependency related changes; @eradman would you mind taking a look? Are we able to change these deps to enable arm64 images?

reverted frontend changes, as they are built separately now

Dockerfile

AndrewChubatiuk · 2023-11-07T18:00:45Z

Here's an example action that we already ran: https://github.com/getredash/redash/actions/runs/6546702269

I'm not sure how that was running so maybe we just did it wrong. :)

@guidopetri Finished doing my changes. you can check build time here
https://github.com/AndrewChubatiuk/redash/actions/runs/6786331640

There's lot's of work still, lot's of python packages which are either not supported or outdated, do not have arm64 wheels. Making them up to date can reduce build time as well

Also node dependencies for cypress can be optimized, cause it also takes several minutes to install them inside container

AndrewChubatiuk · 2023-11-09T15:57:31Z

@guidopetri @eradman
Are there any additional changes needed?

eradman · 2023-11-09T20:48:37Z

This change seems to check out for me. Triggering CI tests

codecov · 2023-11-09T20:55:55Z

Codecov Report

Merging #6577 (1dda186) into master (8bfc574) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #6577   +/-   ##
=======================================
  Coverage   61.89%   61.89%           
=======================================
  Files         158      158           
  Lines       12992    12992           
  Branches     1774     1774           
=======================================
  Hits         8042     8042           
  Misses       4673     4673           
  Partials      277      277

eradman · 2023-11-09T23:46:14Z

The code coverage report is not relavent.

@guidopetri can you re-test a build with these changes on arm64?

guidopetri · 2023-11-11T01:03:17Z

.github/workflows/ci.yml

-          npm install --global --force yarn@1.22.19
-          yarn cache clean && yarn --frozen-lockfile --network-concurrency 1
-
+          name: frontend


I'm not 100% clear where this frontend gets used in the build. Would you mind explaining? Does this also get pushed to Dockerhub as part of the preview image?

@guidopetri frontend is added to docker image during a build for each target arch and then first pushed to redash/preview and then copied from redash/preview to redash/redash

How do we build this locally, then? docker build -t redash_test . seems to not build the frontend

guidopetri

I'm unclear on what these changes are doing, and I think we need better documentation around this. How do we build things locally? What exactly is happening in the GHA build process? Why do we need to split up the frontend build?

AndrewChubatiuk · 2023-11-23T10:13:49Z

I'm unclear on what these changes are doing, and I think we need better documentation around this. How do we build things locally? What exactly is happening in the GHA build process? Why do we need to split up the frontend build?

github actions is using amd64 runners even for arm64 builds, so moving frontend out of docker images makes build faster
to build an image locally is a two step procedure, which requires to build frontend and then build a docker image (i can make a separate dockerfile for CI or add a make action to run a single action for a build)
during a build process frontend is built outside a docker images, cause it's platform independent and then docker copies it inside in image during a build of docker image for each platform

guidopetri · 2023-11-24T22:54:47Z

Hmm, alright. I think we should separate this into two PRs then:

one enabling ARM64 builds (which may be slow if they're building the frontend);
one separating the frontend build out, and making it clear how to build this locally. We should be able to run a docker command for this, and not depend on e.g. yarn being installed locally.

Does this make sense to you too? Or how do you think we should proceed?

AndrewChubatiuk · 2024-04-10T09:53:35Z

merged alternative PR in #6674

guidopetri reviewed Nov 7, 2023

View reviewed changes

multiarch build

0b64109

AndrewChubatiuk force-pushed the multiarch-build branch from e5c84c7 to 0b64109 Compare November 7, 2023 08:29

Andrew Chubatiuk added 6 commits November 7, 2023 11:27

added db init

e644ecd

added db init

b4b2c10

minor fix

8391103

minor fix

ccb5315

fixed var name, moved version to manifest section

607879c

minor fix

ff29cae

eradman reviewed Nov 7, 2023

View reviewed changes

Dockerfile Show resolved Hide resolved

disabled databricks odbc for arm64, removed fork specific ci configs

15e6a6e

guidopetri reviewed Nov 11, 2023

View reviewed changes

Merge branch 'master' into multiarch-build

1dda186

guidopetri requested changes Nov 19, 2023

View reviewed changes

AndrewChubatiuk closed this Apr 10, 2024

AndrewChubatiuk deleted the multiarch-build branch April 10, 2024 09:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added multiarch build #6577

added multiarch build #6577

AndrewChubatiuk commented Nov 6, 2023 •

edited

Loading

guidopetri commented Nov 6, 2023

AndrewChubatiuk commented Nov 6, 2023

guidopetri commented Nov 6, 2023

guidopetri Nov 7, 2023

AndrewChubatiuk Nov 7, 2023

guidopetri Nov 7, 2023

AndrewChubatiuk Nov 7, 2023

AndrewChubatiuk commented Nov 7, 2023 •

edited

Loading

AndrewChubatiuk commented Nov 9, 2023

eradman commented Nov 9, 2023

codecov bot commented Nov 9, 2023 •

edited

Loading

eradman commented Nov 9, 2023

guidopetri Nov 11, 2023

AndrewChubatiuk Nov 11, 2023

guidopetri Nov 19, 2023

guidopetri left a comment

AndrewChubatiuk commented Nov 23, 2023

guidopetri commented Nov 24, 2023

AndrewChubatiuk commented Apr 10, 2024

		@@ -1,7 +1,5 @@
		FROM node:16.20.1-bookworm as frontend-builder

		RUN npm install --global --force yarn@1.22.19

added multiarch build #6577

added multiarch build #6577

Conversation

AndrewChubatiuk commented Nov 6, 2023 • edited Loading

What type of PR is this?

Description

How is this tested?

guidopetri commented Nov 6, 2023

AndrewChubatiuk commented Nov 6, 2023

guidopetri commented Nov 6, 2023

guidopetri Nov 7, 2023

Choose a reason for hiding this comment

AndrewChubatiuk Nov 7, 2023

Choose a reason for hiding this comment

guidopetri Nov 7, 2023

Choose a reason for hiding this comment

AndrewChubatiuk Nov 7, 2023

Choose a reason for hiding this comment

AndrewChubatiuk commented Nov 7, 2023 • edited Loading

AndrewChubatiuk commented Nov 9, 2023

eradman commented Nov 9, 2023

codecov bot commented Nov 9, 2023 • edited Loading

Codecov Report

eradman commented Nov 9, 2023

guidopetri Nov 11, 2023

Choose a reason for hiding this comment

AndrewChubatiuk Nov 11, 2023

Choose a reason for hiding this comment

guidopetri Nov 19, 2023

Choose a reason for hiding this comment

guidopetri left a comment

Choose a reason for hiding this comment

AndrewChubatiuk commented Nov 23, 2023

guidopetri commented Nov 24, 2023

AndrewChubatiuk commented Apr 10, 2024

AndrewChubatiuk commented Nov 6, 2023 •

edited

Loading

AndrewChubatiuk commented Nov 7, 2023 •

edited

Loading

codecov bot commented Nov 9, 2023 •

edited

Loading