Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DockerSize] Docker image size multi arch #8290

Merged
merged 28 commits into from
Aug 25, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
fdec6d0
Get the size of the docker image taking architecture into account
PaulaBarszcz Aug 6, 2022
2f08b69
Getting docker image size for certain architecture now works for both…
PaulaBarszcz Aug 6, 2022
8d12852
Docker image size now works also when tag is supplied
PaulaBarszcz Aug 6, 2022
c7ba732
Removed an unwanted change
PaulaBarszcz Aug 6, 2022
3aae8ce
Merge branch 'master' into docker-imageSize-multiArch
PaulaBarszcz Aug 6, 2022
966f499
Merge branch 'master' into docker-imageSize-multiArch
PaulaBarszcz Aug 9, 2022
bc769e0
updated schema; split transform into 3 functions
PaulaBarszcz Aug 9, 2022
26328b4
Created validDockerArchitectures and updated Joi validator
PaulaBarszcz Aug 9, 2022
59e1bcb
not yet finished attempts to DRY the getImageForArch function
PaulaBarszcz Aug 9, 2022
1cbbd86
Merge branch 'master' into docker-imageSize-multiArch
PaulaBarszcz Aug 10, 2022
95b4d9a
ufffff, finally works; before cleanup
PaulaBarszcz Aug 10, 2022
8e6b10a
before cleanup
PaulaBarszcz Aug 10, 2022
b89df91
after cleanup; seems to be working for all 3 paths
PaulaBarszcz Aug 10, 2022
ab8e4c2
updated the list of valid architectures
PaulaBarszcz Aug 11, 2022
26f1b03
docker-size tester is green
PaulaBarszcz Aug 11, 2022
0190817
add unit tests for noTagWithDateSortTransform()
PaulaBarszcz Aug 11, 2022
52af3d9
added unit tests for noTagWithSemverSortTransform() and yesTagTransfo…
PaulaBarszcz Aug 11, 2022
1f4e243
add missing service tests for docker-size service
PaulaBarszcz Aug 11, 2022
041fe8e
Merge branch 'master' into docker-imageSize-multiArch
PaulaBarszcz Aug 15, 2022
7fd11ea
Merge branch 'master' into docker-imageSize-multiArch
PaulaBarszcz Aug 22, 2022
a991278
part of PR-related corrections
PaulaBarszcz Aug 22, 2022
7c15604
part of PR-related corrections is done
PaulaBarszcz Aug 22, 2022
496e0b8
Merge branch 'master' into docker-imageSize-multiArch
PaulaBarszcz Aug 22, 2022
1fe7bf5
Merge branch 'docker-imageSize-multiArch' of https://github.com/Paula…
PaulaBarszcz Aug 22, 2022
18e79ac
modify archSchema for docker services
PaulaBarszcz Aug 25, 2022
60a0209
Update comments in docker-size service
PaulaBarszcz Aug 25, 2022
758ced9
Merge branch 'master' into docker-imageSize-multiArch
chris48s Aug 25, 2022
489619c
Merge branch 'master' into docker-imageSize-multiArch
chris48s Aug 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 32 additions & 67 deletions services/docker/docker-fixtures.js
Original file line number Diff line number Diff line change
@@ -1,71 +1,36 @@
const sizeDataNoTagSemVerSort = [
{ name: 'master', full_size: 13449470 },
{ name: 'feature-smtps-support', full_size: 13449638 },
{ name: 'latest', full_size: 13448411 },
{ name: '4', full_size: 13448411 },
{ name: '4.3', full_size: 13448411 },
{ name: '4.3.0', full_size: 13448411 },
{ name: '4.2', full_size: 13443674 },
{ name: '4.2.0', full_size: 13443674 },
{ name: '4.1', full_size: 19244435 },
{ name: '4.1.0', full_size: 19244435 },
{ name: 'v4.0.0-alpha2', full_size: 10933605 },
{ name: 'v4.0.0-alpha1', full_size: 10933644 },
{ name: '4.0.0', full_size: 11512227 },
{ name: '4.0', full_size: 11512227 },
{ name: 'v2.1.9', full_size: 29739490 },
{ name: 'v2.1.10', full_size: 29739842 },
{ name: 'v3.0.0', full_size: 32882980 },
{ name: 'v3.0.1', full_size: 32880923 },
{ name: 'v3.1.0', full_size: 32441549 },
{ name: 'v3.1.1', full_size: 32441767 },
{ name: 'v3.1.2', full_size: 32442741 },
{ name: 'v3.1.3', full_size: 32442629 },
{ name: 'v3.1.4', full_size: 32478607 },
{ name: 'v3.2.0', full_size: 33489914 },
{ name: 'v3.3.0', full_size: 33628545 },
{ name: 'v3.3.1', full_size: 33629018 },
{ name: 'v3.3.3', full_size: 33628988 },
{ name: 'v3.3.4', full_size: 33629019 },
{ name: 'v3.3.6', full_size: 33628753 },
{ name: 'v3.3.7', full_size: 33629556 },
{ name: 'v3.3.8', full_size: 33644261 },
{ name: 'v3.3.9', full_size: 33644175 },
{ name: 'v3.3.10', full_size: 33644406 },
{ name: 'v3.3.11', full_size: 33644430 },
{ name: 'v3.3.12', full_size: 33644703 },
{ name: 'v3.3.13', full_size: 33644377 },
{ name: 'v3.3.15', full_size: 33644581 },
{ name: 'v3.3.16', full_size: 33644663 },
{ name: 'v3.3.17', full_size: 33644228 },
{ name: 'v3.3.18', full_size: 33644466 },
{ name: 'v3.3.19', full_size: 33644724 },
{ name: 'v3.4.0', full_size: 34918552 },
{ name: 'v3.4.2', full_size: 33605129 },
{ name: 'v3.5.0', full_size: 33582915 },
{ name: 'v3.6.0', full_size: 34789944 },
{ name: 'develop', full_size: 38129308 },
{ name: 'v3.7.0', full_size: 38179583 },
{ name: 'v3.7.1', full_size: 38614944 },
{ name: 'v3.8.0', full_size: 42962384 },
{ name: 'v3.8.1', full_size: 40000713 },
{ name: 'v3.8.2', full_size: 40000567 },
{ name: 'v3.8.3', full_size: 40040963 },
{ name: 'v3.9.0', full_size: 40044357 },
{ name: 'v3.9.1', full_size: 40048123 },
{ name: 'v3.9.2', full_size: 40047663 },
{ name: 'v3.9.3', full_size: 40048204 },
{ name: 'v3.9.4', full_size: 40049571 },
{ name: 'v3.9.5', full_size: 40049695 },
{ name: 'v3.10.0', full_size: 39940736 },
{ name: 'v3.11.0', full_size: 39928170 },
{ name: 'v3.12.0', full_size: 39966770 },
{ name: 'v3.13.0', full_size: 38556045 },
{ name: 'v3.14.0', full_size: 38574008 },
{ name: 'v3.15.0', full_size: 38578507 },
{ name: 'v3.16.0', full_size: 38852598 },
{ name: 'v3.16.1', full_size: 38851702 },
{ name: 'v3.16.2', full_size: 38969822 },
{
full_size: 300000000,
name: 'v4.0.0-alpha2',
images: [
{ architecture: 'amd64', size: 219939484 },
{ architecture: 'arm64', size: 200000000 },
],
},
{
full_size: 400000000,
name: 'v4.2.4',
images: [
{ architecture: 'amd64', size: 220000000 },
{ architecture: 'arm64', size: 210000000 },
],
},
{
full_size: 100000000,
name: 'v3.9.7',
images: [
{ architecture: 'amd64', size: 120000000 },
{ architecture: 'arm64', size: 110000000 },
],
},
{
full_size: 500000000,
name: 'latest',
images: [
{ architecture: 'amd64', size: 560000000 },
{ architecture: 'arm64', size: 460000000 },
],
},
]
const versionDataNoTagDateSort = {
count: 4,
Expand Down
22 changes: 22 additions & 0 deletions services/docker/docker-helpers.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,28 @@
import Joi from 'joi'
// see https://github.com/badges/shields/pull/1690
import { NotFound } from '../index.js'
const dockerBlue = '066da5'

// Valid architecture values: https://golang.org/doc/install/source#environment (GOARCH)
const archSchema = Joi.alternatives(
Joi.string().valid(
'amd64',
'arm',
'arm64',
's390x',
'386',
'ppc64',
'ppc64le',
'wasm',
'mips',
'mipsle',
'mips64',
'mips64le',
'riscv64'
),
Joi.number().valid(386).cast('string')
)

function buildDockerUrl(badgeName, includeTagRoute) {
if (includeTagRoute) {
return {
Expand Down Expand Up @@ -55,6 +76,7 @@ function getDigestSemVerMatches({ data, digest }) {
}

export {
archSchema,
dockerBlue,
buildDockerUrl,
getDockerHubUser,
Expand Down
125 changes: 108 additions & 17 deletions services/docker/docker-size.service.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { nonNegativeInteger } from '../validators.js'
import { latest } from '../version.js'
import { BaseJsonService, NotFound } from '../index.js'
import {
archSchema,
buildDockerUrl,
getDockerHubUser,
getMultiPageData,
Expand All @@ -12,6 +13,12 @@ import {
const buildSchema = Joi.object({
name: Joi.string().required(),
full_size: nonNegativeInteger.required(),
images: Joi.array().items(
Joi.object({
size: nonNegativeInteger.required(),
architecture: Joi.string().required(),
})
),
}).required()

const pagedSchema = Joi.object({
Expand All @@ -20,14 +27,37 @@ const pagedSchema = Joi.object({
Joi.object({
name: Joi.string().required(),
full_size: nonNegativeInteger.required(),
images: Joi.array().items(
Joi.object({
size: nonNegativeInteger.required(),
architecture: Joi.string().required(),
})
),
})
),
}).required()

const queryParamSchema = Joi.object({
sort: Joi.string().valid('date', 'semver').default('date'),
arch: archSchema,
}).required()

// If user provided the arch parameter,
// check if any of the returned images has an architecture matching the arch parameter provided.
// If yes, return the size of the image with this arch.
// If not, throw the `NotFound` error.
// For details see: https://github.com/badges/shields/issues/8238
function getImageSizeForArch(images, arch) {
const imgWithArch = Object.values(images).find(
img => img.architecture === arch
)

if (!imgWithArch) {
throw new NotFound({ prettyMessage: 'architecture not found' })
}
return imgWithArch.size
}

export default class DockerSize extends BaseJsonService {
static category = 'size'
static route = { ...buildDockerUrl('image-size', true), queryParamSchema }
Expand All @@ -46,6 +76,14 @@ export default class DockerSize extends BaseJsonService {
queryParams: { sort: 'semver' },
staticPreview: this.render({ size: 136000000 }),
},
{
title:
'Docker Image Size with architecture (latest by date/latest semver)',
pattern: ':user/:repo',
namedParams: { user: 'library', repo: 'mysql' },
queryParams: { sort: 'date', arch: 'amd64' },
staticPreview: this.render({ size: 146000000 }),
},
{
title: 'Docker Image Size (tag)',
pattern: ':user/:repo/:tag',
Expand Down Expand Up @@ -73,30 +111,83 @@ export default class DockerSize extends BaseJsonService {
})
}

transform({ tag, sort, data }) {
if (!tag && sort === 'date') {
if (data.count === 0) {
throw new NotFound({ prettyMessage: 'repository not found' })
getSizeFromImageByLatestDate(data, arch) {
if (data.count === 0) {
throw new NotFound({ prettyMessage: 'repository not found' })
} else {
const latestEntry = data.results[0]

if (arch) {
return { size: getImageSizeForArch(latestEntry.images, arch) }
} else {
return { size: data.results[0].full_size }
return { size: latestEntry.full_size }
}
} else if (!tag && sort === 'semver') {
const [matches, versions] = data.reduce(
([m, v], d) => {
m[d.name] = d.full_size
v.push(d.name)
return [m, v]
},
[{}, []]
)
const version = latest(versions)
}
}

getSizeFromImageByLatestSemver(data, arch) {
// If no tag is specified, and sorting is by semver, first filter out the entry containing the latest semver from the response with Docker images.
// If no architecture is supplied by the user, return `full_size` from this entry.
// If the architecture is supplied by the user, check if any of the returned images for this entry has an architecture matching the arch parameter supplied by the user.
// If yes, return the size of the image with this arch.
// If not, throw the `NotFound` error.

const [matches, versions, images] = data.reduce(
([m, v, i], d) => {
m[d.name] = d.full_size
v.push(d.name)
i[d.name] = d.images
return [m, v, i]
},
[{}, [], {}]
)

const version = latest(versions)

let sizeOfImgWithArch

if (arch) {
Object.keys(images).forEach(ver => {
if (ver === version) {
sizeOfImgWithArch = getImageSizeForArch(images[ver], arch)
return { size: sizeOfImgWithArch }
}
})

if (sizeOfImgWithArch) {
return { size: sizeOfImgWithArch }
} else {
throw new NotFound({ prettyMessage: 'architecture not found' })
}
} else {
return { size: matches[version] }
}
}

getSizeFromTag(data, arch) {
// If the tag is specified, and the architecture is supplied by the user,
// check if any of the returned images has an architecture matching the arch parameter supplied by the user.
// If yes, return the size of the image with this arch.
// If no, throw the `NotFound` error.
// If no architecture is supplied by the user, return the value of the `full_size` from the response (the image with the `latest` tag).
if (arch) {
return { size: getImageSizeForArch(data.images, arch) }
} else {
return { size: data.full_size }
}
}

async handle({ user, repo, tag }, { sort }) {
transform({ tag, sort, data, arch }) {
if (!tag && sort === 'date') {
return this.getSizeFromImageByLatestDate(data, arch)
} else if (!tag && sort === 'semver') {
return this.getSizeFromImageByLatestSemver(data, arch)
} else {
return this.getSizeFromTag(data, arch)
}
}

async handle({ user, repo, tag }, { sort, arch }) {
let data

if (!tag && sort === 'date') {
Expand All @@ -111,7 +202,7 @@ export default class DockerSize extends BaseJsonService {
data = await this.fetch({ user, repo, tag })
}

const { size } = await this.transform({ tag, sort, data })
const { size } = await this.transform({ tag, sort, data, arch })
return this.constructor.render({ size })
}
}
Loading