Skip to content

Commit

Permalink
misc: persist upload timings to cy cloud (#28418)
Browse files Browse the repository at this point in the history 
* feat: report artifact upload durations to cy cloud for monitoring

* round upload durations to nearest ms

* update changelog

* revert changelog, fix typo in comment

* no longer round uploadDuration - api now accepts floats

* updates changelog

* update api spec for refactored signature on api.updateInstanceArtifacts

* Update CHANGELOG.md

* Update system-tests/test/record_spec.js

Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>

* rm defunct comment

* rm non-null accessors, this spec is not ts

---------

Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
  • Loading branch information
@cacieprins @chrisbreiding
cacieprins and chrisbreiding authored Nov 29, 2023
1 parent 6a895d2 commit 7ba92b9
Show file tree
Hide file tree
Showing 7 changed files with 137 additions and 124 deletions.
4 changes: 4 additions & 0 deletions cli/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ _Released 12/5/2023 (PENDING)_
- Fixed an issue where pages or downloads opened in a new tab were missing basic auth headers. Fixes [#28350](https://github.com/cypress-io/cypress/issues/28350).
- Fixed an issue where request logging would default the `message` to the `args` of the currently running command even though those `args` would not apply to the request log and are not displayed. If the `args` are sufficiently large (e.g. when running the `cy.task` from the [code-coverage](https://github.com/cypress-io/code-coverage/) plugin) there could be performance/memory implications. Addressed in [#28411](https://github.com/cypress-io/cypress/pull/28411).

**Misc:**

- Artifact upload duration is now reported to Cypress Cloud. Addressed in [#28418](https://github.com/cypress-io/cypress/pull/28418)

## 13.6.0

_Released 11/21/2023_
Expand Down
47 changes: 23 additions & 24 deletions packages/server/lib/cloud/api.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -274,28 +274,31 @@ type CreateRunResponse = {
} | undefined
}

type ArtifactMetadata = {
url: string
fileSize?: number
uploadDuration?: number
success: boolean
error?: string
}

type ProtocolMetadata = ArtifactMetadata & {
specAccess?: {
size: bigint
offset: bigint
}
}

type UpdateInstanceArtifactsPayload = {
screenshots: ArtifactMetadata[]
video?: ArtifactMetadata
protocol?: ProtocolMetadata
}

type UpdateInstanceArtifactsOptions = {
runId: string
instanceId: string
timeout: number | undefined
protocol: {
url: string
success: boolean
fileSize?: number | undefined
error?: string | undefined
} | undefined
screenshots: {
url: string
success: boolean
fileSize?: number | undefined
error?: string | undefined
}[] | undefined
video: {
url: string
success: boolean
fileSize?: number | undefined
error?: string | undefined
} | undefined
}

let preflightResult = {
Expand Down Expand Up @@ -497,17 +500,13 @@ module.exports = {
})
},

updateInstanceArtifacts (options: UpdateInstanceArtifactsOptions) {
updateInstanceArtifacts (options: UpdateInstanceArtifactsOptions, body: UpdateInstanceArtifactsPayload) {
return retryWithBackoff((attemptIndex) => {
return rp.put({
url: recordRoutes.instanceArtifacts(options.instanceId),
json: true,
timeout: options.timeout ?? SIXTY_SECONDS,
body: {
protocol: options.protocol,
screenshots: options.screenshots,
video: options.video,
},
body,
headers: {
'x-route-version': '1',
'x-cypress-run-id': options.runId,
Expand Down
15 changes: 7 additions & 8 deletions packages/server/lib/modes/record.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,7 @@ const uploadArtifactBatch = async (artifacts, protocolManager, quiet) => {
url: artifact.uploadUrl,
fileSize: artifact.fileSize,
key: artifact.reportKey,
duration: performance.now() - startTime,
uploadDuration: performance.now() - startTime,
}
}

Expand All @@ -289,7 +289,7 @@ const uploadArtifactBatch = async (artifacts, protocolManager, quiet) => {
pathToFile: artifact.filePath,
fileSize: artifact.fileSize,
key: artifact.reportKey,
duration: performance.now() - startTime,
uploadDuration: performance.now() - startTime,
}
} catch (err) {
debug('failed to upload artifact %o', {
Expand All @@ -308,7 +308,7 @@ const uploadArtifactBatch = async (artifacts, protocolManager, quiet) => {
allErrors: err.errors,
url: artifact.uploadUrl,
pathToFile: artifact.filePath,
duration: performance.now() - startTime,
uploadDuration: performance.now() - startTime,
}
}

Expand All @@ -318,7 +318,7 @@ const uploadArtifactBatch = async (artifacts, protocolManager, quiet) => {
error: err.message,
url: artifact.uploadUrl,
pathToFile: artifact.filePath,
duration: performance.now() - startTime,
uploadDuration: performance.now() - startTime,
}
}
}),
Expand Down Expand Up @@ -355,8 +355,7 @@ const uploadArtifactBatch = async (artifacts, protocolManager, quiet) => {
return skipped && !report.error ? acc : {
...acc,
[key]: {
// TODO: once cloud supports reporting duration, no longer omit this
..._.omit(report, 'duration'),
...report,
error,
},
}
Expand Down Expand Up @@ -452,8 +451,8 @@ const uploadArtifacts = async (options = {}) => {
try {
debug('upload reprt: %O', uploadReport)
const res = await api.updateInstanceArtifacts({
runId, instanceId, ...uploadReport,
})
runId, instanceId,
}, uploadReport)

return res
} catch (err) {
Expand Down
8 changes: 4 additions & 4 deletions packages/server/lib/util/print-run.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -614,11 +614,11 @@ type ArtifactUploadResultLike = {
success: boolean
error?: string
skipped?: boolean
duration?: number
uploadDuration?: number
}

export const printCompletedArtifactUpload = <T extends ArtifactUploadResultLike> (artifactUploadResult: T, labels: Record<'protocol' | 'screenshots' | 'video', string>, num: string): void => {
const { pathToFile, key, fileSize, success, error, skipped, duration } = artifactUploadResult
const { pathToFile, key, fileSize, success, error, skipped, uploadDuration } = artifactUploadResult

process.stdout.write(` - ${labels[key]} `)

Expand All @@ -630,8 +630,8 @@ export const printCompletedArtifactUpload = <T extends ArtifactUploadResultLike>
process.stdout.write(`- Failed Uploading`)
}

if (duration) {
const durationOut = humanTime.short(duration, 2)
if (uploadDuration) {
const durationOut = humanTime.short(uploadDuration, 2)

process.stdout.write(` ${success ? 'in' : 'after'} ${durationOut}`)
}
Expand Down
12 changes: 9 additions & 3 deletions packages/server/test/unit/cloud/api_spec.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1524,20 +1524,26 @@ describe('lib/cloud/api', () => {

context('.updateInstanceArtifacts', () => {
beforeEach(function () {
this.artifactProps = {
this.artifactOptions = {
runId: 'run-id-123',
instanceId: 'instance-id-123',
}

this.artifactProps = {
screenshots: [{
url: `http://localhost:1234/screenshots/upload/instance-id-123/a877e957-f90e-4ba4-9fa8-569812f148c4.png`,
uploadSize: 100,
uploadDuration: 100,
}],
video: {
url: `http://localhost:1234/video/upload/instance-id-123/f17754c4-581d-4e08-a922-1fa402f9c6de.mp4`,
uploadSize: 122,
uploadDuration: 100,
},
protocol: {
url: `http://localhost:1234/protocol/upload/instance-id-123/2ed89c81-e7eb-4b97-8a6e-185c410471df.db`,
uploadSize: 123,
uploadDuration: 100,
},
}
// TODO: add schema validation
Expand All @@ -1546,7 +1552,7 @@ describe('lib/cloud/api', () => {
it('PUTs/instances/:id/artifacts', function () {
nock(API_BASEURL)
.matchHeader('x-route-version', '1')
.matchHeader('x-cypress-run-id', this.artifactProps.runId)
.matchHeader('x-cypress-run-id', this.artifactOptions.runId)
.matchHeader('x-cypress-request-attempt', '0')
.matchHeader('x-os-name', 'linux')
.matchHeader('x-cypress-version', pkg.version)
Expand All @@ -1557,7 +1563,7 @@ describe('lib/cloud/api', () => {
})
.reply(200)

return api.updateInstanceArtifacts(this.artifactProps)
return api.updateInstanceArtifacts(this.artifactOptions, this.artifactProps)
})
})
})
168 changes: 84 additions & 84 deletions system-tests/__snapshots__/record_spec.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2844,90 +2844,6 @@ exports['e2e record capture-protocol disabled messaging displays disabled messag
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12
`

exports['e2e record capture-protocol enabled passing retrieves the capture protocol and uploads the db 1'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: false, Group: false, Parallel: false │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: record_pass.cy.js (1 of 1)
Estimated: X second(s)
record pass
✓ passes
- is pending
1 passing
1 pending
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 1 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: X seconds │
│ Estimated: X second(s) │
│ Spec Ran: record_pass.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Screenshots)
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
(Uploading Cloud Artifacts)
- Video - Nothing to upload
- Screenshot - 1 kB /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - 1 kB
Uploading Cloud Artifacts: . . . . .
(Uploaded Cloud Artifacts)
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 2/2
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ record_pass.cy.js XX:XX 2 1 - 1 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 2 1 - 1 -
───────────────────────────────────────────────────────────────────────────────────────────────────────
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12
`

exports['e2e record capture-protocol enabled when the tab crashes in chrome posts accurate test results 1'] = `
Expand Down Expand Up @@ -3904,3 +3820,87 @@ exports['capture-protocol api errors error report 500 continues 1'] = `
`

exports['e2e record capture-protocol enabled passing retrieves the capture protocol, uploads the db, and updates the artifact upload report 1'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: false, Group: false, Parallel: false │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: record_pass.cy.js (1 of 1)
Estimated: X second(s)
record pass
✓ passes
- is pending
1 passing
1 pending
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 1 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: X seconds │
│ Estimated: X second(s) │
│ Spec Ran: record_pass.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Screenshots)
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
(Uploading Cloud Artifacts)
- Video - Nothing to upload
- Screenshot - 1 kB /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - 1 kB
Uploading Cloud Artifacts: . . . . .
(Uploaded Cloud Artifacts)
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 2/2
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ record_pass.cy.js XX:XX 2 1 - 1 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 2 1 - 1 -
───────────────────────────────────────────────────────────────────────────────────────────────────────
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12
`
Loading

4 comments on commit 7ba92b9

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on 7ba92b9 Nov 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the linux x64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.6.1/linux-x64/develop-7ba92b91e5131252438c6beb413cc5cca971e249/cypress.tgz

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on 7ba92b9 Nov 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the darwin x64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.6.1/darwin-x64/develop-7ba92b91e5131252438c6beb413cc5cca971e249/cypress.tgz

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on 7ba92b9 Nov 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the darwin arm64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.6.1/darwin-arm64/develop-7ba92b91e5131252438c6beb413cc5cca971e249/cypress.tgz

@cypress-bot
Copy link
Contributor

@cypress-bot cypress-bot bot commented on 7ba92b9 Nov 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Circle has built the win32 x64 version of the Test Runner.

Learn more about this pre-release build at https://on.cypress.io/advanced-installation#Install-pre-release-version

Run this command to install the pre-release locally:

npm install https://cdn.cypress.io/beta/npm/13.6.1/win32-x64/develop-7ba92b91e5131252438c6beb413cc5cca971e249/cypress.tgz

Please sign in to comment.