[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs #7430

jmagman · 2024-08-16T17:34:33Z

Native Xcode tests will output a helpful "xcresult" package on failure containing logs, screenshots, and screen recordings. Zip and upload these results when tests fail.
Pass flutter test --debug-logs-dir flag to upload logs like Upload DerivedData logs in CI flutter#142643.
Pass flutter drive --screenshot flag to upload screenshots on timeout like Take drive screenshot on test failure before app is stopped flutter#96973.

Example of failing Xcode analyzer build has the zipped xcresult attached as a log.

The unzipped xcresult looks like this in Xcode:

A failing "native test" step build:

Fixes flutter/flutter#144795

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

stuartmorgan · 2024-10-08T19:40:57Z

From triage: What's the status of this? Is it still something we plan to deploy? It would definitely be useful for debugging!

jmagman · 2024-10-24T23:58:18Z

From triage: What's the status of this? Is it still something we plan to deploy? It would definitely be useful for debugging!

Sorry, I lost track of this. I need to page back in why the tests are failing.

This reverts commit d329678.

jmagman · 2024-10-29T17:04:48Z

@stuartmorgan this is ready to review, when you get a chance.

stuartmorgan

LGTM

Is the FLUTTER_LOGS_DIR support low-level enough that it already applies to this repo, or do we need recipe changes too?

script/tool/lib/src/common/xcode.dart

stuartmorgan · 2024-10-31T14:22:39Z

script/tool/lib/src/drive_examples_command.dart

+      Directory? screenshotDirectory;
+      if (logsDirectory != null) {
+        screenshotDirectory = logsDirectory.childDirectory('$exampleName-drive');
+      }


Directory? screenshotDirectory = logsDirectory?.childDirectory(...);

Oops, I copied code that was written before null safety and clumsily migrated (likely by me).

stuartmorgan · 2024-10-31T14:26:52Z

script/tool/lib/src/drive_examples_command.dart


    for (final File target in targets) {
+      Directory? screenshotDirectory;
+      if (logsDirectory != null) {
+        screenshotDirectory = logsDirectory.childDirectory('$exampleName-drive');


exampleName is a relative posix path; IIRC childDirectory will interpret that as a path and actually make intervening directories. Is that what we want? If not we should replace / with _ or something before this.

stuartmorgan · 2024-10-31T14:30:24Z

script/tool/lib/src/drive_examples_command.dart

@@ -416,6 +428,7 @@ class DriveExamplesCommand extends PackageLoopingCommand {
    required List<File> testFiles,
  }) async {
    final String enableExperiment = getStringArg(kEnableExperiment);
+    final Directory? logsDirectory = testFiles.isNotEmpty ? ciLogsDirectory(platform, testFiles.first.fileSystem) : null;


Why is this gated on testFiles not being empty? (I'd have to check but I wouldn't think this would ever be called with an empty test list.)

I think I was just guarding the .first, I'll remove since I think you're right it's never called with an empty list.

jmagman · 2024-11-04T19:51:12Z

Is the FLUTTER_LOGS_DIR support low-level enough that it already applies to this repo, or do we need recipe changes too?

It already applies as of https://flutter-review.googlesource.com/c/recipes/+/39703

You can see it in the env:
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8732173142577412545/+/u/Run_package_tests/xcode_analyze_deprecation/l_execution_details

…rive and integration_test runs (flutter/packages#7430)

flutter/packages@7219431...bb5a258 2024-11-06 magder@google.com [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter/packages#7430) 2024-11-05 737941+loic-sharma@users.noreply.github.com Remove use_modular_headers! from Podfiles (flutter/packages#7796) 2024-11-05 30872003+misos1@users.noreply.github.com [camera_avfoundation] enable more than 30 fps (flutter/packages#7394) 2024-11-05 engine-flutter-autoroll@skia.org Roll Flutter from 8591d0c to 29d40f7 (25 revisions) (flutter/packages#8027) 2024-11-05 stuartmorgan@google.com [ci] Add vector_graphics and flutter_svg to autolabeler (flutter/packages#8025) 2024-11-05 alex@mariuti.com [vector_graphics_compiler] wasm compatibility (flutter/packages#8021) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

* main: (1187 commits) [various] Update example app minSdkVersions (flutter#8035) [go_router] Activate leak testing (flutter#7546) [in_app_purchase_storekit] Add restore purchases and receipts (flutter#7964) [interactive_media_ads] Adds remaining methods for internal wrapper of the Android native `BaseManager` (flutter#7943) [google_sign_in/google_identity_services] Clear-up documentation of callbacks in various APIs and uses of those APIs (flutter#8029) [flutter_svg] wasm compatibility (flutter#8014) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (Part 2) (flutter#8019) Roll Flutter from 29d40f7f6826 to 73546b3b71a7 (20 revisions) (flutter#8028) [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs (flutter#7430) Remove use_modular_headers! from Podfiles (flutter#7796) [camera_avfoundation] enable more than 30 fps (flutter#7394) Roll Flutter from 8591d0c16a6c to 29d40f7f6826 (25 revisions) (flutter#8027) [ci] Add vector_graphics and flutter_svg to autolabeler (flutter#8025) [vector_graphics_compiler] wasm compatibility (flutter#8021) [vector_graphics*] Relax dependency constraints of vector_graphics, vector_graphics_codec, vector_graphics_compiler, flutter_svg (flutter#8018) [various] Add `missing_code_block_language_in_doc_comment` lint to flutter/packages. (flutter#6473) [various] Update example apps to Kotlin 1.9.0 (flutter#7998) [go_router] add current state getter (flutter#7651) Applied Gradle Plugins Declaratively for Multiple Plugin Example Apps (flutter#7968) Roll Flutter from f86b77721524 to 8591d0c16a6c (16 revisions) (flutter#8015) ... # Conflicts: # packages/quick_actions/quick_actions/CHANGELOG.md # packages/quick_actions/quick_actions_ios/CHANGELOG.md # packages/quick_actions/quick_actions_platform_interface/CHANGELOG.md

jmagman marked this pull request as ready for review October 25, 2024 16:11

jmagman marked this pull request as draft October 25, 2024 16:11

jmagman force-pushed the screenshot-drive branch from 2a21d04 to d329678 Compare October 25, 2024 16:48

github-actions bot added platform-ios platform-macos p: video_player labels Oct 25, 2024

jmagman added 3 commits October 25, 2024 10:22

Add screenshot functionality to drive

f39f734

test failures

8ac52ff

Revert "test failures"

54003a8

This reverts commit d329678.

jmagman force-pushed the screenshot-drive branch from d329678 to 54003a8 Compare October 25, 2024 17:23

github-actions bot removed platform-ios platform-macos p: video_player labels Oct 25, 2024

jmagman changed the title ~~Upload screenshots and Xcode test results for drive and integration_test runs~~ [ci] Upload screenshots and Xcode test results for drive and integration_test runs Oct 25, 2024

jmagman self-assigned this Oct 25, 2024

jmagman marked this pull request as ready for review October 25, 2024 17:24

jmagman changed the title ~~[ci] Upload screenshots and Xcode test results for drive and integration_test runs~~ [ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs Oct 25, 2024

jmagman requested a review from stuartmorgan October 25, 2024 21:10

stuartmorgan approved these changes Oct 31, 2024

View reviewed changes

jmagman added 5 commits November 4, 2024 21:34

Review edits

937765c

path separator

7e6e02c

deletesync

885a407

Merge branch 'main' into screenshot-drive

5719da2

New formatter

a45f634

jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 6, 2024

auto-submit bot merged commit bb5a258 into flutter:main Nov 6, 2024
76 checks passed

jmagman deleted the screenshot-drive branch November 6, 2024 06:58

engine-flutter-autoroll mentioned this pull request Nov 6, 2024

Roll Packages from 721943194945 to bb5a25815cae (6 revisions) flutter/flutter#158267

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 6, 2024

bb5a25815 [ci] Upload screenshots, logs, and Xcode test results for d…

16cad07

…rive and integration_test runs (flutter/packages#7430)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs #7430

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs #7430

jmagman commented Aug 16, 2024 •

edited

Loading

stuartmorgan commented Oct 8, 2024

jmagman commented Oct 24, 2024

jmagman commented Oct 29, 2024

stuartmorgan left a comment

stuartmorgan Oct 31, 2024

jmagman Nov 5, 2024

stuartmorgan Oct 31, 2024

jmagman Nov 6, 2024

stuartmorgan Oct 31, 2024

jmagman Nov 5, 2024

jmagman commented Nov 4, 2024

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs #7430

[ci] Upload screenshots, logs, and Xcode test results for drive and integration_test runs #7430

Conversation

jmagman commented Aug 16, 2024 • edited Loading

Pre-launch Checklist

stuartmorgan commented Oct 8, 2024

jmagman commented Oct 24, 2024

jmagman commented Oct 29, 2024

stuartmorgan left a comment

Choose a reason for hiding this comment

stuartmorgan Oct 31, 2024

Choose a reason for hiding this comment

jmagman Nov 5, 2024

Choose a reason for hiding this comment

stuartmorgan Oct 31, 2024

Choose a reason for hiding this comment

jmagman Nov 6, 2024

Choose a reason for hiding this comment

stuartmorgan Oct 31, 2024

Choose a reason for hiding this comment

jmagman Nov 5, 2024

Choose a reason for hiding this comment

jmagman commented Nov 4, 2024

jmagman commented Aug 16, 2024 •

edited

Loading