custom history name duplicates history?

[rpappalax]

Parallel test execution does save time, but ideally we'd also be able to execute in a parallel across more than one device type. From what I see in flank.yml, it's currently only possibly to execute on one device at a time. Is that correct?

[bootstraponline]

Nope, multiple devices are fully supported. Try it and let me know :)

Here's an example:

https://github.com/TestArmada/flank/blob/master/test_runner/src/test/kotlin/ftl/fixtures/flank.gcs.yml

[rpappalax]

thanks so much @bootstraponline. unfortunately not working for me, but perhaps I'm missing something?? I have 31 tests I'm executing across 2 devices, but flank skips the first device listed and only executes 31 tests (on 31 shards) against 1 device. Afterwards it outputs the results and quits

# https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
gcloud:
  results-bucket: tmp_flank_rpapa 
  record-video: true
  timeout: 60m
  async: false
  project: my-gnarly-proj

  # RPX 
  results-history-name: tmp_parallel 

  # test and app are the only required args
  app: ./app/build/outputs/apk/demoX86/debug/app-demo-x86-debug.apk
  test: ./app/build/outputs/apk/androidTest/demoX86/debug/app-demo-x86-debug-androidTest.apkdemo
  auto-google-login: true
  use-orchestrator: true
  environment-variables:
    clearPackageData: true
  directories-to-pull:
    - /sdcard/screenshots
  performance-metrics: true

  test-targets:
    - class org.rpappalax.demo.activity.AccessAboutAndYourRightsPagesTest#AboutAndYourRightsPagesTest
    - class org.rpappalax.demo.activity.AccessSettingsTest#AccessSettingsTest
    - class org.rpappalax.demo.activity.AddToHomescreenTest#AddToHomeScreenTest
    - class org.rpappalax.demo.activity.AddToHomescreenTest#NonameTest
    - class org.rpappalax.demo.activity.AddToHomescreenTest#SearchTermShortcutTest
    - class org.rpappalax.demo.activity.BadURLTest#BadURLcheckTest
    - class org.rpappalax.demo.activity.ChangeSearchEngineTest#SearchTest
    - class org.rpappalax.demo.activity.CustomTabTest#testCustomTabUI
    - class org.rpappalax.demo.activity.DownloadFileTest#DownloadTest
    - class org.rpappalax.demo.activity.EraseAllUserDataTest#TrashTest
    - class org.rpappalax.demo.activity.EraseAllUserDataTest#systemBarTest
    - class org.rpappalax.demo.activity.EraseAllUserDataTest#systemBarHomeViewTest
    - class org.rpappalax.demo.activity.FirstRunDialogueTest#FirstRunDialogueTest
    - class org.rpappalax.demo.activity.ImageSelectTest#ImageMenuTest
    - class org.rpappalax.demo.activity.ImageSelectTest#ShareImageTest
    - class org.rpappalax.demo.activity.ImageSelectTest#DownloadImageMenuTest
    - class org.rpappalax.demo.activity.MultitaskingTest#testVisitingMultipleSites
    - class org.rpappalax.demo.activity.OpenInExternalBrowserDialogueTest#OpenInExternalBrowserDialogueTest
    - class org.rpappalax.demo.activity.PullDownToRefreshTest#pullDownToRefreshTest
    - class org.rpappalax.demo.activity.SettingsBlockToggleTest#SettingsToggleTest
    - class org.rpappalax.demo.activity.ShareWebsiteTest#ShareWebsiteTest
    - class org.rpappalax.demo.activity.SwitchContextTest#ForegroundTest
    - class org.rpappalax.demo.activity.SwitchContextTest#eraseAndOpenTest
    - class org.rpappalax.demo.activity.SwitchLocaleTest#EnglishSystemLocaleTest
    - class org.rpappalax.demo.activity.SwitchLocaleTest#FrenchLocaleTest
    - class org.rpappalax.demo.activity.ToggleBlockTest#SimpleToggleTest
    - class org.rpappalax.demo.activity.URLAutocompleteTest#CompletionTest
    - class org.rpappalax.demo.activity.URLAutocompleteTest#CustomCompletionTest
    - class org.rpappalax.demo.activity.URLAutocompleteTest#DisableCCwithSiteTest
    - class org.rpappalax.demo.activity.URLAutocompleteTest#DuplicateACSiteTest
    - class org.rpappalax.demo.activity.URLMismatchTest#MismatchTest

  device:
    - model: Nexus6P 
      version: 23 

  device:
    - model: Pixel2 
      version: 26 

flank:
  # test shards - the amount of groups to split the test suite into
  # set to -1 to use one shard per test.
  testShards: -1 
  # repeat tests - the amount of times to run the tests.
  # 1 runs the tests once. 10 runs all the tests 10x
  repeatTests: 1
  # always run - these tests are inserted at the beginning of every shard
  # useful if you need to grant permissions or login before other tests run
  #test-targets-always-run:
    #- class com.example.app.ExampleUiTest#testPasses
    # - class org.rpappalax.demo.activty.SwitchContextTest#testPasses

[rpappalax]

Also, for some reason the tests show up in the Firebase console in 3 separate groups all with the same "results_history_name"

one tab shows 28 tests, another 2 and the last one 1 test respectivily 🤔

[bootstraponline]

hey, you are using the wrong yaml format.

try this:

 device:
    - model: Nexus6P 
      version: 23 
    - model: Pixel2 
      version: 26 

[bootstraponline]

also join https://firebase.community #flank channel, will be faster to discuss in real time.

[bootstraponline]

delete results-history-name: tmp_parallel from your yaml as well, then FTL will use the app default name.

[bootstraponline]

discussed on slack:

• multi-device is working
• custom history name appears to duplicate the history?

[bootstraponline]

talked with FTL, history name duplicate is a feature. web UI may be revised in the future to show the underlying package name in addition to the history name.

[bootstraponline]

reopening... looks like duplicates for the same package name are indeed a potential bug.