Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement wda/stf auto healing #151

Closed
vdelendik opened this issue Dec 24, 2022 · 6 comments
Closed

implement wda/stf auto healing #151

vdelendik opened this issue Dec 24, 2022 · 6 comments
Labels
enhancement New feature or request
Milestone

Comments

@vdelendik
Copy link
Contributor

we need stable solution to track state of the wda status. We already know that it might be done by connecting to mjpeg wda socket with nc localhost port.

As only wda crashed connection is closed and we can do some actions asap. Amobg the actions we should do start of wda and stf.
Auto recovery should be done to support multiply recoveries after wda crashes.

@vdelendik vdelendik added the enhancement New feature or request label Dec 24, 2022
@vdelendik vdelendik added this to the 2.4 milestone Dec 24, 2022
@vdelendik
Copy link
Contributor Author

done. during setup we configure custom LaunchAgent script per device which can run ./zebrunner.sh recover udid at any stage. As result right after setup we start all services for all connected devices and simulators asap.

to kickoff recovery manually you can use such command launchctl kickstart gui/$UID/com.zebrunner.mcloud.<udid> replacing udid by actual value:

launchctl kickstart gui/$UID/com.zebrunner.mcloud.44568699-BA22-4A88-8DD5-413AC1663051

as result we try to start one more tme: wda, appium and stf. all launchagent actions are tracked in ./logs/agent.log file.
So you can stop services per device nd start it using launchctl kickstart.. command. It will be used by stf to do recovery right after wda crash

@dhreben
Copy link

dhreben commented Jan 6, 2023

Test scenario for Simulator:

  1. ./zebrunnner.sh stop 44568699-BA22-4A88-8DD5-413AC1663051
  2. launchctl kickstart gui/$UID/com.zebrunner.mcloud.44568699-BA22-4A88-8DD5-413AC1663051

Expected result: Simulator restarted and available on STF

Test scenario for iOS:

  1. ./zebrunner.sh stop b09fa26ac..............3f862b5
  2. launchctl kickstart gui/$UID/com.zebrunner.mcloud.b09fa26ac..............3f862b5

Actual result: iOS device is not up and running on STF.

WorkAround:
./zebrunner.sh setup
./zebruner.sh start b09fa26ac..............3f862b5

Logs:

idmclouds-Mac-mini:mcloud-ios idmcloud$ launchctl kickstart gui/$UID/com.zebrunner.mcloud.b09fa26............e3348b7243f862b5
idmclouds-Mac-mini:mcloud-ios idmcloud$ tail -n 100 -f ./logs/iPhone_8.log
2023-01-06 13:31:10:885 - [Appium]   simpleIsVisibleCheck: true
2023-01-06 13:31:10:885 - [Appium]   wdaLocalPort: 20001
2023-01-06 13:31:10:885 - [Appium]   usePrebuiltWDA: true
2023-01-06 13:31:10:886 - [Appium]   useNewWDA: false
2023-01-06 13:31:10:886 - [Appium]   platformVersion: 
2023-01-06 13:31:10:886 - [Appium]   automationName: XCUITest
2023-01-06 13:31:10:886 - [Appium]   deviceName: iPhone_8
2023-01-06 13:31:10:905 - [HTTP] Could not start REST http interface listener. The requested port may already be in use. Please make sure there is no other instance of this server running already.
Fatal Error: listen EADDRINUSE: address already in use 0.0.0.0:4841
    at Server.setupListenHandle [as _listen2] (node:net:1334:16)
    at listenInCluster (node:net:1382:12)
    at doListen (node:net:1520:7)
    at processTicksAndRejections (node:internal/process/task_queues:84:21)
2023-01-06T13:31:11.070Z INF/db 37666 [b09fa2........e3348b7243f862b5] Connecting to demo.zebrunner.farm:28015
2023-01-06T13:31:11.077Z INF/device:support:push 37666 [b09fa26ac............43f862b5] Ios device sending output to "tcp://demo.zebrunner.farm:7270"
2023-01-06T13:31:11.079Z INF/ios-device 37666 [b09fa2............48b7243f862b5] Preparing device options:  {
  serial: 'b09fa26a.............43f862b5',
  provider: 'mcloud-ios',
  publicIp: 'demo.zebrunner.farm',
  endpoints: {
    sub: [ 'tcp://demo.zebrunner.farm:7250' ],
    push: [ 'tcp://demo.zebrunner.farm:7270' ],
    appDealer: [ 'tcp://demo.zebrunner.farm:7160' ],
    devDealer: [ 'tcp://demo.zebrunner.farm:7260' ]
  },
  groupTimeout: 3600000,
  storageUrl: 'https://demo.zebrunner.farm:443/',
  adbHost: '127.0.0.1',
  adbPort: 5037,
  screenJpegQuality: 30,
  screenPingInterval: 30000,
  screenPort: 7701,
  screenWsUrlPattern: 'wss://demo.zebrunner.farm:443/d/192.168.89.189/b09fa26af.............3f862b5/7701/',
  connectUrlPattern: '${publicIp}:${publicPort}',
  connectPort: 20002,
  wdaHost: 'localhost',
  wdaPort: 20001,
  vncInitialSize: [ 600, 800 ],
  heartbeatInterval: 10000,
  bootCompleteTimeout: 60000,
  muteMaster: 'never',
  lockRotation: false,
  cleanup: false,
  screenReset: true,
  udidStorage: false,
  iproxy: false,
  deviceName: 'iPhone_8',
  deviceType: 'Phone',
  host: '192.168.89.189'
}
{"bundleID":"com.facebook.WebDriverAgentRunner.xctrunner","channel_id":"com.apple.instruments.server.services.processcontrol","level":"info","msg":"Launching process","time":"2023-01-06T16:31:11+03:00"}
{"channel_id":"com.apple.instruments.server.services.processcontrol","level":"info","msg":"Process started successfully","pid":4830,"time":"2023-01-06T16:31:11+03:00"}
{"fields.msg":"2023-01-06 16:31:11.347286+0300 WebDriverAgentRunner-Runner[4830:361863] Running tests...\n","fields.time":2141305670487,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:11+03:00"}
2023-01-06T13:31:11.362Z INF/device:support:sub 37666 [b09fa26...............3f862b5] Receiving input from "tcp://demo.zebrunner.farm:7250"
2023-01-06T13:31:11.364Z INF/device:support:sub 37666 [b09fa26acc...............243f862b5] Subscribing to permanent channel "*ALL"
2023-01-06T13:31:11.366Z INF/device:plugins:solo 37666 [b09fa26acc4................3f862b5] Subscribing to permanent channel "dzTlyg6o9YlxdBizOqIGbhlToWI="
2023-01-06T13:31:11.369Z INF/wdaClient 37666 [b09fa26acc...............43f862b5] WdaClient.js initializing...
connecting to WdaMjpeg
2023-01-06T13:31:11.376Z INF/device:info 37666 [b09fa26acc4..............7243f862b5] device.name: iPhone_8
Connected to WdaMjpeg localhost:20002
WdaMjpeg connection was closed
2023-01-06T13:31:11.383Z INF/util:lifecycle 37666 [b09fa2...........b7243f862b5] Winding down for graceful exit, with optional error :  WdaMjpeg connection is lost
2023-01-06T13:31:11.393Z FTL/util:lifecycle 37666 [b09fa26acc...............243f862b5] Shutting down due to fatal error with optional error :  WdaMjpeg connection is lost
{"level":"info","msg":"device called local method","sel":"_XCT_didBeginExecutingTestPlan","time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_didBeginInitializingForUITesting","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"Test Suite 'All tests' started at 2023-01-06 16:31:12.783\n","fields.time":2141340143006,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testSuite:didStartAt:","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"XCTestOutputBarrier","fields.time":2141340148621,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"Test Suite 'WebDriverAgentRunner.xctest' started at 2023-01-06 16:31:12.784\n","fields.time":2141340156784,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testSuite:didStartAt:","time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testSuite:didStartAt:","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"XCTestOutputBarrierTest Suite 'UITestingUITests' started at 2023-01-06 16:31:12.785\nXCTestOutputBarrier","fields.time":2141340167143,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:willStartActivity:","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"    t =      nans Suite Set Up\n","fields.time":2141340173901,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.812499+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    KeyboardAutocorrection\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141340833375,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.813430+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_changedAt\"\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141340857285,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.813898+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_buildAtChange\"\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141340866608,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.814317+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_previousValue\"\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141340876512,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.814747+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_analyzedAt\"\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141340887003,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.831188+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    KeyboardPrediction\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141341283098,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
2023-01-06 13:31:12:768 - [HTTP] --> GET /wd/hub/status
{"fields.msg":"2023-01-06 16:31:12.831745+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    KeyboardShowPredictionBar\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141341300393,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
2023-01-06 13:31:12:768 - [HTTP] {}
2023-01-06 13:31:12:769 - [debug] [GENERIC] Calling AppiumDriver.getStatus() with args: []
2023-01-06 13:31:12:769 - [debug] [GENERIC] Responding to client with driver.getStatus() result: {"build":{"version":"1.22.3"}}
{"fields.msg":"2023-01-06 16:31:12.832371+0300 WebDriverAgentRunner-Runner[4830:361863] [User Defaults] Couldn't write values for keys (\n    DidShowGestureKeyboardIntroduction\n) in CFPrefsPlistSource\u003c0x281ead980\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":2141341315047,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
2023-01-06 13:31:12:770 - [HTTP] <-- GET /wd/hub/status 200 1 ms - 68
2023-01-06 13:31:12:770 - [HTTP] 
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:didFinishActivity:","time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCaseDidStartForTestClass:method:","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"Test Case '-[UITestingUITests testRunner]' started.\n","fields.time":2141341433736,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:willStartActivity:","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"XCTestOutputBarrier    t =     0.00s Start Test at 2023-01-06 16:31:12.838\n","fields.time":2141341456753,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"    t =     0.00s Set Up\n","fields.time":2141341487764,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.839965+0300 WebDriverAgentRunner-Runner[4830:361863] Built at Dec 13 2022 16:29:59\n","fields.time":2141341523487,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:didFinishActivity:","time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:willStartActivity:","time":"2023-01-06T16:31:12+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:didFinishActivity:","time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.855674+0300 WebDriverAgentRunner-Runner[4830:361863] ServerURLHere-\u003ehttp://192.168.88.208:20001\u003c-ServerURLHere\n","fields.time":2141341871976,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
{"fields.msg":"2023-01-06 16:31:12.855965+0300 WebDriverAgentRunner-Runner[4830:361866] Using singleton test manager\n","fields.time":2141341877891,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":4830,"time":"2023-01-06T16:31:12+03:00"}
2023-01-06 13:31:17:773 - [HTTP] --> GET /wd/hub/status
2023-01-06 13:31:17:773 - [HTTP] {}
2023-01-06 13:31:17:773 - [debug] [GENERIC] Calling AppiumDriver.getStatus() with args: []
2023-01-06 13:31:17:774 - [debug] [GENERIC] Responding to client with driver.getStatus() result: {"build":{"version":"1.22.3"}}
2023-01-06 13:31:17:775 - [HTTP] <-- GET /wd/hub/status 200 1 ms - 68
2023-01-06 13:31:17:775 - [HTTP] 
Starting WDA: iPhone_8, udid: b09fa26acc............b7243f862b5, WDA_PORT: 20001, MJPEG_PORT: 20002
Starting iSTF ios-device: b09fa2..........348b7243f862b5 device name : iPhone_8
[06/01/2023 16:31:20] Starting WebDriverAgent application on port 20001
Starting appium: b09fa2............e3348b7243f862b5 - device name : iPhone_8
{"bundleid":"com.facebook.WebDriverAgentRunner.xctrunner","level":"info","msg":"Running wda","testbundleid":"com.facebook.WebDriverAgentRunner.xctrunner","time":"2023-01-06T16:31:20+03:00","xctestconfig":"WebDriverAgentRunner.xctest"}
Now using node v17.1.0 (npm v8.3.0)
WDA started successfully on iPhone_8 within 0 seconds.
2023-01-06 13:31:21:394 - [Appium] Welcome to Appium v1.22.3
2023-01-06 13:31:21:395 - [Appium] Non-default server args:
2023-01-06 13:31:21:396 - [Appium]   port: 4841
2023-01-06 13:31:21:396 - [Appium]   sessionOverride: true
2023-01-06 13:31:21:396 - [Appium]   logTimestamp: true
2023-01-06 13:31:21:396 - [Appium]   logNoColors: true
2023-01-06 13:31:21:397 - [Appium]   nodeconfig: ./metaData/b09fa............b7243f862b5.json
2023-01-06 13:31:21:397 - [Appium]   tmpDir: /Users/idmcloud/tools/mcloud-ios/tmp/AppiumData/b09..........348b7243f862b5
2023-01-06 13:31:21:397 - [Appium]   defaultCapabilities: {
2023-01-06 13:31:21:398 - [Appium]     mjpegServerPort: 20002
2023-01-06 13:31:21:398 - [Appium]     webkitDebugProxyPort: 20003
2023-01-06 13:31:21:398 - [Appium]     clearSystemFiles: false
2023-01-06 13:31:21:398 - [Appium]     webDriverAgentUrl: http://localhost:20001
2023-01-06 13:31:21:399 - [Appium]     preventWDAAttachments: true
2023-01-06 13:31:21:399 - [Appium]     simpleIsVisibleCheck: true
2023-01-06 13:31:21:400 - [Appium]     wdaLocalPort: 20001
2023-01-06 13:31:21:400 - [Appium]     usePrebuiltWDA: true
2023-01-06 13:31:21:400 - [Appium]     useNewWDA: false
2023-01-06 13:31:21:400 - [Appium]     platformVersion: 
2023-01-06 13:31:21:401 - [Appium]     automationName: XCUITest
2023-01-06 13:31:21:401 - [Appium]     deviceName: iPhone_8
2023-01-06 13:31:21:401 - [Appium]   }
2023-01-06 13:31:21:401 - [Appium]   deviceName: iPhone_8
2023-01-06 13:31:21:402 - [Appium]   udid: b09fa..............b7243f862b5
2023-01-06 13:31:21:403 - [Appium] Deprecated server args:
2023-01-06 13:31:21:403 - [Appium]   --device-name => --default-capabilities
2023-01-06 13:31:21:403 - [Appium]   -U => --default-capabilities
2023-01-06 13:31:21:403 - [Appium] Default capabilities, which will be added to each request unless overridden by desired capabilities:
2023-01-06 13:31:21:404 - [Appium]   mjpegServerPort: 20002
2023-01-06 13:31:21:404 - [Appium]   webkitDebugProxyPort: 20003
2023-01-06 13:31:21:404 - [Appium]   clearSystemFiles: false
2023-01-06 13:31:21:404 - [Appium]   webDriverAgentUrl: http://localhost:20001
2023-01-06 13:31:21:405 - [Appium]   preventWDAAttachments: true
2023-01-06 13:31:21:405 - [Appium]   simpleIsVisibleCheck: true
2023-01-06 13:31:21:405 - [Appium]   wdaLocalPort: 20001
2023-01-06 13:31:21:406 - [Appium]   usePrebuiltWDA: true
2023-01-06 13:31:21:406 - [Appium]   useNewWDA: false
2023-01-06 13:31:21:406 - [Appium]   platformVersion: 
2023-01-06 13:31:21:406 - [Appium]   automationName: XCUITest
2023-01-06 13:31:21:407 - [Appium]   deviceName: iPhone_8


@dhreben dhreben reopened this Jan 6, 2023
@vdelendik vdelendik modified the milestones: 2.4, 2.4.2 Jan 6, 2023
@vdelendik
Copy link
Contributor Author

closing again. right after setup we don't do start anymore..
also, there is no need to kickoff recovery script manually.

so new scenario:

  1. setup
  2. start
  3. kill existing wda using shell command:
// for simulator:
xcrun simctl terminate <UDID> com.facebook.WebDriverAgentRunner.xctrunner
// for real device
ios kill com.facebook.WebDriverAgentRunner.xctrunner --udid=<UDID>

-> recovery script is activated (use tailf ./logs/agents.log to see it)
-> services restarted and device is available in STF and grid console

@dhreben
Copy link

dhreben commented Jan 9, 2023

Still repro after run:

// for real device
ios kill com.facebook.WebDriverAgentRunner.xctrunner --udid=<UDID>


NOTE:

iOS device restarted and up and running on STF after 8-9min.
If trying run it again ios kill com.facebook.WebDriverAgentRunner.xctrunner --udid=<UDID> Script doesn't work with error---> "error":"Timed out waiting for response for message:5

Please see Logs 2 for this issue

Logs:

idmclouds-Mac-mini:mcloud-ios idmcloud$ ios kill com.facebook.WebDriverAgentRunner.xctrunner --udid=b09fa26acc.........3f862b5
{"level":"info","msg":"com.facebook.WebDriverAgentRunner.xctrunner killed, Pid: 5642","time":"2023-01-09T17:18:13+03:00"}
idmclouds-Mac-mini:mcloud-ios idmcloud$ tail -f ./logs/agents.log
Recovering services for iPhone_8 (b09fa............b7243f862b5)
Recovering services for iPhone_8 (b09f................778b7243f862b5)
Recovering services for iPhone_8 (b0.........8b7243f862b5)
Recovering services for iPhone_8 (b09fa2............8b7243f862b5)
Recovering services for iPhone_8 (b09f..b7243f862b5)
Recovering services for iPhone_8 (b09.............b7243f862b5)
Recovering services for iPhone_8 (b09f................7243f862b5)
Recovering services for iPhone_8 (b09f....................243f862b5)
Recovering services for iPhone_8 (b09fa...............8b7243f862b5)
Recovering services for iPhone_8 (b09f......................8b7243f862b5)
Recovering services for iPhone_8 (b09f...................7243f862b5)
Recovering services for iPhone_8 (b09...................48b7243f862b5)
Recovering services for iPhone_8 (b09f...................3f862b5)
Recovering services for iPhone_8 (b09fa2...............243f862b5)


idmclouds-Mac-mini:mcloud-ios idmcloud$ tail -n 100 -f ./logs/iPhone_8.log
2023-01-09 14:21:26:247 - [Appium]   clearSystemFiles: false
2023-01-09 14:21:26:247 - [Appium]   webDriverAgentUrl: http://localhost:20001
2023-01-09 14:21:26:247 - [Appium]   preventWDAAttachments: true
2023-01-09 14:21:26:247 - [Appium]   simpleIsVisibleCheck: true
2023-01-09 14:21:26:247 - [Appium]   wdaLocalPort: 20001
2023-01-09 14:21:26:247 - [Appium]   usePrebuiltWDA: true
2023-01-09 14:21:26:247 - [Appium]   useNewWDA: false
2023-01-09 14:21:26:247 - [Appium]   platformVersion: 
2023-01-09 14:21:26:248 - [Appium]   automationName: XCUITest
2023-01-09 14:21:26:248 - [Appium]   deviceName: iPhone_8
2023-01-09 14:21:26:265 - [HTTP] Could not start REST http interface listener. The requested port may already be in use. Please make sure there is no other instance of this server running already.
Fatal Error: listen EADDRINUSE: address already in use 0.0.0.0:4841
    at Server.setupListenHandle [as _listen2] (node:net:1334:16)
    at listenInCluster (node:net:1382:12)
    at doListen (node:net:1520:7)
    at processTicksAndRejections (node:internal/process/task_queues:84:21)
2023-01-09 14:21:26:386 - [HTTP] --> GET /wd/hub/status
2023-01-09 14:21:26:386 - [HTTP] {}
2023-01-09 14:21:26:386 - [debug] [GENERIC] Calling AppiumDriver.getStatus() with args: []
2023-01-09 14:21:26:386 - [debug] [GENERIC] Responding to client with driver.getStatus() result: {"build":{"version":"1.22.3"}}
2023-01-09 14:21:26:387 - [HTTP] <-- GET /wd/hub/status 200 1 ms - 68
2023-01-09 14:21:26:387 - [HTTP] 
2023-01-09T14:21:26.555Z INF/db 70860 [b09fa26.................8b7243f862b5] Connecting to demo.zebrunner.farm:28015
2023-01-09T14:21:26.562Z INF/device:support:push 70860 [b09fa26ac..............b7243f862b5] Ios device sending output to "tcp://demo.zebrunner.farm:7270"
2023-01-09T14:21:26.563Z INF/ios-device 70860 [b09fa.............48b7243f862b5] Preparing device options:  {
  serial: 'b09fa2..............e3348b7243f862b5',
  provider: 'mcloud-ios',
  publicIp: 'demo.zebrunner.farm',
  endpoints: {
    sub: [ 'tcp://demo.zebrunner.farm:7250' ],
    push: [ 'tcp://demo.zebrunner.farm:7270' ],
    appDealer: [ 'tcp://demo.zebrunner.farm:7160' ],
    devDealer: [ 'tcp://demo.zebrunner.farm:7260' ]
  },
  groupTimeout: 3600000,
  storageUrl: 'https://demo.zebrunner.farm:443/',
  adbHost: '127.0.0.1',
  adbPort: 5037,
  screenJpegQuality: 30,
  screenPingInterval: 30000,
  screenPort: 7701,
  screenWsUrlPattern: 'wss://demo.zebrunner.farm:443/d/192.168.89.189/b09fa26..................8b7243f862b5/7701/',
  connectUrlPattern: '${publicIp}:${publicPort}',
  connectPort: 20002,
  wdaHost: 'localhost',
  wdaPort: 20001,
  vncInitialSize: [ 600, 800 ],
  heartbeatInterval: 10000,
  bootCompleteTimeout: 60000,
  muteMaster: 'never',
  lockRotation: false,
  cleanup: false,
  screenReset: true,
  udidStorage: false,
  iproxy: false,
  deviceName: 'iPhone_8',
  deviceType: 'Phone',
  host: '192.168.89.189'
}
{"bundleID":"com.facebook.WebDriverAgentRunner.xctrunner","channel_id":"com.apple.instruments.server.services.processcontrol","level":"info","msg":"Launching process","time":"2023-01-09T17:21:26+03:00"}
{"channel_id":"com.apple.instruments.server.services.processcontrol","level":"info","msg":"Process started successfully","pid":5683,"time":"2023-01-09T17:21:26+03:00"}
{"fields.msg":"2023-01-09 17:21:26.668371+0300 WebDriverAgentRunner-Runner[5683:779790] Running tests...\n","fields.time":5673057456931,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:26+03:00"}
2023-01-09T14:21:26.841Z INF/device:support:sub 70860 [b09fa................b7243f862b5] Receiving input from "tcp://demo.zebrunner.farm:7250"
2023-01-09T14:21:26.842Z INF/device:support:sub 70860 [b09f.................243f862b5] Subscribing to permanent channel "*ALL"
2023-01-09T14:21:26.845Z INF/device:plugins:solo 70860 [b09fa..............8b7243f862b5] Subscribing to permanent channel "dzTlyg6o9YlxdBizOqIGbhlToWI="
2023-01-09T14:21:26.847Z INF/wdaClient 70860 [b09fa2...........7243f862b5] WdaClient.js initializing...
connecting to WdaMjpeg
2023-01-09T14:21:26.855Z INF/device:info 70860 [b09fa................b7243f862b5] device.name: iPhone_8
Connected to WdaMjpeg localhost:20002
WdaMjpeg connection was closed
2023-01-09T14:21:26.862Z INF/util:lifecycle 70860 [b09f................7243f862b5] Winding down for graceful exit, with optional error :  WdaMjpeg connection is lost
2023-01-09T14:21:26.871Z FTL/util:lifecycle 70860 [b09fa.............48b7243f862b5] Shutting down due to fatal error with optional error :  WdaMjpeg connection is lost
{"level":"info","msg":"device called local method","sel":"_XCT_didBeginExecutingTestPlan","time":"2023-01-09T17:21:27+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_didBeginInitializingForUITesting","time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"Test Suite 'All tests' started at 2023-01-09 17:21:28.108\n","fields.time":5673092005870,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"XCTestOutputBarrier","fields.time":5673092011065,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testSuite:didStartAt:","time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testSuite:didStartAt:","time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"Test Suite 'WebDriverAgentRunner.xctest' started at 2023-01-09 17:21:28.109\n","fields.time":5673092020079,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"XCTestOutputBarrierTest Suite 'UITestingUITests' started at 2023-01-09 17:21:28.109\n","fields.time":5673092026125,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testSuite:didStartAt:","time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"XCTestOutputBarrier","fields.time":5673092031168,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:willStartActivity:","time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"    t =      nans Suite Set Up\n","fields.time":5673092036100,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.137060+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    KeyboardAutocorrection\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673092704215,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.138228+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_changedAt\"\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673092729966,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.138633+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_buildAtChange\"\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673092740090,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.139028+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_previousValue\"\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673092749286,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.139418+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    \"KeyboardAutocorrection_analyzedAt\"\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673092758541,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.155688+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    KeyboardPrediction\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n2023-01-09 17:21:28.156177+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    KeyboardShowPredictionBar\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673093168533,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.156766+0300 WebDriverAgentRunner-Runner[5683:779790] [User Defaults] Couldn't write values for keys (\n    DidShowGestureKeyboardIntroduction\n) in CFPrefsPlistSource\u003c0x280122800\u003e (Domain: com.apple.keyboard.preferences, User: kCFPreferencesCurrentUser, ByHost: No, Container: (null), Contents Need Refresh: No): setting preferences outside an application's container requires user-preference-write or file-write-data sandbox access\n","fields.time":5673093175711,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:didFinishActivity:","time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCaseDidStartForTestClass:method:","time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:willStartActivity:","time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:didFinishActivity:","time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:willStartActivity:","time":"2023-01-09T17:21:28+03:00"}
{"level":"info","msg":"device called local method","sel":"_XCT_testCase:method:didFinishActivity:","time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"Test Case '-[UITestingUITests testRunner]' started.\nXCTestOutputBarrier    t =     0.00s Start Test at 2023-01-09 17:21:28.162\n    t =     0.00s Set Up\n2023-01-09 17:21:28.164353+0300 WebDriverAgentRunner-Runner[5683:779790] Built at Dec 13 2022 16:29:59\n","fields.time":5673093357424,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.181249+0300 WebDriverAgentRunner-Runner[5683:779790] ServerURLHere-\u003ehttp://169.254.29.89:20001\u003c-ServerURLHere\n","fields.time":5673093765565,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
{"fields.msg":"2023-01-09 17:21:28.181602+0300 WebDriverAgentRunner-Runner[5683:779793] Using singleton test manager\n","fields.time":5673093772150,"level":"info","msg":"outputReceived:fromProcess:atTime:","pid":5683,"time":"2023-01-09T17:21:28+03:00"}
2023-01-09 14:21:31:390 - [HTTP] --> GET /wd/hub/status
2023-01-09 14:21:31:390 - [HTTP] {}
2023-01-09 14:21:31:391 - [debug] [GENERIC] Calling AppiumDriver.getStatus() with args: []
2023-01-09 14:21:31:391 - [debug] [GENERIC] Responding to client with driver.getStatus() result: {"build":{"version":"1.22.3"}}
2023-01-09 14:21:31:392 - [HTTP] <-- GET /wd/hub/status 200 2 ms - 68
2023-01-09 14:21:31:392 - [HTTP] 
Starting iSTF ios-device: b09fa26acc4..............243f862b5 device name : iPhone_8
Starting appium: b09f.............8b7243f862b5 - device name : iPhone_8
Starting WDA: iPhone_8, udid: b09fa.............e3348b7243f862b5, WDA_PORT: 20001, MJPEG_PORT: 20002
[09/01/2023 17:21:35] Starting WebDriverAgent application on port 20001
{"bundleid":"com.facebook.WebDriverAgentRunner.xctrunner","level":"info","msg":"Running wda","testbundleid":"com.facebook.WebDriverAgentRunner.xctrunner","time":"2023-01-09T17:21:35+03:00","xctestconfig":"WebDriverAgentRunner.xctest"}
2023-01-09 14:21:36:395 - [HTTP] --> GET /wd/hub/status
2023-01-09 14:21:36:396 - [HTTP] {}
2023-01-09 14:21:36:396 - [debug] [GENERIC] Calling AppiumDriver.getStatus() with args: []
2023-01-09 14:21:36:396 - [debug] [GENERIC] Responding to client with driver.getStatus() result: {"build":{"version":"1.22.3"}}
2023-01-09 14:21:36:397 - [HTTP] <-- GET /wd/hub/status 200 1 ms - 68
2023-01-09 14:21:36:397 - [HTTP] 
WDA started successfully on iPhone_8 within 0 seconds.
Now using node v17.1.0 (npm v8.3.0)
2023-01-09 14:21:36:941 - [Appium] Welcome to Appium v1.22.3
2023-01-09 14:21:36:942 - [Appium] Non-default server args:
2023-01-09 14:21:36:943 - [Appium]   port: 4841
2023-01-09 14:21:36:943 - [Appium]   sessionOverride: true
2023-01-09 14:21:36:944 - [Appium]   logTimestamp: true
2023-01-09 14:21:36:944 - [Appium]   logNoColors: true
2023-01-09 14:21:36:944 - [Appium]   nodeconfig: ./metaData/b09fa...........348b7243f862b5.json
2023-01-09 14:21:36:944 - [Appium]   tmpDir: /Users/idmcloud/tools/mcloud-ios/tmp/AppiumData/b09f..........49e3348b7243f862b5
2023-01-09 14:21:36:945 - [Appium]   defaultCapabilities: {
2023-01-09 14:21:36:945 - [Appium]     mjpegServerPort: 20002
2023-01-09 14:21:36:946 - [Appium]     webkitDebugProxyPort: 20003
2023-01-09 14:21:36:947 - [Appium]     clearSystemFiles: false
2023-01-09 14:21:36:947 - [Appium]     webDriverAgentUrl: http://localhost:20001
2023-01-09 14:21:36:947 - [Appium]     preventWDAAttachments: true
2023-01-09 14:21:36:948 - [Appium]     simpleIsVisibleCheck: true
2023-01-09 14:21:36:948 - [Appium]     wdaLocalPort: 20001
2023-01-09 14:21:36:948 - [Appium]     usePrebuiltWDA: true
2023-01-09 14:21:36:948 - [Appium]     useNewWDA: false
2023-01-09 14:21:36:949 - [Appium]     platformVersion: 
2023-01-09 14:21:36:949 - [Appium]     automationName: XCUITest
2023-01-09 14:21:36:949 - [Appium]     deviceName: iPhone_8
2023-01-09 14:21:36:949 - [Appium]   }
2023-01-09 14:21:36:950 - [Appium]   deviceName: iPhone_8
2023-01-09 14:21:36:950 - [Appium]   udid: b09fa2...........8b7243f862b5
2023-01-09 14:21:36:950 - [Appium] Deprecated server args:
2023-01-09 14:21:36:951 - [Appium]   --device-name => --default-capabilities
2023-01-09 14:21:36:951 - [Appium]   -U => --default-capabilities
2023-01-09 14:21:36:951 - [Appium] Default capabilities, which will be added to each request unless overridden by desired capabilities:
2023-01-09 14:21:36:951 - [Appium]   mjpegServerPort: 20002
2023-01-09 14:21:36:952 - [Appium]   webkitDebugProxyPort: 20003
2023-01-09 14:21:36:952 - [Appium]   clearSystemFiles: false
2023-01-09 14:21:36:952 - [Appium]   webDriverAgentUrl: http://localhost:20001
2023-01-09 14:21:36:952 - [Appium]   preventWDAAttachments: true
2023-01-09 14:21:36:953 - [Appium]   simpleIsVisibleCheck: true
2023-01-09 14:21:36:953 - [Appium]   wdaLocalPort: 20001
2023-01-09 14:21:36:953 - [Appium]   usePrebuiltWDA: true
2023-01-09 14:21:36:953 - [Appium]   useNewWDA: false
2023-01-09 14:21:36:953 - [Appium]   platformVersion: 
2023-01-09 14:21:36:953 - [Appium]   automationName: XCUITest
2023-01-09 14:21:36:954 - [Appium]   deviceName: iPhone_8
2023-01-09 14:21:36:973 - [HTTP] Could not start REST http interface listener. The requested port may already be in use. Please make sure there is no other instance of this server running already.
Fatal Error: listen EADDRINUSE: address already in use 0.0.0.0:4841
    at Server.setupListenHandle [as _listen2] (node:net:1334:16)

Logs 2:

idmclouds-Mac-mini:mcloud-ios idmcloud$ ios kill com.facebook.WebDriverAgentRunner.xctrunner --udid=b09fa..........3348b7243f862b5
{"channel_id":"com.apple.instruments.server.services.processcontrol","error":"Timed out waiting for response for message:5 channel:0","level":"error","msg":"failed requesting channel","time":"2023-01-09T17:31:22+03:00"}
{"channel_id":"com.apple.instruments.server.services.deviceinfo","error":"Timed out waiting for response for message:5 channel:0","level":"error","msg":"failed requesting channel","time":"2023-01-09T17:31:27+03:00"}
{"level":"error","msg":"DeviceConnection failed writing 204 bytes, only 0 sent","time":"2023-01-09T17:31:27+03:00"}
{"level":"error","msg":"Failed sending: write unix -\u003e/var/run/usbmuxd: write: broken pipe","time":"2023-01-09T17:31:27+03:00"}
{"channel_id":"com.apple.instruments.server.services.deviceinfo","error":"write unix -\u003e/var/run/usbmuxd: write: broken pipe","level":"info","methodselector":"runningProcesses","msg":"failed starting invoking method","time":"2023-01-09T17:31:27+03:00"}
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/danielpaulus/go-ios/ios/instruments.DeviceInfoService.ProcessList({0xc0000a45a0, 0xc0000a4120})
	/Users/runner/work/go-ios/go-ios/ios/instruments/instruments_deviceinfo.go:26 +0x137
main.Main()
	/Users/runner/work/go-ios/go-ios/main.go:586 +0x30f1
main.main()
	/Users/runner/work/go-ios/go-ios/main.go:52 +0x17
idmclouds-Mac-mini:mcloud-ios idmcloud$ 

@dhreben dhreben reopened this Jan 9, 2023
@vdelendik vdelendik modified the milestones: 2.4.2, 2.4.3 Jan 9, 2023
vdelendik added a commit that referenced this issue Jan 9, 2023
#151: added wda stop for recovery to make prot forwarding workable af…
@vdelendik
Copy link
Contributor Author

added wda stop during recovery step. so this time we expect to restart wda, port forwarding for wda and mjpeg and stf. Appium is not restarted so exception to bind to appium port is ok during recovery. It is better to keep old appium to avoid all kind of selenium grid problems...

@vdelendik vdelendik removed this from the 2.4.3 milestone Jan 9, 2023
@vdelendik vdelendik added this to the 2.4.2 milestone Jan 9, 2023
@dhreben
Copy link

dhreben commented Jan 10, 2023

Verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants