refactor: Optimize screenshots preprocessing #812

mykola-mokhnach · 2023-11-12T15:34:01Z

I can observe that videos generated from MJPEG streams supplied by WDA have rotation issues if device orientation is changed dynamically. No video issues are observed if screenshots orientation is adjusted properly according to the current implementation.

mykola-mokhnach · 2023-11-12T15:34:20Z

@KazuCocoa Could you please test it with real iPad?

Dan-Maor · 2023-11-12T17:33:53Z

Will run a verification on my test devices tomorrow once I get to the office.

KazuCocoa · 2023-11-12T18:05:08Z

Checked glance over a browser to http://<device address>:9100/. Then, the screens showed a grey screen with errors below a couple of times after 10 sec or so. Then, the xcodebuild stopped even when I kept the same orientation at the beginning.

2023-11-12 09:52:20.552489-0800 WebDriverAgentRunner-Runner[24432:3091682] IOSurface creation failed: e00002bd parentID: 00000000 properties: {
    IOSurfaceAllocSize = 12582912;
    IOSurfaceBytesPerElement = 4;
    IOSurfaceBytesPerRow = 6144;
    IOSurfaceCacheMode = 0;
    IOSurfaceHeight = 2048;
    IOSurfaceMapCacheAttribute = 1;
    IOSurfaceName = CMPhoto;
    IOSurfaceOffset = 0;
    IOSurfacePixelFormat = 1111970369;
    IOSurfacePreallocPages = 0;
    IOSurfaceWidth = 1536;
}
2023-11-12 09:52:20.553251-0800 WebDriverAgentRunner-Runner[24432:3091682] IOSurface creation failed: e00002bd parentID: 00000000 properties: {
    IOSurfaceAllocSize = 12582912;
    IOSurfaceBytesPerElement = 4;
    IOSurfaceBytesPerRow = 6144;
    IOSurfaceCacheMode = 0;
    IOSurfaceHeight = 2048;
    IOSurfaceMapCacheAttribute = 1;
    IOSurfaceName = CMPhoto;
    IOSurfaceOffset = 0;
    IOSurfacePixelFormat = 1111970369;
    IOSurfacePreallocPages = 0;
    IOSurfaceWidth = 1536;
}
2023-11-12 09:52:20.553654-0800 WebDriverAgentRunner-Runner[24432:3091682] [WebDriverAgentRunner-Runner] callDecodeImage:2006: *** ERROR: decodeImageImp failed - NULL _blockArray
2023-11-12 09:52:20.553884-0800 WebDriverAgentRunner-Runner[24432:3091682] [WebDriverAgentRunner-Runner] callDecodeImage:2006: *** ERROR: decodeImageImp failed - NULL _blockArray

The current master did not have the error.

Just a note:
Btw, my env worked orientation via the video stream by following the device orientation with current master. I haven't checked the implementation details well (below log is an example of when I changed the device orientation with the current master).

2023-11-12 09:54:14.007873-0800 WebDriverAgentRunner-Runner[24465:3094189] Got screenshots broadcast client connection at 192.168.4.248:55258
2023-11-12 09:54:14.008612-0800 WebDriverAgentRunner-Runner[24465:3094189] Starting screenshots broadcast for the client at 192.168.4.248:55257
2023-11-12 09:54:14.012400-0800 WebDriverAgentRunner-Runner[24465:3094189] Using singleton test manager
2023-11-12 09:54:16.352697-0800 WebDriverAgentRunner-Runner[24465:3094189] Disconnected a client from screenshots broadcast
    t =   220.14s Device orientation changed to Portrait Upside Down
    t =   221.99s Interface orientation changed to Portrait Upside Down
    t =   221.99s Device orientation changed to Landscape Left
    t =   229.26s Interface orientation changed to Landscape Right
    t =   229.26s Device orientation changed to Portrait Upside Down
    t =   231.66s Interface orientation changed to Portrait Upside Down
    t =   231.67s Device orientation changed to Landscape Left

I have a plan today, so my next look will be late

mykola-mokhnach · 2023-11-12T20:33:11Z

Checked glance over a browser to http://<device address>:9100/. Then, the screens showed a grey screen with errors below a couple of times after 10 sec or so. Then, the xcodebuild stopped even when I kept the same orientation at the beginning.

2023-11-12 09:52:20.552489-0800 WebDriverAgentRunner-Runner[24432:3091682] IOSurface creation failed: e00002bd parentID: 00000000 properties: {
    IOSurfaceAllocSize = 12582912;
    IOSurfaceBytesPerElement = 4;
    IOSurfaceBytesPerRow = 6144;
    IOSurfaceCacheMode = 0;
    IOSurfaceHeight = 2048;
    IOSurfaceMapCacheAttribute = 1;
    IOSurfaceName = CMPhoto;
    IOSurfaceOffset = 0;
    IOSurfacePixelFormat = 1111970369;
    IOSurfacePreallocPages = 0;
    IOSurfaceWidth = 1536;
}
2023-11-12 09:52:20.553251-0800 WebDriverAgentRunner-Runner[24432:3091682] IOSurface creation failed: e00002bd parentID: 00000000 properties: {
    IOSurfaceAllocSize = 12582912;
    IOSurfaceBytesPerElement = 4;
    IOSurfaceBytesPerRow = 6144;
    IOSurfaceCacheMode = 0;
    IOSurfaceHeight = 2048;
    IOSurfaceMapCacheAttribute = 1;
    IOSurfaceName = CMPhoto;
    IOSurfaceOffset = 0;
    IOSurfacePixelFormat = 1111970369;
    IOSurfacePreallocPages = 0;
    IOSurfaceWidth = 1536;
}
2023-11-12 09:52:20.553654-0800 WebDriverAgentRunner-Runner[24432:3091682] [WebDriverAgentRunner-Runner] callDecodeImage:2006: *** ERROR: decodeImageImp failed - NULL _blockArray
2023-11-12 09:52:20.553884-0800 WebDriverAgentRunner-Runner[24432:3091682] [WebDriverAgentRunner-Runner] callDecodeImage:2006: *** ERROR: decodeImageImp failed - NULL _blockArray

The current master did not have the error.

Just a note: Btw, my env worked orientation via the video stream by following the device orientation with current master. I haven't checked the implementation details well (below log is an example of when I changed the device orientation with the current master).

2023-11-12 09:54:14.007873-0800 WebDriverAgentRunner-Runner[24465:3094189] Got screenshots broadcast client connection at 192.168.4.248:55258
2023-11-12 09:54:14.008612-0800 WebDriverAgentRunner-Runner[24465:3094189] Starting screenshots broadcast for the client at 192.168.4.248:55257
2023-11-12 09:54:14.012400-0800 WebDriverAgentRunner-Runner[24465:3094189] Using singleton test manager
2023-11-12 09:54:16.352697-0800 WebDriverAgentRunner-Runner[24465:3094189] Disconnected a client from screenshots broadcast
    t =   220.14s Device orientation changed to Portrait Upside Down
    t =   221.99s Interface orientation changed to Portrait Upside Down
    t =   221.99s Device orientation changed to Landscape Left
    t =   229.26s Interface orientation changed to Landscape Right
    t =   229.26s Device orientation changed to Portrait Upside Down
    t =   231.66s Interface orientation changed to Portrait Upside Down
    t =   231.67s Device orientation changed to Landscape Left

I have a plan today, so my next look will be late

thanks for checking it @KazuCocoa

I did some optimisations. How do you record your video? I tried GStreamer with videoflip pipeline method set to automatic. It only properly records the video if the device initial orientation is landscape, but fails to properly adjust the picture if it changes during the test

KazuCocoa · 2023-11-12T20:37:36Z

I didn't record the video at that time. I just opened the stream URL on Chrome to go through refactored methods in order to compare before/after as a quick check. The orientation was both portrait and landscape at the beginning and changed the rotation in the middle.
The error itself was one orientation (no change), I haven't checked well for more complex pattern like change the orientation fequently

mykola-mokhnach · 2023-11-12T21:21:38Z

I didn't record the video at that time. I just opened the stream URL on Chrome to go through refactored methods in order to compare before/after as a quick check. The orientation was both portrait and landscape at the beginning and changed the rotation in the middle. The error itself was one orientation (no change), I haven't checked well for more complex pattern like change the orientation fequently

Sorry for not being too precise. This issue is only visible if the actual mjpeg stream is recorded to a video, like x264 or AVC one. Browsers properly handle EXIF tags and adjust shown images automatically.

KazuCocoa · 2023-11-13T08:50:18Z

Oh, i see

This issue is only visible if the actual mjpeg stream is recorded to a video, like x264 or AVC one. Browsers properly handle EXIF tags and adjust shown images automatically.

KazuCocoa · 2023-11-13T09:24:30Z

I'll need to prepare the recording env on my local, so the video recording test will be late.

At this time, I checked my previous comment's error in #812 (comment). I observed the wda process stop after 20 sec or more (at least in 1 min or less) when I opened the http://<device address>:9100/ in Chrome. I noticed that from the animation perspective, such as YouTube content, the current master was smoother than 8aa5c06.

Hm, the error didn't have the stacktrace in the Xcode so my investigation will take more time for the process stop as well

WebDriverAgentTests/IntegrationTests/FBImageProcessorTests.m

WebDriverAgentLib/Utilities/FBImageProcessor.m

This reverts commit 5b5e4cc.

Dan-Maor

The performance seems to be much better, the WDA process CPU threshold on my 13 Pro Max doesn't pass the 60% mark in the debugger and memory usage appears to be stable.

However, when I tested with ffmpeg it doesn't look like it solved the original problem (i.e. starting with portrait, going to landscape and then to portrait again - the orientation is incorrect).

For reference, the command I used is ffmpeg -f mjpeg -c:v mjpeg -i http://x.x.x.x:n test.mp4

WebDriverAgentLib/Utilities/FBImageProcessor.m

mykola-mokhnach · 2023-11-14T19:05:22Z

The performance seems to be much better, the WDA process CPU threshold on my 13 Pro Max doesn't pass the 60% mark in the debugger and memory usage appears to be stable.

However, when I tested with ffmpeg it doesn't look like it solved the original problem (i.e. starting with portrait, going to landscape and then to portrait again - the orientation is incorrect).

For reference, the command I used is ffmpeg -f mjpeg -c:v mjpeg -i http://x.x.x.x:n test.mp4

Ok, I've finally made it work with the UIGraphicsImageRenderer
the thumbnail creator is nice, but looks like it only works for scaling and does not care about rotation fixing :(

Dan-Maor · 2023-11-15T08:10:24Z

Ran a check, unfortunately this approach looks to be too heavy resource-wise as well. When the device is in landscape it doesn't take long until it is stopped due to high CPU and memory utilization.

Adding an autoreleasepool to fixedImageDataWithImageData as @KazuCocoa suggested helps with the memory utilization, however the CPU usage is still extremely high.

mykola-mokhnach · 2023-11-15T13:31:29Z

Ran a check, unfortunately this approach looks to be too heavy resource-wise as well. When the device is in landscape it doesn't take long until it is stopped due to high CPU and memory utilization.

Adding an autoreleasepool to fixedImageDataWithImageData as @KazuCocoa suggested helps with the memory utilization, however the CPU usage is still extremely high.

thanks for confirming it @Dan-Maor
So, let then skip rotation fixes for MJPEG images as it creates too much of the CPU load and only keep resize op which is done via prepareThumbnailOfSize API (which has been confirmed as good enough from the performance perspective)

I will only preserve the UIGraphicsImageRenderer-based transformation for regular PNG screenshots as these are not supposed to be taken so frequently

Dan-Maor

Looks good.
I've verified that the original recording behavior is preserved with the setting turned off and that the orientation fix is working with the setting turned on.

Dan-Maor · 2023-11-16T13:40:34Z

WebDriverAgentLib/Utilities/FBImageProcessor.m

+                                                                uti:UTTypeJPEG
+                                                 compressionQuality:recompressionQuality
+    // iOS always returns screnshots in portrait orientation, but puts the real value into the metadata
+    // Use it with care. See https://github.com/appium/WebDriverAgent/pull/812


## [5.15.1](v5.15.0...v5.15.1) (2023-11-16) ### Bug Fixes * Content-Type of the MJPEG server ([b4704da](b4704da)) ### Code Refactoring * Optimize screenshots preprocessing ([#812](#812)) ([0b41757](0b41757))

refactor: Fix screenshots orientation for the MJPEG stream

9eca854

mykola-mokhnach requested a review from KazuCocoa November 12, 2023 15:34

mykola-mokhnach requested a review from Dan-Maor November 12, 2023 17:20

Simplify

2cf17df

Optimize

0f234f1

mykola-mokhnach added 6 commits November 12, 2023 22:27

Add comments

ef3e2aa

wording

fc755e4

Optimizie

067ff2d

Check rotation

f9f1c2f

More fixes

6cd0975

Tune numbers

8aa5c06

Dan-Maor reviewed Nov 13, 2023

View reviewed changes

WebDriverAgentTests/IntegrationTests/FBImageProcessorTests.m Show resolved Hide resolved

WebDriverAgentLib/Utilities/FBImageProcessor.m Outdated Show resolved Hide resolved

mykola-mokhnach added 4 commits November 13, 2023 20:44

Try UIGraphicsImageRenderer

5b5e4cc

Revert "Try UIGraphicsImageRenderer"

f5bae97

This reverts commit 5b5e4cc.

Remove unused helpers

c422a52

Try prepareThumbnailOfSize

e08b48f

Dan-Maor reviewed Nov 14, 2023

View reviewed changes

WebDriverAgentLib/Utilities/FBImageProcessor.m Outdated Show resolved Hide resolved

WebDriverAgentLib/Utilities/FBImageProcessor.m Outdated Show resolved Hide resolved

mykola-mokhnach added 5 commits November 14, 2023 15:12

address comments

9d7c558

Tune

c561322

Tune

a7e0959

Use UIImage

960146d

Use renderer

1383134

Simplify more

2c3095d

Skip fixing the rotation

129bc91

mykola-mokhnach changed the title ~~refactor: Fix screenshots orientation for the MJPEG stream~~ refactor: Optimize screenshots preprocessing Nov 15, 2023

mykola-mokhnach added 5 commits November 15, 2023 14:35

optimize

8b0c567

fix transform

4a3ab33

Add autoreleasepool

63a7d1c

Add setting

91354ee

Add explanation

7b0ff93

Dan-Maor approved these changes Nov 16, 2023

View reviewed changes

mykola-mokhnach merged commit 0b41757 into master Nov 16, 2023
43 of 46 checks passed

mykola-mokhnach deleted the orient branch November 16, 2023 14:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Optimize screenshots preprocessing #812

refactor: Optimize screenshots preprocessing #812

mykola-mokhnach commented Nov 12, 2023

mykola-mokhnach commented Nov 12, 2023 •

edited

Loading

Dan-Maor commented Nov 12, 2023

KazuCocoa commented Nov 12, 2023 •

edited

Loading

mykola-mokhnach commented Nov 12, 2023

KazuCocoa commented Nov 12, 2023 •

edited

Loading

mykola-mokhnach commented Nov 12, 2023

KazuCocoa commented Nov 13, 2023

KazuCocoa commented Nov 13, 2023

Dan-Maor left a comment

mykola-mokhnach commented Nov 14, 2023 •

edited

Loading

Dan-Maor commented Nov 15, 2023

mykola-mokhnach commented Nov 15, 2023 •

edited

Loading

Dan-Maor left a comment

Dan-Maor Nov 16, 2023

refactor: Optimize screenshots preprocessing #812

refactor: Optimize screenshots preprocessing #812

Conversation

mykola-mokhnach commented Nov 12, 2023

mykola-mokhnach commented Nov 12, 2023 • edited Loading

Dan-Maor commented Nov 12, 2023

KazuCocoa commented Nov 12, 2023 • edited Loading

mykola-mokhnach commented Nov 12, 2023

KazuCocoa commented Nov 12, 2023 • edited Loading

mykola-mokhnach commented Nov 12, 2023

KazuCocoa commented Nov 13, 2023

KazuCocoa commented Nov 13, 2023

Dan-Maor left a comment

Choose a reason for hiding this comment

mykola-mokhnach commented Nov 14, 2023 • edited Loading

Dan-Maor commented Nov 15, 2023

mykola-mokhnach commented Nov 15, 2023 • edited Loading

Dan-Maor left a comment

Choose a reason for hiding this comment

Dan-Maor Nov 16, 2023

Choose a reason for hiding this comment

mykola-mokhnach commented Nov 12, 2023 •

edited

Loading

KazuCocoa commented Nov 12, 2023 •

edited

Loading

KazuCocoa commented Nov 12, 2023 •

edited

Loading

mykola-mokhnach commented Nov 14, 2023 •

edited

Loading

mykola-mokhnach commented Nov 15, 2023 •

edited

Loading