Use of Compose over (on top of) Swing/AWT Heavyweight component

[yaroslavkulinich]

Problem

Very often in modern user interfaces, you need to place some UI elements over the main component of the application, like 'locate to my position' button over the Map component (in GoogleMaps), or 'play/pause' button over VideoPlayer. It's common practice.
But there is NO normal way of using jb-compose components over Swing/AWT Heavyweight components like 3D Maps, video players, etc... When you try to use the normal method of layering (Box composable) with Heavyweight Swing/AWT components, you will never see your Compose overlay content, because the Heavyweight Swing/AWT component always redraws your Compose layer.

Temporary solution

After many failed attempts I found the way of using Compose over Heavyweight Swing/AWT component (in my case it's 3DMap - WoldWindGLCanvas component). The trick is to use Swing/Compose switching for hierarchy root and then use Box component overlaying with 1) Heavyweight Swing/AWT component wrapped in SwingPanel and 2) Compose content wrapped into SwingPanel and switched back to ComposePanel (Swing/Compose switching).
So the hierarchy looks like this:

PSEUDO-CODE

SwingPanel {
   factory = ComposePanel {
       Box {
            SwingPanel{
               factory = { YourHeavyweightSwingAWTComponent() } // <--- 3D Map in my case
            }
            SwingPanel {
              factory = ComposePanel {
                 YourComposableOverlay() // <--- Your overlay
              }
            }
       }
   }
}

You can find the working sample in this repository in the master branch. And you can watch how it works on this YouTube video. I used beta5 version of jb-compose.

But starting from version 1.0.0-beta6-dev455 of jb-compose the temporary solution is not working.
You see the overlay, but you can't do any actions with it (all mouse events not recognized).
You can find described NOT working behavior in this repo in broken branch. And watch how it looks like on this YouTube video. I used the rc3 version of jb-compose.

Request

Dear jb-compose developers. @igordmn .
Please add the ability to use Compose components over Heavyweight Swing/AWT components in the normal way with Box component or similar. It would be a huge step! It's a missing piece for many devs.

If there is no possibility to make it the normal way, please, return the ability to use Compose over Heavyweight components like described in my temporary solution. Because after 1.0.0-beta6-dev455 it doesn't work. And in my personal case, I need to stay on beta5 or use ugly UI techniques without overlaying.

Thank you. You do a great job!

[yaroslavkulinich]

In addition to this, when we try to make overlays on top of Heavyweight components on Mac we can make it transparent but on Windows, it always gives you the White background.
As result, you can't make semi-transparent overlays with rounded corners on Windows.

[yaroslavkulinich]

Okay. I went deeper. Switched to 1.0.0. The same problem with no pointer event responses.
But this time, in addition, I have an Exception in the console when launching the sample (app not crashes):

2021-12-14 13:50:54.802 java[11986:857142] [CAMetalLayer nextDrawable] returning nil because allocation failed.
2021-12-14 13:50:54.953 java[11986:857142] [CAMetalLayer nextDrawable] returning nil because allocation failed.
Exception in thread "AWT-EventQueue-0" java.lang.RuntimeException: Can't wrap nullptr
	at org.jetbrains.skia.impl.Native.<init>(Native.jvm.kt:40)
	at org.jetbrains.skia.impl.Managed.<init>(Managed.jvm.kt:5)
	at org.jetbrains.skia.impl.Managed.<init>(Managed.jvm.kt:5)
	at org.jetbrains.skia.BackendRenderTarget.<init>(BackendRenderTarget.kt:8)
	at org.jetbrains.skiko.redrawer.MetalRedrawer.makeRenderTarget(MetalRedrawer.kt:123)
	at org.jetbrains.skiko.context.MetalContextHandler.initCanvas(MetalContextHandler.kt:36)
	at org.jetbrains.skiko.context.ContextHandler.draw(ContextHandler.kt:49)
	at org.jetbrains.skiko.redrawer.MetalRedrawer.performDraw(MetalRedrawer.kt:98)
	at org.jetbrains.skiko.redrawer.MetalRedrawer.access$performDraw(MetalRedrawer.kt:14)
	at org.jetbrains.skiko.redrawer.MetalRedrawer$draw$2$1.invokeSuspend(MetalRedrawer.kt:82)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
Exception in thread "AWT-EventQueue-0" java.lang.RuntimeException: Can't wrap nullptr
	at org.jetbrains.skia.impl.Native.<init>(Native.jvm.kt:40)
	at org.jetbrains.skia.impl.Managed.<init>(Managed.jvm.kt:5)
	at org.jetbrains.skia.impl.Managed.<init>(Managed.jvm.kt:5)
	at org.jetbrains.skia.BackendRenderTarget.<init>(BackendRenderTarget.kt:8)
	at org.jetbrains.skiko.redrawer.MetalRedrawer.makeRenderTarget(MetalRedrawer.kt:123)
	at org.jetbrains.skiko.context.MetalContextHandler.initCanvas(MetalContextHandler.kt:36)
	at org.jetbrains.skiko.context.ContextHandler.draw(ContextHandler.kt:49)
	at org.jetbrains.skiko.redrawer.MetalRedrawer.performDraw(MetalRedrawer.kt:98)
	at org.jetbrains.skiko.redrawer.MetalRedrawer.access$performDraw(MetalRedrawer.kt:14)
	at org.jetbrains.skiko.redrawer.MetalRedrawer$draw$2$1.invokeSuspend(MetalRedrawer.kt:82)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

[yaroslavkulinich]

compose-overlay-bug.2021-12-14.14.06.25.mov

After some testing, I found that pointer-events are processed by overlay, but only when you trigger recomposition.
You can see the behavior on video.

[yaroslavkulinich]

screen-capture.2021-12-14.14.40.15.mov

I just commented Swing panel with the Heavyweight component and got the same behavior. So the problem with my temporary solution of using overlays over heavyweight components is related to the use of ComposePanel inside SwingPanel. In this video, you can see that we can't interact with ComposePanel content inside SwingPanel.

@akurasov, please pay attention to these details. Maybe you have some solution or workaround for this case.

[zeruth]

This single problem is one of the most annoying things about Compose for me. Drawing on top as a default/only behavior is kind of crappy and feels like a half implementation.

EDIT: I feel as though I should say Compose is great, this is a huge annoyance, but It really is great work.

[akurasov]

Updated https://github.com/JetBrains/compose-jb/tree/master/tutorials/Swing_Integration to cover this case

[yaroslavkulinich]

@akurasov Great! Thanks! Will try it with the heavyweight 3d-Map Swing component soon.
Will leave a feedback with the results here.

Please tell from which version of Compose the documented overlaying approach is available?

[zeruth]

@akurasov Great! Thanks! Will try it with the heavyweight 3d-Map Swing component soon. Will leave a feedback with the results here.

Please tell from which version of Compose the documented overlaying approach is available?

They didn't fix the issue of rendering compose over a SwingPanel. They just made a semi informative page about it and said literally "or stop using the SwingPanel and still try to implement the missing component, thereby contributing to the development of technology and making life easier for other developers.".

hahaha such a smug response that didnt help the issue AT ALL, basically just said not our problem when IT IS.

You basically automatically lose the ability to Scaffold as well as many other typical features when using Swing in Compose if the SwingPanel happens to be your main component. 👎

Might as well not support Swing at all if you are going to have a broken implementation.

[dragossusi]

@zeruth They released a stable version of compose for desktop(and for web using dom) and they are now working hard on bringing this to ios and web(using canvas). They also have to reimplement base compose(by google for android) to support the latest changes.

I have used compose with SwingPanel and the issue is still there, I know it sucks, but I live with it for now(I only use it for 3D scenes).

I think this is not a rude comment from Jetbrains, but pushing a fix for this into the future, as everybody is more excited and waits for multiplatform support.

[mahozad]

I think this issue should not be closed.

Another use case is to overlay controls on the current experimental video player.

[ionull]

Any update on this? Since there is no official WebView on the desktop. We can now only one choice: JavaFx WebView and can only use it on SwingPanel?

[ionull]

I use another SwingPanel in PoupUp, and now it can pop over the SwingPanel with JavaFx WebView. But seems that the theme color is wrong. There is a background color with SwingPanel, how can we make it TRANSPARENT?

[MatkovIvan]

Any update on this?

JetBrains/compose-multiplatform-core#915

There is a background color with SwingPanel, how can we make it TRANSPARENT?

There is no such way due to heavyweight/lightweight AWT components mixing limitations

[yaroslavkulinich]

@MatkovIvan @igordmn
Just tested the basic case with both raw Composable and Composable wrapped with SwingPanel+ComposePanel displayed as overlay on top of the Heavyweight 3D-map (WorldWind) Swing component (wrapped in SwingPanel). Version 1.6.0-rc03.
Unfortunately the situation with Heavyweight component became worse and now I can't use overlays as before even with hacks.

The version 1.6.0-alpha01 is the last version where I controlled z-order mixing Heavyweight component with SwingPanels+Compose overlays. The trick was to show the SwingPanels in the Box element AFTER (some fixed milliseconds period) Heavyweight component is rendered. It was good because I could optionally show/hide the overlaying SwingPanels and had flexible UI. But starting 1.6.0-beta01 I lost this possibility. The only way I can show SwingPanel overlays over the 3D-map after hiding it is to hide/show the actual Heavyweight compoent(3D-map). It seems like the order of rendering changed or something.

In cases described above I don't use compose.interop.blending and compose.swing.render.on.graphics flags. Because in some cases and another OS's it becomes more complicated. I tested with flags - there'is no helpful impact for Heavyweight components case.

Additional comments:
On Mac: 1) there's no way to show SwingPanel over Heavyweight component at all when compose.swing.render.on.graphics = true. 2) There are positioning problems (rendering problem) with SwingPanel content when use without flags changing the layout. 3) Rendering order is OK - I can hide/show SwingPanel overlays.

On Windows: 1) Heavyweight component is not rendered (only parent's JPanel background visible) if blending = true. 2) Generally can't hide/show SwingPanel overlays - it's possible only by hiding/showing Heavyweight component.

On Ubuntu: 1) Same as on Windows - can't hide/show SwingPanel overlays

As a result, the situation with current issue became better only if we talk about regular Lightweight Swing components. But with Heavyweight component it became worse.

You can find the project with sample application where you can play with elements placed side-by-side and over the Heavyweight WorldWind 3D-map component. There are 2 branches: compose-1.5.12 where I can hide/show overlays and compose-1.6.0-rc03 where there's no such possibility.

[MatkovIvan]

Hi @yaroslavkulinich, thanks for the update!

The version 1.6.0-alpha01 is the last version

It's essentially just 1.5.x + WASM, so it didn't contain any code changes for 1.6.x

there's no way to show SwingPanel over Heavyweight component at all when compose.swing.render.on.graphics = true

That's tricky. On macOS I managed to overlap GLCanvas only via another not just "heavyweight" but directly GPU managed component. compose.swing.render.on.graphics switches rendering from display to texture and then renders it as regular Swing graphics, so it seems logical that it behaves like that.

There are positioning problems (rendering problem) with SwingPanel content when use without flags changing the layout

Yeah, it's unrelated, known one - there are some cases where ComposePanel doesn't update position. It's a bug of the "skiko" library that we're using under the hood. It seems the problems on Windows/Ubuntu you mentioned are also because of this. This one should be fixable - it's in my mid-term TODO list

Heavyweight component is not rendered (only parent's JPanel background visible) if blending = true

It's a known limitation on windows - rendering by GPU just wipes everything that was drawn before via the same method. It's even reproducible via two overlapping ComposePanels. To explain why this map doesn't show, I have to explain what blending = true really does - it places interop (SwingPanels) under main Compose but cutting transparent-alpha hole in Compose canvas. So, it also seems "expected"

As you can see, there are a lot of OS specifics and hacks for this. The issue with GLCanvas here is that it's not just a "heavyweight" component, but another directly GPU-rendered view.

The only fixable thing from your post is missing re-draws of ComposePanel. I guess that you can check that it's not "can't hide/show", but missing re-draw by manually triggering it via resizing the window for example.

But aside from this long list of corner cases, it's totally not clear why you need to rely on GLCanvas if you need overlay above that. The mentioned library, WorldWind, provides WorldWindowGLJPanel that you can use instead WorldWindowGLCanvas - it should solve most of your problems.

[yaroslavkulinich]

@MatkovIvan thank you for the answer.
I understand that there are a lot of hacks and tricks, so I'm trying to understand what has to be changed migrating to 1.6.*.

Thanks for pointing me to WorldWindowGLJPanel - I tried to use it earlier but it didn't work properly with Compose, so I have chosen the WorldWindowGLCanvas. But now I reviewed it again and see that it works, and even gives better results in the context of 1.6.* migration, blending, overlaying.

I have tested WorldWindowGLJPanel by playing with blending and graphics flags and already have some results.
Same code, WorldWindowGLJPanel instead WorldWindowGLCanvas, Button added inside raw Compose overlay to check clicks (only on Mac and Windows). Box composable has two elements: SwingPanel(WorldWindowGLJPanel) and Column with SwingPanel(ComposePanel(Composable)) and Compose overlays.

Ubuntu:

Expectations: We do not expect to see raw Compose overlay over the map because there's no implemented blending functionally for Linux, but want to see SwingPanel over the Map when it is rendered AFTER the Map component.
Results: SwingPanel is always overlayed by the Map when rendered after the Map. Hide/Show actions for SwingPanel give no effect. The only way to show SwingPanel overlay is to Hide/Show the map element. It feels like the order of rendering is reversed and it's not logical from the usage perspective + it's a huge limit in UX, because you can't show overlays dynamically over the Map. Resizing of the window gives nothing.

Screencast.from.26-02-24.19_26_46.webm

Windows:

2 tests were made: with and without blending flag. graphics flag provides additional problems - so was not used.
Without blending: A result is completely the same as on Ubuntu - "reversed rendering order".

windows-no-flags.mov

With blending: Surprisingly good results. Raw Compose overlay is visible and clickable, always - no matter when rendered. SwingPanel isn't, but who cares when you can use just normal Composable without SwingPanel wraps. This result it the expected final goal of this GitHub issue. Performance of the blending approach will be checked later.

windows-blending.mov

MacOS:

2 tests with and without blending flag.
Without blending: Behavior is the same as in 1.5.* versions - SwingPanel overlay rendered AFTER the Map will be shown OVER the Map as expected. The only difference is that SwingPanel overlay is always shown - no matter what rendering order but clickable only when rendered AFTER the Map.

mac-no-flags.mov

With blending: Same result as without blending but non-clickable row Compose overlay shown. So you can draw something over the Map but without interactions.
Positioning/Rendering problems is out of scope.

mac-blending.mov

Summary for 1.6.* migration:

• Windows: We should use overlays as row Composables with blending.
• MacOS: We should use SwingPanel overlays as before with or without blending.
• Ubuntu: We have a problem with "reversed order of rendering" and can't use dynamic overlays anymore - is a blocker for this migration. Some hack/trick/fix is needed.

[MatkovIvan]

So you can draw something over the Map but without interactions.

Another hack needed 😅 : JetBrains/compose-multiplatform-core#915 (comment)

As for order of adding/rendering - I already see how to impllement interop order not by adding but by render order, I'll defenitely recheck your case during the fix (no ETA though). An issue for tracking it #2926

[MatkovIvan]

Actually restoring the order based on adding like it was before is easy. Made it in JetBrains/compose-multiplatform-core#1143

[MatkovIvan]

@yaroslavkulinich
Restoring 1.5.x behaviour is released in 1.6.1
Proper fix that sorts interop views based on Compose hierarchy is available in a few latest dev builds (1.6.10-dev1514 is the latest at the moment).

Let me know if you still have issues with it

[yaroslavkulinich]

@MatkovIvan Great news! Thank you!
I will test as soon as I can. I believe that everything should be ok=)
One more thing I want to ask is how to workaround that positioning+rendering SwingPanel problem on MacOS? Because It present in 1.5.* versions and I need to do very dirty trick patching SwingPanel to trigger additional revalidation and redraw (simulate window size change) to deal with it. Any help would be appreciated.
Thank you.

[MatkovIvan]

I've added a few force invalidations to cover some cases, but I know that there are more. Not sure what case exactly you mean. It's better to track it as a separate issue with simple reproduction.

[yaroslavkulinich]

@MatkovIvan thanks again for your insights. I need your opinion about about new theoretical approach of better overlaying over Heavyweight components.

The WorldWindowGLJPanel, you have suggested, with blending feature perfectly deals with overlaying, rounded corners, and transparency. However, WorldWindowGLJPanel performance is noticeably poorer compared to the heavyweight WorldWindowGLCanvas, even during simple map dragging tasks, which prevents us from replacing the GLCanvas entirely. So we fallback to old overlaying approach.

In the recent 1.6.* versions of Compose Multiplatform, I noticed the introduction of a WINDOW layer, mainly for handling dialogs and popups. I'm curious if this feature could be used to improve our overlaying of heavyweight components. I'm thinking of using popups (or similar elements) fixed over the main UI containing the heavyweight component. For this to work, we would need:

• The ability to interact with elements behind the popups.
• Popups to maintain their positioning relative to the window when it moves (without resizing).
• The use of multiple popups simultaneously.

Could the WINDOW layer potentially solve our problem with proper overlaying? Your thoughts or any alternative suggestions would be invaluable.

***Additional aspect and reason for searching new overlaying approach for us:
We've encountered a problem with old overlaying technique described in this issue using ComposePanel/SwingPanel wrappers. Every wrapper increases RAM usage with each added layer. Our interface uses these wrappers constantly for displaying buttons and other elements over the heavyweight map - we have around 5-6 constantly displayed wrappers. And it leads to additional 100-200mb RAM usage (content of wrappers doesn't matter, it can be empty wrappers).

Thank you

[MatkovIvan]

The ability to interact with elements behind the popups.

It will work only for out-of-bounds interactions (currently Popup blocks outside clicks in case of focusable flag)

Popups to maintain their positioning relative to the window when it moves (without resizing).

Works with some delays caused by AWT

The use of multiple popups simultaneously.

Just works

Could the WINDOW layer potentially solve our problem with proper overlaying?

Please note that it will be real separate OS windows, with some side effects like participating in OS window switcher. So, I'm not sure that this is what you want. But it will solve sorting issues for sure. Also, I doubt that it will require less RAM