Bad conversion in macOS

[jgranie]

Hi, under mac OS (not catalyst) I experience problems with translation. It seems that the Y coordinates is reversed.
I've made a quick and dirty test with RUISwitch scale 0.5, rotation 0, translation (0, 0.5, 0) and I have the button animation if I click between the middle and the bottom of the view while the button is drawn between the middle and the top.
I've tried to replace the conversion by event.locationInWindow on the "let touchInView =" lines for macOS, it seems to work better.
I believe it can't be so simple (I didn't test with rotations for example), so I hope you have enough information if you need to investigate.
Regards
Thanks for this code.

[maxxfrazer]

@jgranie thanks for reporting the issue! makes sense, as the macOS screen space starts at the bottom left instead of iOS's top left.

will update it soon and possibly add test files, as currently the tests are for iOS only.

[maxxfrazer]

I made a quick recording now, seems the touch events are passing through in the y axis okay, I must be misunderstanding your question. Could you post a clip of what's going wrong?

Screen.Recording.2023-04-07.at.12.06.26.mov

[jgranie]

Thanks for your reply, here's the recording

RUISwitch.mov

The code used to create the switch :
let ruiSwitch: RUISwitch = RUISwitch()
ruiSwitch.transform = Transform(
scale: .init(repeating: 0.1),
rotation: .init(), //.init(angle: .pi, axis: [0, 1, 0]),
translation: SIMD3(0.0, 0.5, 0)
)

When I use event.locationInWindow.y for the y coordinate instead of the converted y value it seems ok (I've tried many cases of translations and rotations and it works fine)
My modification (in 2 locations in the code mouseDown & mouseDragged):
let convertedPoint = self.arView.convert(event.locationInWindow, from: event.window?.contentView)
let touchInView = NSPoint(x: convertedPoint.x, y: event.locationInWindow.y)

Maybe I'm misusing something ?

[jgranie]

I used the top of the finder as a landmark to locate the position of the collision region, this is why it comes in front during the recording.

[maxxfrazer]

I've tried a lot of things, but I can't replicate.

This is my entire ViewController code:

class ViewController: NSViewController {

    override func viewDidLoad() {
        super.viewDidLoad()

        let arview = ARView(frame: self.view.bounds)
        self.view.addSubview(arview)
        arview.frame = self.view.frame
        let viewAnchor = AnchorEntity(world: .zero)
        let cam = PerspectiveCamera()
        cam.look(at: [0, -1, 0], from: [0, -1, 3], relativeTo: nil)
        viewAnchor.addChild(cam)

        RealityUI.enableGestures(.all, on: arview)
        let toggle = RUISwitch()
        viewAnchor.addChild(toggle)

        arview.scene.addAnchor(viewAnchor)
    }
}

I tried resizing the window, various sizes for the ARView etc too:

Screen.Recording.2023-04-07.at.17.57.55.mov

Is there any step I'm not doing here that would cause the issue? I'm running macOS 13.1 and Xcode 14.3.

[jgranie]

Sorry, while reading your ViewController code I realize that I forgot to mention a detail that may be important, I’m using RUISwitch in an NSViewRepresentable with SwiftUI

[jgranie]

I've created a fresh project template Augmented Reality App iOS, added destination Mac (macOS) and removed mac iPad.
Then I change the ContentView code as follow:

import SwiftUI
import RealityKit
import RealityUI

struct ContentView: View {
    var body: some View {
        ARViewContainer()
    }
}

struct ARViewContainer: NSViewRepresentable {
    typealias NSViewType = ARView
    
    func makeNSView(context: Context) -> ARView {
        let arview = ARView(frame: .zero)
        let viewAnchor = AnchorEntity(world: .zero)
        let cam = PerspectiveCamera()
        cam.look(at: [0, -1, 0], from: [0, -1, 3], relativeTo: nil)
        viewAnchor.addChild(cam)
        
        RealityUI.enableGestures(.all, on: arview)
        let toggle = RUISwitch()
        viewAnchor.addChild(toggle)
        
        arview.scene.addAnchor(viewAnchor)
        
        return arview
    }
    
    func updateNSView(_ nsView: ARView, context: Context) {
    }
}

And I can reproduce the issue I've reported, I'm new to iOS/macOS development and I didn't use NSViewController before, is it mandatory ?

[maxxfrazer]

You're making an ARView with SwiftUI the right way, no worries there!

I'll try it out today or tomorrow, but looks it may need to add an invert option in the enableGestures method to get it working on all platforms/environments… that's a pain! Well done finding it though.

[maxxfrazer]

Seems like changing both convert methods to this works for all cases:
self.arView.convert(event.locationInWindow, from: nil)

Going to test some more scenarios, and will add it to my PR and get that wrapped up this weekend.

[maxxfrazer]

Ran through a few scenarios of HStacks and VStacks with SwiftUI, similarly offset with AppKit on macOS and all seems well now!
If you have any further issues, please feel free to reopen this issue or create a new one.

[jgranie]

It works fine for me. Thanks again for your support and for your code very clean and well documented, I've learned a lot by reading it.