Can FlaUI run in Java?

[coolalexhuang]

Hi Roemer, as most of our projects are built in Java, can FlaUI run in Java project?

[Roemer]

Depends on what UI Framework you're using. You can try inspecting it with any Inspect tool (like FlaUInspect). Whatever you see there can be automated with FlaUI.

[maxinfet]

Did you mean "Can I use the FlaUI dlls from Java?" or did you mean "Can I automate a Java UI using FlaUI?" If you are asking about using the FlaUI dlls from the JVM I am not sure how dll interop works from the JVM to managed .NET dlls. If you are asking about automating a Java UI using FlaUI this should be possible based on the implementation of the UI framework. I was able to automate a Jave UI a long time ago using AutoIT and Swing. AutoIT just uses windows messages to automate the UI so I am not sure if Swing or other frameworks for that matter implement all the correct functionality for UIAv2 or UIAv3.

I will throw together a small JavaFX and Swing UI to see what they look like in inspect.

JavaFX

So JavaFX is a little odd since it sets all the automation ids to "JavaFX{1..n}" Here is a screen shot of what it looks like in inspect and a screen shot of the test application. I tried using invoke from inspect on the button but it did not click it even though it says it supports the invoke pattern. Also when adding an removing controls it is changing the AutomationID which is not ideal. This would mean using the Name property is the way to go since that can be set using setAccessibleText.

Here is some code I wrote for setting a few different properties on a button and seeing what kind of change it made in inspect.

Here is my test JavaFX app

package sample;

import javafx.application.Application;
import javafx.fxml.FXMLLoader;
import javafx.scene.AccessibleRole;
import javafx.scene.Parent;
import javafx.scene.Scene;
import javafx.scene.control.Alert;
import javafx.scene.control.Button;
import javafx.scene.layout.FlowPane;
import javafx.stage.Stage;

public class Main extends Application {

    @Override
    public void start(Stage stage) throws Exception{
        Parent root = FXMLLoader.load(getClass().getResource("sample.fxml"));
        stage.setTitle("Hello World");

        Scene scene = new Scene(root, 300, 275);

        Button button1 = createButton("1");
        Button button2 = createButton("2");

        FlowPane flow = new FlowPane();
        flow.setVgap(8);
        flow.setHgap(4);
        flow.setPrefWrapLength(300);
        flow.getChildren().add(button1);
        flow.getChildren().add(button2);

        scene.setRoot(flow);
        stage.setScene(scene);
        stage.show();
    }

    public static Button createButton(String id) {
        Button button = new Button();
        button.setId("id");
        button.setText("text" + id);
        button.setAccessibleRole(AccessibleRole.BUTTON);
        button.setAccessibleText("accessible-text");
        button.setAccessibleHelp("accessible-help");
        button.setAccessibleRoleDescription("accessible-role-description");

        button.setOnMouseClicked(event -> {
            Alert alert = new Alert(Alert.AlertType.INFORMATION);
            alert.setTitle("Message Here...");
            alert.setHeaderText("Look, an Information Dialog");
            alert.showAndWait();
        });

        return button;
    }

    public static void main(String[] args) {
        launch(args);
    }
}

Swing

Swing did not work at all with UIA. It will find the window but will not find any children within the window. I tested this with a test application I built and a accessibility demo application that oracle built which can be found here.

The swing application I built was also using some AWT just by the nature of how the Swing APIs are put together so I would imagine AWT would not work as well. Like I said previously though these will respond to windows messages quite well so if you are supporting a AWT or Swing application you could look into sending raw windows messages.

Here is my test Swing app

package sample;

import javax.accessibility.AccessibleContext;
import javax.swing.*;
import java.awt.*;

public class Main {

    private static void createAndShowGUI() {
        //Create and set up the window.
        JFrame frame = new JFrame("HelloWorldSwing");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

        JButton button1 = createButton("1", frame);
        JButton button2 = createButton("2", frame);

        Container container = frame.getContentPane();
        container.add(button1);
        container.add(button2);

        //Display the window.
        frame.pack();
        frame.setVisible(true);
    }

    private static JButton createButton(String id, JFrame frame) {
        JButton button = new JButton();
        button.setText("text" + id);

        AccessibleContext accessibleContext = button.getAccessibleContext();
        accessibleContext.setAccessibleName("accessible-name");
        accessibleContext.setAccessibleDescription("accessible-description");

        button.addActionListener(actionListener -> {
            JOptionPane.showMessageDialog(frame, "Message", "Title", JOptionPane.INFORMATION_MESSAGE);
        });

        return button;
    }

    public static void main(String[] args) {
        javax.swing.SwingUtilities.invokeLater(new Runnable() {
            public void run() {
                createAndShowGUI();
            }
        });
    }
}

[coolalexhuang]

@Roemer @maxinfet , I apologize to make you confused, my question is "Can I use the FlaUI dlls from Java?", if not, do you know any other libraries can do it?

[maxinfet]

In theory you should be able to consume the FlaUI dlls from Java. I am not familiar with how Java interop with managed .Net dlls works but it seems like their are some libraries to help with that.

EDIT
I removed my previous link after realizing it was for a paid piece of software and I started doing some research into alternatives for calling a .NET dll from Java.

After doing a little more research into this it looks like you would have to build a C++ native dll that wraps the .NET dlls of FlaUI so that you can call into them. Jni4net seems to be an interesting solution since it actually generates the wrapper classes for you but it looks pretty messy still.

@Roemer I could more than likely throw together these wrappers using jni4net or some other tool for doing this if you were interested in making this available on some Java package management system. Would you accept a pull request for adding the wrappers and a powershell script for generating them and some type of package for Java?

I think it is worth noting that my initial investigation with a C# test harness attempting to automate a JavaFX application showed that the support is very sparse. I think FlaUI would be usable by using the move and click with the mouse but it doesn't seem like any of the patterns work at all. I don't know how you feel about making FlaUI available when many of the APIs in FlaUI wouldn't work property with the JavaFX controls.

[Roemer]

@maxinfet Sure, the wrappers could be in an own project probably. Although I'm not sure how much overhead it is to maintain them.

[Roemer]

Closed due to inactivity.