Implement support for writing UI Automation Remote Operations nativel…

…y in NVDA using Python (#16214)

Several years ago Microsoft added a low-level API in Windows to allow the caller to execute multiple actions in a UI Automation remote provider via only one cross-process call.
Limiting the amount of cross-process calls is necessary for providing a responsive experience, especially for interacting with applications hosted in Windows Defender Application Guard (WDAG) or as a Remote Application Integrated Locally (RAIL). There are also scenarios such as searching for / listing headings in large documents, which can bennifit from limiting cross-process calls, even when the application is running on the local machine.
the Windows.UI.UIAutomation.Core.CoreAutomationRemoteOperation is a winRT interface that allows the caller to specify the actions as a string of byte code representing a specific set of instructions for a conceptual virtual machine which is executed within the remote provider. The instructions allow you to manipulate UI Automation elements, patterns and text ranges, as well as perform basic logic and collect information using ints, floats, strings, GUIDs and arrays created within the conceptual virtual machine.
Although these instructions are well-defined, public documentation is limited and programming with these raw instructions can be slow and complicated to read.
Microsoft released the Microsoft-UI-UIAutomation Remote Operations Library which provides some higher level ways of building and executing these instructions, including intermediate winRT interfaces, and high-level pure c++/winRT interfaces.
NV Access has made many prototypes with this library, and has made significant contributions to that project over the years.
Today, NVDA includes this library as a depedency, and currently uses it to access the new UI Automation Custom Extensions feature, which is required to access certain information in Microsoft Word. In theory, NVDA could make a much greater uuse of UI automation Remote Operations to improve performance across many parts of Windows and other applications.
However, there are several issues with the Microsoft UI Automation Remote Operations Library which slow down or limit greater usage in NVDA, including:

All remote operations logic must be written in C++ and not Python. This creates a large learning curve, if not a quite uncomfortable context switch at very least.
the Remote Operations Library seems to be no longer maintained by Microsoft.
The Remote Operations Library currently contributes an extra massive 16 minutes to NvDA's build time, as it must compile the whole of the C++/winRT API set.
Solution
This PR introduces a new remote operations library written in pure Python, which wraps the low-level CoreAutomationRemoteOperation winRT interfaces in Windows. This new library replaces the need for the older Microsoft Remote Operations Library, and therefore is no longer a dependency.

I had several initial goals in mind with this new library:

To support our current usage of remote operations (namely UI Automation custom extensions: isExtension and CallExtension). If I could not achieve our current requirements, there would be no point continuing.
To make a significant performance improvement to at least one extra action or scenario in NvDA (E.g. quick navigation by heading in Microsoft Word). This would prove that implementation of a fix / feature was easier with the new library, and would make it worth spending the initial time on it.
I was able to achieve both of these goals.

Description of user facing changes
In browse mode in Microsoft Word with UIA enabled, quick nav to headings, and listing headings in Elements list, is now up to 30 times faster. E.g. Jumping to a heading at the bottom of a 160 page document all the way from the top previously took ~4 seconds, now it takes ~0.13 seconds.
Description of development approach
A new UIAHandler._remoteOps package has been added, which interfaces with the low-level Windows Remote Operations API, providing higher-level APIs and types to aide in building and executing easy to read remote algorithms. See The NVDA remote operations readme for explanations and example code for all features.
UIAHandler.remote's msWord_getCustomAttributeValue has been re-written to use the new NVDA remote operations library.
Some extra functions were added to UIAHandler.remote, including findFirstHeadingInTextRange and collectAllHeadingsInTextRange, built on the NVDA remote ops library. UIA browse mode makes use of these functions on Windows 11, to majorly speed up jumping to / listing headings.

.gitmodules

@@ -30,9 +30,6 @@
 [submodule "include/nsis"]
 	path = include/nsis
 	url = https://github.com/nvaccess/nsis-build
-[submodule "include/microsoft-ui-uiautomation"]
-	path = include/microsoft-ui-uiautomation
-	url = https://github.com/michaeldcurran/microsoft-ui-uiautomation
 [submodule "include/wil"]
 	path = include/wil
 	url = https://github.com/microsoft/wil

include/readme.md

@@ -24,12 +24,6 @@ Fetch latest from master.
 ### liblouis
 TODO
 
-### microsoft-ui-uiautomation
-https://www.github.com/michaeldcurran/microsoft-ui-uiautomation/
-
-We are stuck on our own fork.
-The fork specifically adds support for connection bound objects to the high-level API, see PR microsoft/microsoft-ui-uiautomation#95.
-
 ### nsis
 [NSIS-build readme](https://github.com/nvaccess/NSIS-build)
 

nvdaHelper/UIARemote/lowLevel.cpp

@@ -0,0 +1,192 @@
+#include <uiAutomationClient.h>
+#include <winrt/windows.foundation.h>
+#include <winrt/windows.foundation.collections.h>
+#include <winrt/windows.ui.uiautomation.core.h>
+
+using namespace winrt::Windows::Foundation;
+using namespace winrt::Windows::UI::UIAutomation;
+using namespace winrt::Windows::UI::UIAutomation::Core;
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_create(void** ppRemoteOp) {
+	*ppRemoteOp = winrt::detach_abi(CoreAutomationRemoteOperation());
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_free(void* arg_pRemoteOp) {
+	CoreAutomationRemoteOperation operation {nullptr};
+	winrt::attach_abi(operation, arg_pRemoteOp);
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_importElement(void* arg_pRemoteOp, int arg_registerID, IUIAutomationElement* arg_pElement) {
+	CoreAutomationRemoteOperation operation {nullptr};
+	winrt::copy_from_abi(operation, arg_pRemoteOp);
+	if(!operation) {
+		return E_FAIL;
+	}
+	AutomationElement element {nullptr};
+	winrt::copy_from_abi(element, arg_pElement);
+	if(!element) {
+		return E_FAIL;
+	}
+	operation.ImportElement(AutomationRemoteOperationOperandId{arg_registerID}, element);
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_importTextRange(void* arg_pRemoteOp, int arg_registerID, IUIAutomationTextRange* arg_pTextRange) {
+	CoreAutomationRemoteOperation operation {nullptr};
+	winrt::copy_from_abi(operation, arg_pRemoteOp);
+	if(!operation) {
+		return E_FAIL;
+	}
+	AutomationTextRange textRange {nullptr};
+	winrt::copy_from_abi(textRange, arg_pTextRange);
+	if(!textRange) {
+		return E_FAIL;
+	}
+	operation.ImportTextRange(AutomationRemoteOperationOperandId{arg_registerID}, textRange);
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_addToResults(void* arg_pRemoteOp, int arg_registerID) {
+	CoreAutomationRemoteOperation operation {nullptr};
+	winrt::copy_from_abi(operation, arg_pRemoteOp);
+	if(!operation) {
+		return E_FAIL;
+	}
+	operation.AddToResults(AutomationRemoteOperationOperandId{arg_registerID});
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_isOpcodeSupported(void* arg_pRemoteOp, uint32_t arg_opcode, bool* pIsSupported) {
+	CoreAutomationRemoteOperation operation {nullptr};
+	winrt::copy_from_abi(operation, arg_pRemoteOp);
+	if(!operation) {
+		return E_FAIL;
+	}
+	*pIsSupported = operation.IsOpcodeSupported(arg_opcode);
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOp_execute(void* arg_pRemoteOp, uint8_t arg_byteCodeBuffer[], int arg_byteCodeBufferLength, void** ppResults) {
+	CoreAutomationRemoteOperation operation {nullptr};
+	winrt::copy_from_abi(operation, arg_pRemoteOp);
+	if(!operation) {
+		return E_FAIL;
+	}
+	auto results = operation.Execute(winrt::array_view<uint8_t>(arg_byteCodeBuffer, arg_byteCodeBufferLength));
+	*ppResults = winrt::detach_abi(results);
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOpResult_getErrorLocation(void* arg_pResults, int* pErrorLocation) {
+	AutomationRemoteOperationResult results {nullptr};
+	winrt::copy_from_abi(results, arg_pResults);
+	if(!results) {
+		return E_INVALIDARG;
+	}
+	*pErrorLocation = results.ErrorLocation();
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOpResult_getExtendedError(void* arg_pResults, HRESULT* pExtendedError) {
+	AutomationRemoteOperationResult results {nullptr};
+	winrt::copy_from_abi(results, arg_pResults);
+	if(!results) {
+		return E_INVALIDARG;
+	}
+	*pExtendedError = results.ExtendedError();
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOpResult_getStatus(void* arg_pResults, int* pStatus) {
+	AutomationRemoteOperationResult results {nullptr};
+	winrt::copy_from_abi(results, arg_pResults);
+	if(!results) {
+		return E_INVALIDARG;
+	}
+	auto status = results.Status();
+	*pStatus = static_cast<int>(status);
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOpResult_hasOperand(void* arg_pResults, int arg_registerID, bool* pHasOperand) {
+	AutomationRemoteOperationResult results {nullptr};
+	winrt::copy_from_abi(results, arg_pResults);
+	if(!results) {
+		return E_INVALIDARG;
+	}
+	*pHasOperand = results.HasOperand(AutomationRemoteOperationOperandId{arg_registerID});
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOpResult_free(void* arg_pResults) {
+	AutomationRemoteOperationResult results {nullptr};
+	winrt::attach_abi(results, arg_pResults);
+	return S_OK;
+}
+
+HRESULT IInspectableToVariant(winrt::Windows::Foundation::IInspectable result, VARIANT* arg_pVariant) {
+	if(!result) {
+		// operand is NULL.
+		return S_OK;
+	}
+	auto propVal = result.try_as<IPropertyValue>();
+	if(propVal) {
+		// Unbox property value into VARIANT
+		auto propType = propVal.Type();
+		switch(propVal.Type()) {
+			case PropertyType::Int32:
+				arg_pVariant->vt = VT_I4;
+				arg_pVariant->lVal = propVal.GetInt32();
+				break;
+			case PropertyType::String:
+				arg_pVariant->vt = VT_BSTR;
+				arg_pVariant->bstrVal = SysAllocString(propVal.GetString().c_str());
+				break;
+			case PropertyType::Boolean:
+				arg_pVariant->vt = VT_BOOL;
+				arg_pVariant->boolVal = propVal.GetBoolean() ? VARIANT_TRUE : VARIANT_FALSE;
+				break;
+			case PropertyType::Inspectable:
+				arg_pVariant->vt = VT_UNKNOWN;
+				arg_pVariant->punkVal = static_cast<::IUnknown*>(winrt::detach_abi(propVal.as<winrt::Windows::Foundation::IUnknown>()));
+				break;
+			default:
+				return E_NOTIMPL;
+		}
+		return S_OK;
+	}
+	auto vec = result.try_as<winrt::Windows::Foundation::Collections::IVector<winrt::Windows::Foundation::IInspectable>>();
+	if(vec) {
+		// Unbox vector into VARIANT array.
+		auto vecSize = vec.Size();
+		arg_pVariant->vt = VT_ARRAY | VT_VARIANT;
+		arg_pVariant->parray = SafeArrayCreateVector(VT_VARIANT, 0, vecSize);
+		if(!arg_pVariant->parray) {
+			return E_OUTOFMEMORY;
+		}
+		for(ULONG i = 0; i < vecSize; i++) {
+			auto vecItem = vec.GetAt(i);
+			auto hr = IInspectableToVariant(vecItem, &static_cast<VARIANT*>(arg_pVariant->parray->pvData)[i]);
+			if(FAILED(hr)) {
+				return hr;
+			}
+		}
+		return S_OK;
+	}
+	// Just treat it as an IUnknown.
+	arg_pVariant->vt = VT_UNKNOWN;
+	arg_pVariant->punkVal = static_cast<::IUnknown*>(winrt::detach_abi(result.as<winrt::Windows::Foundation::IUnknown>()));
+	return S_OK;
+}
+
+extern "C" __declspec(dllexport) HRESULT __stdcall remoteOpResult_getOperand(void* arg_pResults, int arg_registerID, VARIANT* arg_pVariant) {
+	AutomationRemoteOperationResult results {nullptr};
+	winrt::copy_from_abi(results, arg_pResults);
+	if(!results) {
+		return E_INVALIDARG;
+	}
+	auto result = results.GetOperand(AutomationRemoteOperationOperandId{arg_registerID});
+	return IInspectableToVariant(result, arg_pVariant);
+}