What is this project

This is a fully featured, free*, functional Multiplayer VR UNITY 6 project with a speech driven LLM NPC. Features:

• Netcode for GameObjects Host and Client option
• Ad-hoc join capability with Just In Time client/server protocol and a server repository to synchronize prefabs across all clients post app launch
• 2-Stage Client Sign-In process with freedom to build and use your own rigged avatars (FBX), no need to register and package the avatars in Network Manager due to the included Client-Server-Join protocol and ServerRepository object
• Hands, head and detailed finger tracking from your XR system to your avatars
• 4 preconfigured avatars (male/female, human/bot), easily replace with your own avatars
• Auto-IK component to automatically configure all IK constraints of your avatars hands, head and fingers
• Checkbox for using a local or a web-based prefab/avatar resource repository
• Tool to generate asset bundles to store on the web-based repository
• 2 different object grab implementations
• First Person and Third Person mode via the UI (FP3P)
• Vivox Voice Services with Spatial audio (need to add your own credentials)
• Debug Console (3rd party free component) for getting debug information on standalone VR/XR systems
• OpenXR based so in principle platform-agnostic
• NPC with speech interface: using STT+LLM+TTS cloud services

(* => excluding paid cloud services for the NPC and assumes you don't exceed the Vivox Voice services complementary service tresholds)

What is new in this Branch

This branch is now running UNITY 6! Upgrading the branch did not work so I had to start with a completely new project and import all scripts, objects and components, and obviously fix a massive avalanche of script errors due to major changes in the OpenXR foundational components :(
Youtube video

Some additional new features in this version are:

• Added RAG-Lite (context awareness) to the NPC Youtube video
• Show/Hide the On-Screen menu by pressing both the Select and Trigger buttons on the LEFT controller
• Uses the new Unity 6 XR Origin with NearFar Interactor instead of the Unity 2022.3 XR Ray Interactor and XR Direct Interactor
• Uses the URP shaders vs the Standard shaders - allows for more accurate transparency settings
• Ability to RESIZE grabbed objects by grabbing them with 2 controllers
• Improved FP grabbing: Ability to throw grabbed objects in First Person mode, both standalone as in multiplayer mode
• The annoying XRInputModalityManager errors are fixed
• Uses the latest Vivox Voice services (partially free)

Steps to get started

1. Pull the branch
2. Open Unity Editor, when prompted about Errors, select Ignore and do not start in safe mode
3. Go to File -> Open Scene -> BaMMain
4. In the VivoxVoiceManager, enter your Vivox service credentials (if not used (yet), leave them blank)
5. Import the uLipSync package: Top Menu -> Assets -> Import Package -> Custom Package -> Navigate to the uLipSync package in this repository and import it
6. You may need to re-import the XR Interaction Toolkit samples and the XR Hands samples
7. If you want the InGame Debug Console then go to the Unity Store and purchase it (free)
8. If you want to talk with the NPC, be sure to get API keys from the following cloud providers: HuggingFace (free), GroqCloud (free), Speechify (paid), RapidAPI (partially free)
9. In Assets/Prefabs, click on the DynamicPrefabStarter object, select its ClientServerJoinProtocol script and ensure that UseWebRepository is UNCHECKED
10. Check the Platform setting under Unity Menu->File->Build Settings, if you want to compile for standalone VR (eg. Meta Quest 2,3) select Android otherwise select Windows
11. Open the Network Manager and change IP address of the host to the IP address of the device/PC that will act as the host

If you want to use remote prefabs

1. In Assets/Prefabs, click on the DynamicPrefabStarter object, select its ClientServerJoinProtocol script and ensure that UseWebRepository is CHECKED
2. Enter the url of your own webserver where you want the AssetBundles to be stored
3. In the Assets menu in Unity, select Build Asset Bundles
4. When that is completed, open a File Explorer and find your Assets folder, there is an AssetBundle folder with a windows and an android subfolder
5. ZIP the AssetBundle folder and transfer it to your webserver, unpack under the the url you entered in step 2

Compatibility

• Currently this code is tested for Windows and Meta Quest 2 and 3 (Pico 4 not yet!)
• If you want to add Mac with a Web Repository then you must update the Editor/AssetBundlesBuild script to generate Mac AssetBundles ...
• AND open Assets/Scripts/Network/ClientServerJoinProtocol.cs and add code in the OnNetworkSpawn() method (pretty straightforward)
• If you have another VR brand headset, you will need to configure that headset in Project Settings -> XR Plugin Management