Using OpenCV C++ code with Godot 3.5.2 on Ubuntu 20.04

Godot example with C++ and OpenCV functions in GDNative.

Purpose:

• Acquire video with a webcam
• Process frames with OpenCV
• Display output in AR/VR headset

Referred from: https://docs.godotengine.org/en/3.5/tutorials/scripting/gdnative/gdnative_cpp_example.html

How to use this repository?

1. Clone
2. Compile godot-cpp
3. Compile our OpenCV code with Godot wrapper
4. Using our code to access webcam in a 2D scene | Watch Video
5. Showing the webcam frames in a 3D scene | Watch Video
6. Displaying the OpenCV output in HMD | Watch Video

Clone Recursive

git clone --recursive https://github.com/mgpadalkar/godot-cpp-opencv.git
cd godot-cpp-opencv
# git submodule update --init --recursive

P.S.: If you are downloading the .zip file, ensure that the godot-cpp folder if populated. See the structure below to be sure.

Structure that we now have (click to expand)

godot-cpp-opencv                  # the repository
├── godot-cpp                     #
│   ├── godot-headers             #
│   │   ├── <lots of files>       #
│   ├── <lots of other files>     #
├── demo                          # project folder
│   ├── bin                       #
│   │   ├── gd_opencv.gdnlib      #
│   │   ├── gd_opencv.gdns        #
│   ├── default_env.tres          #
│   ├── icon.png                  #
│   ├── Main2D.tscn               #
│   ├── Main2D.gd                 #
│   ├── project.godot             #
├── misc                          #
│   ├── set_opencv4.sh            # my opencv install settings
├── README.md                     #
├── SConstruct                    #
├── src                           # C++ code folder
│   ├── gd_opencv.h               # header for wrapper to use OpenCV in Godot
│   ├── gd_opencv.cpp             # wrapper to use OpenCV in Godot
│   ├── gd_opencv_library.cpp     # library for the wrapper
│   ├── own_opencv_processing.h   # our OpenCV header
│   ├── own_opencv_processing.cpp # out OpenCV code
└── ...

Compile godot-cpp

cd godot-cpp
scons platform=linux generate_bindings=yes -j8 
# It generates: godot-cpp/bin/libgodot-cpp.linux.debug.64.a

Compile our OpenCV code with Godot wrapper

• Compile OpenCV from source if needed.
  I had followed this for Ubuntu 20.04: https://gist.github.com/raulqf/f42c718a658cddc16f9df07ecc627be7

# This step requires opencv to be installed and available in the path.
# In my case, I had compiled OpenCV myself and set the path
# using the script misc/set_opencv4.sh
# (OpenCV was installed in ${HOME}/localInstalls/opencv-4.5.2/).
# I had to do `source misc/set_opencv4.sh` to set the paths. 

cd ..
scons platform=linux

# It generates: demo/bin/x11/libgdopencvexample.so

Now use Godot (I used v3.5.2) to use the generated library.
Note: If OpenCV is built from source, like in my case, start Godot from the terminal in which the OpenCV paths are already set so that the OpenCV shared libs are available to Godot.

Using our code to access webcam in a 2D scene

I am using Godot 3.5.2 downloaded from https://github.com/godotengine/godot/releases/download/3.5.2-stable/Godot_v3.5.2-stable_x11.64.zip

The scene is already created using the following steps.
Simply run to view the webcam output in Godot.

1. Add a TextureRect to Main2D and rename it to webcamView
2. Add a Sprite to Main2D
3. Attach the bin\gd_opencv.gdns script to Sprite
4. Create a Main2D.gd script and attach it to Main2D
5. Connect frame_updated signal of Sprite to Main2D
6. Modify the Main2D.gd scripts' func _on_Sprite_frame_updated(node image) function with the following lines:

func _on_Sprite_frame_updated(node, image):
	var image_texture:ImageTexture = ImageTexture.new()
	image_texture.create_from_image(image)
	$webcamView.texture = image_texture

For more details, watch the following video.

Showing the webcam frames in a 3D scene

1. Create a new 3D Scene
2. Rename Spatial to Main3D
3. Save the scene as Main3D.tscn and set it as the main scene.
4. Go to AssetLib and search for xr
5. Download and install Godot XR Tools - AR and VR helper library
6. Go to Project -> Project Settings -> Plugins tab and enable the Godot XR Tools plugin.
7. Save and go to Project -> Reload Current Project
8. Add a WorldEnvironment to Main3D
9. Add a ARVROrigin to Main3D
10. Add a ARVRCamera to ARVROrigin
11. In FileSystem got to addons -> godot-xr-tools -> objects -> viewport_2d_in_3d.tscn, drag and drop it onto ARVRCamera
12. The above step creates a Viewport2Din3D.
    • Set Viewport Size to 640x480 and Transform to [0, 0, -1.63]
    • Drag and drop Main2D.tscn from FileSystem onto Scene for the Viewport2Din3D
    • Select Unshaded
13. In FileSystem got to addons -> godot-xr-tools -> xr -> start_xr.gd, drag and drop it onto ARVROrigin
14. Save the scene and run to view the output in the 3D scene.

For more details watch the following video.

Displaying the OpenCV output in HMD

1. Go to AssetLib and search for xr
2. Download and install OpenXR Plugin
3. Go to Project -> Project Settings -> Plugins tab and enable the Godot OpenXR plugin.
4. Save and go to Project -> Reload Current Project
5. Start your XR runtime:
   • If you have a XR runtime like Monado start with in a new terminal with monado-service
   • If you have SteamVR, start it so that your HMD is available
6. Now run the project. The output should be visible to you in your HMD.
   (In my case, I did not have a HMD attached, so the output was shown on screen in a Dummy HMD)

How I installed Monado? (click to expand)

# the following should be sufficient
sudo add-apt-repository ppa:monado-xr/monado
sudo apt-get update
sudo apt-get install libopenxr-loader1 libopenxr-dev libopenxr1-monado
sudo apt-get install xr-hardware libopenxr-utils openxr-layer-apidump monado-cli monado-gui

# if you need to build from source, see https://monado.freedesktop.org/getting-started.html#monado-installation
sudo apt-get install build-essential cmake libgl1-mesa-dev libvulkan-dev libx11-xcb-dev libxcb-dri2-0-dev libxcb-glx0-dev libxcb-icccm4-dev libxcb-keysyms1-dev libxcb-randr0-dev libxrandr-dev libxxf86vm-dev mesa-common-dev

# to run cretain demos on https://gitlab.freedesktop.org/monado/demos, you may also need the following
sudo apt-get install libsdl2-dev
sudo apt-get install libglm-dev
sudo apt-get install glslang-tools

For more details watch the following video.

How was this repository created?

1. Downloading the GDNative C++ Example | Watch Video
2. Downloading Godot 3.5.2 | Watch Video
3. Making an empty project | Watch Video
4. Writing our own OpenCV code | Watch Video
5. Creating the GDNative wrapper | Watch Video
6. Compiling our code + wrapper with SConstruct | Watch Video
7. Creating files necessary for using our code in Godot | Watch Video
8. Using our code to access webcam in a 2D scene | Watch Video

P.S.: All above videos are without audio.