Running the application

Quick start

This github repo contains small example datasets. They can be used to test if the application is working correctly. To run an example, you might need to alter the paths of options -i and -j to point to where the example is stored on your machine relative to the executable:

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json

The axial system used in example_opengl.json is that of OpenGL. In the examples/ folder, there are json files for the 3 axial systems that OpenDIBR supports: OpenGL, Colmap and OMAF (of MPEG).

I recommend using --triangle_deletion_margin as well, see Command line examples. To learn more about how to navigate around the scene and change important settings in the application, read this.

If you run into unexpected problems, please read Common Problems before opening an issue.

All command line options

The supported command line options are described in the tables below. Note that if the type is a bool, the option does not need to be followed by true or false on the command line, as illustrated in the Command Line Examples.

argument	type	description	default	required
-i, --input_dir	string	Path to the folder that contains the light field images/videos		Y
-j, --input_json	string	Path to the .json file with the input light field camera parameters		Y
--vr	bool	Render the output to a VR headset		N
--static	bool	The input light field consists of PNGs, or of videos where only the '--framenr' frame needs to be decoded		N
--frame_nr	int	The frame that needs to be shown if the input light field consists of videos and option '--static' is set	0	N
--target_fps	int	The target application fps in case of video inputs. Needs to be a multiple of 30, which is the assumed framerate of the videos.	90	N
--max_nr_inputs	int	The maximum number of input images/videos that will be processed per frame (-1 if all need to be processed)	-1	N
--show_inputs	bool	This setting will display the positions and rotations of the input and output cameras on screen, as well as which inputs are used to render the current frame.		N
-t	int	The number of threads used by the thread pool to decode the MP4 videos (if --static is not defined). Should be >= 2. Recommended: #CPU cores - 1	2	N
--asap	bool	Decode and play the image/video frames as soon as possible (basically disabling the Vsync@90Hz)		N
--mesh_subdivisions	int	The detail level of the triangle meshes, full resolution if 0, 1/2 resolution if 1, 1/3 resolution if 2, etc. Must lie in [0,5]	0	N
--blending_factor	int	The higher this factor, the more blending between inputs there is, as an int in [0,10]	1	N
--triangle_deletion_margin	float	The higher this value, the less strict the threshold for deletion of stretched triangles. Should be > 1. Note that this threshold loosens up for larger depth values, where the depth map is less accurate.	10.0	N
--cam_speed	float	The speed at which the GUI camera moves around	0.01	N

Command line options related to saving output images and/or performance stats to disk:

argument	type	description	default	required
-p, --output_json	string	Path to the .json file with the camera parameters for which the output image needs to be saved to disk		N(*)
-o, --output_dir	string	Path to the folder where the output will be saved		N(*)
--fps_csv	string	Path to the .csv file to write the time needed to render each frame to		N

(*) Options -p (or --output_json) and -o (or --output_dir) belong together, meaning that either both need to be defined or neither. When they are defined, what is rendered to the display will also be saved to a YCbCr file, with pixel format yuv444p.

Command line options to control the output image:

argument	type	description	default	required
--background	3 ints	The RGB color of the background, as 3 ints in [0,255]	128,128,128	N

Command line examples

I recommend playing around with --triangle_deletion_margin, since each scene has a different optimal value. For Fan and Painter, the default value of 10 is fine though.

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --triangle_deletion_margin 10
RealtimeDIBR -i ../../examples/Painter -j ../../examples/Painter/example_opengl.json --triangle_deletion_margin 10

To see the application in Virtual Reality using SteamVR and a SteamVR-supported HMD:

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --vr

To freeze the videos in time on a specific frame (starting from zero), e.g. 50, do:

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --static --frame_nr 50

To only use a subset of the input cameras for each frame, e.g. 2, execute the command below. This is useful to speed up performance when using larger datasets. The application decides for each frame which 'max_nr_inputs' input views are most useful to render the current output view and only processes these. Note that in the case of videos, that still means that all frames of all videos need to be decoded, so do not expect a linear performance improvement.

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --max_nr_inputs 2

Say you want to save the output images for specific viewpoints and camera intrinsics. First you need to make a JSON file similar to example_output.json that instead of input cameras, contains the output cameras for which you want to render and save the results. You also need to specify where you want the output images to be saved and make sure this folder exists. For this example, we will simply save everything to the directory with the executable, i.e. ".". The result is a .yuv file that contains a sequence of the output frames in YCbCr yuv444p format.

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json -p ../../examples/Fan/example_output.json -o .

To measure the time (in milliseconds) it takes to render each frame, you need to specify the path to a CSV file:

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --fps_csv ./milliseconds_per_frame.csv

Output camera with a nice blue background color:

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --background 135,206,235

Navigation in the application

While the application is running (not in --vr mode), the viewer can navigate through the scene as follows:

• Keyboard: WASD (left,up,down,right) and QZ (forward,backward).
• Mouse: hold down the middle mouse button (left,up,down,right) and move the mouse. Scroll the mouse wheel to go forward ord backward. You can rotate the camera by holding down the left mouse button and move the mouse.

Other useful keyboard inputs (that will also work in --vr mode):

• Esc to exit the application.
• G and H to decrease/increase --triangle_deletion_margin in steps of 0.5.
• C and V to slow down/speed up the camera.
• B and N to decrease/increase the blend_factor. The blend_factor will be clipped to [0, 10].
• R toggles the control of the small window in the bottom right corner created through option --show_inputs on the command line. So after tapping R once, the mouse buttons and wheel can be used to control the camera movement in the small window.

When in --vr mode, the joysticks or trackpads on the VR controllers can be used to fly around. Both controllers will be necessary. The movement is relative to the orientation of the head, not relative to the scene. This might take some getting used to.

Common problems

1. Option <> does not exist or Argument <> starts with a - but has incorrect syntax. This occurs when an option should be preceded by -- instead of -, for example:

RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json -static // Returns: Option 's' does not exist
RealtimeDIBR -i ../../examples/Fan -j ../../examples/Fan/example_opengl.json --static // Works

2. The scene does not look as expected, like in the illustration below. This means that you should try a different value for triangle_deletion_margin, which determines how triangles are discarded based on their circumference.