add vispy renderer to allow offscreen rendering with python

[wangg12]

Hi, in this pull request, I add a vispy version renderer to allow offscreen rendering with python. The code is mainly adapted from the glumpy version python renderer while the rendering engine is changed to vispy. glumpy and vispy have similar apis, however, vispy allows us to use egl for offscreen rendering.

I have done two types of tests:

(1) compare the difference of the rendered results using "cpp", "python", and "vispy" renderer:

from transforms3d.axangles import axangle2mat
import time

# Path to a 3D object model (in PLY format).
model_path = osp.expanduser("~/Storage/BOP_DATASETS/lm/models/obj_000001.ply") #'/path/to/a/ply/file'
obj_id = 1

renderer_type="vispy"  # "python"  | "cpp"
print(renderer_type)

# Path to output RGB and depth images.
out_rgb_path = f'{renderer_type}_out_rgb.png'
out_depth_path = f'{renderer_type}_out_depth.png'

# Object pose and camera parameters.
# R = np.eye(3)
# t = np.array([0.0, 0.0, 300.0])
R1 = axangle2mat((1, 0, 0), angle=0.5 * np.pi)
R2 = axangle2mat((0, 0, 1), angle=-0.7 * np.pi)
R = np.dot(R1, R2)
t = np.array([-0.1, 0.1, 0.7], dtype=np.float32) * 1000

fx, fy, cx, cy = 572.41140, 573.57043, 325.26110, 242.04899
width = 640
height = 480
################################################################################

# Initialization of the renderer.
ren = create_renderer(width, height, renderer_type=renderer_type)
ren.add_object(obj_id, model_path)
total_time = 0
runs = 1000
for i in range(runs):
    tic = time.time()
    res = ren.render_object(obj_id, R, t, fx, fy, cx, cy)
    total_time += time.time() - tic
print(f"{total_time*1000/runs}ms/obj {runs/total_time}Hz")
rgb = res['rgb']
depth = (10*res['depth']).astype("uint16")
# import ipdb; ipdb.set_trace()
print("write rgb")
imageio.imwrite(out_rgb_path, rgb)
print("write depth")
imageio.imwrite(out_depth_path, depth)

run the code for different types of renderers, and check the difference:

import mmcv
depth_factor = 10000.
color_cpp = mmcv.imread("cpp_out_rgb.png", flag="color", channel_order="rgb")
depth_cpp = mmcv.imread("cpp_out_depth.png", flag="unchanged") / depth_factor

color_py = mmcv.imread("python_out_rgb.png", flag="color", channel_order="rgb")
depth_py = mmcv.imread("python_out_depth.png", flag="unchanged") / depth_factor

color_vispy = mmcv.imread("vispy_out_rgb.png", flag="color", channel_order="rgb")
depth_vispy = mmcv.imread("vispy_out_depth.png", flag="unchanged") / depth_factor

diff_depth_cpp_py = depth_cpp - depth_py
diff_depth_cpp_vispy = depth_cpp - depth_vispy
diff_depth_py_vispy = depth_py - depth_vispy
print(f"diff_depth_cpp_py: min {diff_depth_cpp_py.min()} max {diff_depth_cpp_py.max()} mean {diff_depth_cpp_py.mean()}")
print(f"diff_depth_cpp_vispy: min {diff_depth_cpp_vispy.min()} max {diff_depth_cpp_vispy.max()} mean {diff_depth_cpp_vispy.mean()}")
print(f"diff_depth_py_vispy: min {diff_depth_py_vispy.min()} max {diff_depth_py_vispy.max()} mean {diff_depth_py_vispy.mean()}")

diff_color_cpp_py = color_cpp - color_py
diff_color_cpp_vispy = color_cpp - color_vispy
diff_color_py_vispy = color_py - color_vispy
print(f"diff_color_cpp_py: min {diff_color_cpp_py.min()} max {diff_color_cpp_py.max()} mean {diff_color_cpp_py.mean()}")
print(f"diff_color_cpp_vispy: min {diff_color_cpp_vispy.min()} max {diff_color_cpp_vispy.max()} mean {diff_color_cpp_vispy.mean()}")
print(f"diff_color_py_vispy: min {diff_color_py_vispy.min()} max {diff_color_py_vispy.max()} mean {diff_color_py_vispy.mean()}")

# diff_depth_cpp_py: min -0.000500000000000056 max 0.0005999999999999339 mean 6.780598958334426e-07
# diff_depth_cpp_vispy: min -0.000500000000000056 max 0.0005999999999999339 mean 6.780598958334422e-07
# diff_depth_py_vispy: min -0.00010000000000010001 max 9.999999999998899e-05 mean -3.6140072416183483e-22
# diff_color_cpp_py: min 0 max 255 mean 0.32834092881944443
# diff_color_cpp_vispy: min 0 max 255 mean 0.3169943576388889
# diff_color_py_vispy: min 0 max 1 mean 0.0003200954861111111

The difference is very small, especially for the glumpy("python") and vispy, since they only differ in some apis. BTW, it seems the vispy version is faster than the cpp and glumpy version.

(2) Run the bop evaluation with a sample result file hodan-iros15_lmo-test.csv:

Evaluation results using cpp renderer:

11/9|05:28:45: Evaluation of hodan-iros15_lmo-test.csv took 69.88159584999084s.
11/9|05:28:45: FINAL SCORES:
11/9|05:28:45: - bop19_average_recall_vsd: 0.3988581314878892
11/9|05:28:45: - bop19_average_recall_mssd: 0.491764705882353
11/9|05:28:45: - bop19_average_recall_mspd: 0.5121799307958477
11/9|05:28:45: - bop19_average_recall: 0.4676009227220299
11/9|05:28:45: - bop19_average_time_per_image: 69.1710237650415
11/9|05:28:45: Done.

Evaluation results using python (glumpy) renderer:

11/9|05:33:04: Average recall: 0.5121799307958477
11/9|05:33:04: Evaluation of hodan-iros15_lmo-test.csv took 64.30053234100342s.
11/9|05:33:04: FINAL SCORES:
11/9|05:33:04: - bop19_average_recall_vsd: 0.3993702422145329
11/9|05:33:04: - bop19_average_recall_mssd: 0.491764705882353
11/9|05:33:04: - bop19_average_recall_mspd: 0.5121799307958477
11/9|05:33:04: - bop19_average_recall: 0.4677716262975779
11/9|05:33:04: - bop19_average_time_per_image: 69.1710237650415
11/9|05:33:04: Done.

Evaluation results using vispy renderer:

11/9|05:31:15: Evaluation of hodan-iros15_lmo-test.csv took 62.2388596534729s.
11/9|05:31:15: FINAL SCORES:
11/9|05:31:15: - bop19_average_recall_vsd: 0.3993702422145329
11/9|05:31:15: - bop19_average_recall_mssd: 0.491764705882353
11/9|05:31:15: - bop19_average_recall_mspd: 0.5121799307958477
11/9|05:31:15: - bop19_average_recall: 0.4677716262975779
11/9|05:31:15: - bop19_average_time_per_image: 69.1710237650415
11/9|05:31:15: Done.

The vispy and python (glumpy) evaluation results are exactly the same, and the cpp version is slightly different.
Again, the vispy version looks faster.

So I think the vispy renderer is safe to be added. Maybe some more tests are also needed.
@thodan @MartinSmeyer Could you take a look into this?

[MartinSmeyer]

Hi @wangg12 ,

thanks for your addition of the vispy renderer. If it yields the same results faster than the glumpy renderer and offscreen, we should definitely add it. I guess pyrender would be another alternative, but vispy has a conveniently similar API to glumpy.

• Please also update the renderer documentation in the README.md. You can recommend the vispy renderer there and also explain how to use EGL offscreen rendering (Is this hack still required?)
• Instead of adding the sys path of bop_toolkit manually everywhere lets just create a setup.py file and make the bop_toolkit_lib installable.

We can still let users adapt config.py without reinstalling if we instruct them to install the package this way:

pip install -r requirements.txt -e .

I use the package this way myself and I think others would benefit as well since the utility functions are also useful outside the scripts folder. I created a PR #63 for that so you can remove the sys path additions here.

[wangg12]

Hi @MartinSmeyer, I think the hack for egl is not needed after pyopengl updated.

I will update the README.md and remove the sys.path stuff.

[wangg12]

Hi @MartinSmeyer @thodan, I have updated the doc, but I am not sure whether it is appropriate. Feel free to tell me how it can be improved.

[MartinSmeyer]

Thanks @wangg12, I have tested it and it works well.