keypoints annotation in BlenderProc

[soans1994]

Hello Authors,

Thank you for your project. Is there any example to save pose, keypoints annotation in COCO format or any other formats along with the bounding box information.

[cornerfarmer]

Hey @soans1994,

until now we do not support keypoint annotations at all. So you would need to implement this feature from scratch.
As the current CocoAnnotationsWriter is specified on segmentation masks, it would probably make sense to create a separate CocoKeypointAnnotationsWriter.
Getting the keypoint coordinates, should be as simple as projecting the corresponding 3D points to 2D.
Determining the 3D keypoints is of course heavily dependent on your data.

Let me know if you need any further guidance.

[soans1994]

@cornerfarmer Thank you for your reply. Is it easy to create the script similar to CocoAnnotaionsWriter for keypoints? I will try to generate the script. Please let me know if you build this and add a simple example.

[cornerfarmer]

Writing the keypoints to a coco annotations file should not be that hard, as in the end its just a json file.
However, determining the keypoints is probably the harder part, but this depends heavily on the data you want to use.

[soans1994]

Can you please update an example to save keypoints location from 3d model. I have used the annotating tool in blender to put the points on the edge of a simple cube. How can i convert these to 2d location and export to a text file.

[cornerfarmer]

Hey @soans1994

here is a simple module that projects all annotation points to 2D and writes them to a npy file:

import bpy
import bpy_extras
import os

from src.main.Module import Module
import numpy as np
from src.utility.Utility import Utility

class AnnotationWriter(Module):
    def __init__(self, config):
        Module.__init__(self, config)

    def run(self):
        width, height = bpy.context.scene.render.resolution_x, bpy.context.scene.render.resolution_y

        for frame_id in range(bpy.context.scene.frame_start, bpy.context.scene.frame_end):
            bpy.context.scene.frame_set(frame_id)

            annotatons = []
            for pencil in bpy.data.grease_pencils.values():
                for layer in pencil.layers.values():
                    for frame in layer.frames.values():
                        for stroke in frame.strokes.values():
                            for point in stroke.points.values():

                                point2d_blender = bpy_extras.object_utils.world_to_camera_view(bpy.context.scene, bpy.context.scene.camera, point.co)
                                annotatons.append([point2d_blender[0] * width, (1 - point2d_blender[1]) * height])

            np.save(os.path.join(self._determine_output_dir(), "annotations_" + ("%04d" % frame_id) + ".npy"), np.array(annotatons))

        Utility.register_output(self._determine_output_dir(), "annotations_", "annotations", ".npy", "1.0.0")

Let me know if this solves your problem.

[soans1994]

@cornerfarmer
Thank you very much for the example code. I have some doubts regarding the annotation tools in Blender. What is the difference between the annotation tool and grease pencil? Should i perform manual annotation on the 3d objects or can i extract the points from the 3d mesh. Sorry since i am new to blender.
Edit:Can you add the screenshot of your blender project which shows the collection of objects related to keypoints in the right side panel.
Below is the screenshot of the example blend file from blenderproc. I just added a default grease pencil object (Suzanne 001), and i added your new AnnotationWriter module to the config file and ran the python code, but it seems the module is not running.

[cornerfarmer]

Hey @soans1994

the annotation writer was properly executed in your example, however it writes it output to the temporary directory per default and as you do not use the HDF5 Writer it is not written automatically to the output directory.
To write the annotations directly to the output directory, you need to set output_is_temp to false:

{
      "module": "writer.AnnotationWriter",
      "config": {
           "output_is_temp": False
      }
},

The module I wrote should output annotations as well as grease pencil objects. They are basically the same thing, only that grease pencil objects are actual objects in the scene graph.

I personally would not use the annotations or grease pencil objects for specifying keypoints, as they are more built for adding notes to 3d models and not for precisely setting 3d points. Alternatively you could for example add empty objects in blender and then use their locations as keypoints. Or you can also iterate over the vertices of an object an use their locations as keypoints:

for vert in bpy.context.scene.objects["Cube"].data.vertices:
    point2d_blender = bpy_extras.object_utils.world_to_camera_view(bpy.context.scene, bpy.context.scene.camera, vert.co)
    annotatons.append([point2d_blender[0] * width, (1 - point2d_blender[1]) * height])

[soans1994]

@cornerfarmer

Thank you very much, now i can save the keypoints into numpy format from the 3d model. I will try to combine this function with the COCO writer module to get json format.

[cornerfarmer]

Glad that it works 👍
I am closing this issue for now. Feel free to reopen it, if you run into further related problems or open a new issue, if you run into separate problems.

[luigifaticoso]

Hey, @cornerfarmer,
thank you for the helpful reply, when I ran into an issue while running this writer module.
The file module.py doesn't have any _register_output. Is it from an old version?

    pipeline.run()
  File "./src/main/Pipeline.py", line 86, in run
    module.run()
  File "./src/writer/AnnotationWriter.py", line 30, in run
    self._register_output("annotations_", "annotations", ".npy", "1.0.0")
AttributeError: 'AnnotationWriter' object has no attribute '_register_output'

[themasterlink]

Hey,

on which BlenderProc version are you? There is no class AnnotationWriter?

Best,
Max

[luigifaticoso]

I'm on the master branch,
I can't find any AnnotationWriter class, I added my own based on @cornerfarmer solution.

[cornerfarmer]

Hey @luigifaticoso,

since when I posted the code, the way to register a new output moved into the Utility class.
Therefore the last line of the AnnotationWriter module has to be changed to

Utility.register_output(self._determine_output_dir(), "annotations_", "annotations", ".npy", "1.0.0")

The utility class also needs to be imported via

from src.utility.Utility import Utility

Let me know if that solves the issue

[themasterlink]

@cornerfarmer If so, could you also update the example above? :)

[luigifaticoso]

@cornerfarmer yes it did, thank you very much!

[themasterlink]

27 minutes from problem to fix, not bad ;)

[prrw]

Hey @cornerfarmer @themasterlink ,
what is the status of this issue ? I see the example script is using Blender API. Could you include keypoint annotations in BlenderProc ? While I agree that getting keypoints is just a simple projection task, what really would be helpful is to know whether a keypoint is visible or obstructed. This can be integrated into COCO Annotations because segmentation masks here are generated based on their visibility?