Ask questions here about the usage

[Nicholasli1995]

Please don't ask questions in Issues, which are used only for bug report.

[KAB555]

Hello,
I am new to github, so I apologize if my formatting isn't correct or if I'm missing important information.

I've gone through the data preparation guide and am now trying to run the inference demo. I had to change the line at the end of demo.md from:
python inference.py --cfg "../configs/KITTI_inference:demo.yml" --visualize True --batch_to_show 2
to
python inference.py --cfg "../configs/KITTI_inference_demo.yml" --visualize True --batch_to_show 2

When it runs it:

1. gives this warning, "Warning: <img_name> not included in detected images!" every prediction
2. continually pops up unresponsive figures that don't close till it throws this runtime error, "RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained until explicitly closed and may consume too much memory."

I assume something went awry when I was setting up everything, and I haven't been able to root out what's going wrong. Any help would be greatly appreciated. Thanks!

[Nicholasli1995]

Hello,
I am new to github, so I apologize if my formatting isn't correct or if I'm missing important information.

I've gone through the data preparation guide and am now trying to run the inference demo. I had to change the line at the end of demo.md from:
python inference.py --cfg "../configs/KITTI_inference:demo.yml" --visualize True --batch_to_show 2
to
python inference.py --cfg "../configs/KITTI_inference_demo.yml" --visualize True --batch_to_show 2

When it runs it:

1. gives this warning, "Warning: <img_name> not included in detected images!" every prediction
2. continually pops up unresponsive figures that don't close till it throws this runtime error, "RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained until explicitly closed and may consume too much memory."

I assume something went awry when I was setting up everything, and I haven't been able to root out what's going wrong. Any help would be greatly appreciated. Thanks!

1. The warning is normal. It means some ground truth cars are not detected by the provided bounding boxes from D4LCN.
2. This should mean the loop did not break correctly. You may check whether batch_to_show is correctly set. You may try print cfgs['batch_to_show'] to see what happened.

[KAB555]

@Nicholasli1995 Thank you for the quick response.

   if cfgs['use_pred_box']:
        # use detected bounding box from any 2D/3D detector
        thres = cfgs['conf_thres'] if 'conf_thres' in cfgs else 0.
        annot_dict = gather_dict(meta, results['pred'], thres=thres)
        if len(annot_dict['path']) == 0:
            continue

After some testing, I found the issue was the 'continue' being triggered here in the inference function, therefore skipping the input prompt from 'visualize' being true and the 'break' from 'batch_to_show'.

As far as I know, I haven't deviated from the readme guides, so I'm not sure why it's acting like that.

[Nicholasli1995]

@Nicholasli1995 Thank you for the quick response.

   if cfgs['use_pred_box']:
        # use detected bounding box from any 2D/3D detector
        thres = cfgs['conf_thres'] if 'conf_thres' in cfgs else 0.
        annot_dict = gather_dict(meta, results['pred'], thres=thres)
        if len(annot_dict['path']) == 0:
            continue

After some testing, I found the issue was the 'continue' being triggered here in the inference function, therefore skipping the input prompt from 'visualize' being true and the 'break' from 'batch_to_show'.

As far as I know, I haven't deviated from the readme guides, so I'm not sure why it's acting like that.

I have made a minor change and now the 'break' will not be skipped. You can pull the code and try again.The previous issue should be caused by len(annot_dict['path']) == 0 which means no cars are detected in that image. That should not be too frequent and you may print(annot_dict) to see what happened.

[steveice]

Thank you for your work!
I try to use the inference code on the project with the script
" python inference.py --cfg "../configs/KITTI_inference:demo.yml" --visualize True --batch_to_show 2"
Is there a way to save the visualize result image to a png/jpg on the result folder?

[Nicholasli1995]

Thank you for your work!
I try to use the inference code on the project with the script
" python inference.py --cfg "../configs/KITTI_inference:demo.yml" --visualize True --batch_to_show 2"
Is there a way to save the visualize result image to a png/jpg on the result folder?

Yes. You can save the plots before closing using plt.savefig():
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html

[steveice]

Thank you.
I was wondering about how to save the images with bounding box for kitti dataset image.
Because I saw your description about visualized 3D bounding boxes in the document. So I was expecting the Kitti images.

[steveice]

The training stuck on this user warning message.
I follow the warning message to find the error on your train_IGRs.py, but I didn't find the usage of lr_scheduler.step().
Any suggestions?

[Nicholasli1995]

The training stuck on this user warning message.
I follow the warning message to find the error on your train_IGRs.py, but I didn't find the usage of lr_scheduler.step().
Any suggestions?

This warning should not be relevant to the cause of stuck. You can Ctrl+C and see if the process is stuck at the dataloader part. For some environments, it seems the pytorch dataloader has deadlock when using with opencv. The solutions that may work are: use num_workers=0; disable opencv multi-threading...

If the problem persists, you can provide your detailed environment information and error message.

[DakotaFrico33]

Hi, @Nicholasli1995, thank you for your work.
I am trying to fully understand the paper you wrote and its relation with this GitHub repository.

One thing I do not fully understand is the source of the 3D bbox ground truth points ('kpts_3d_gt'). My understanding is that there are 8 of these 3D bbox points which identify the 8 vertices of the 3D bbox (surrounding a vehicle); moreover, after regressing them, you then proceed to use them for your calculations related to the IGRs (interpolated cuboids).

To my knowledge, the KITTI dataset provides only 2D data for bbox that are present in the frame. So, would you be able to help me understand how you obtain these 3D points that you use as ground truth for your 3D bbox regression?

[claytonkanderson]

Hey @DakotaFrico33,

I was wondering about this too. The KITTI dataset has 2d bounding box data, but also 3d dimensions, 3d location, and rotation about the y-axis.

From the 3d dimensions of the object, you can get the 8 3d corners of a bounding box without translation or rotation. Then the y-rotation can be applied and the 3d translation to give you the 8 3d corners of an oriented bounding box. See

EgoNet/libs/dataset/KITTI/car_instance.py

Line 744 in 522c1e4

def get_cam_cord(self, cam_cord, shift, ids, pose_vecs, rot_xz=False):

It's my understanding that EgoNet refines this orientation using the cropped image, and the 'first-order' predictions provided by the above method.

That's for the training data, where the ground truth information is available. For the testing set, I believe EgoNet applies the D4LCN algorithm to the raw KITTI images to predict the 2d bounding box, local-yaw, 3d size, and 3d translation (although I don't think this one is used). Then EgoNet refines the D4LCN predictions, mostly the orientation.

[Nicholasli1995]

Hi, @Nicholasli1995, thank you for your work. I am trying to fully understand the paper you wrote and its relation with this GitHub repository.

One thing I do not fully understand is the source of the 3D bbox ground truth points ('kpts_3d_gt'). My understanding is that there are 8 of these 3D bbox points which identify the 8 vertices of the 3D bbox (surrounding a vehicle); moreover, after regressing them, you then proceed to use them for your calculations related to the IGRs (interpolated cuboids).

To my knowledge, the KITTI dataset provides only 2D data for bbox that are present in the frame. So, would you be able to help me understand how you obtain these 3D points that you use as ground truth for your 3D bbox regression?

Hi, as claytonkanderson mentioned, the 3d kpts are computed from the dimension and orientation annotations in the get_cam_cord method.

[w17995675]

Hello, @Nicholasli1995 , first of all, thank you for your research contributions. I have the following questions that I would like to ask you.

the results using two different inference files are as follows:
1.python inference.py --cfg "../configs/KITTI_inference:demo.yml" --visualize True --batch_to_show 1
fig.1:

2.python inference.py --cfg "../configs/KITTI_inference:test_submission.yml" --visualize True --batch_to_show 1
fig.2:

I would like to know how to perform the following tasks during inference on the test set:
1.How to convert the keypoints on the image into the cuboid shown in Figure 1.
2.How to visualize the direction of the vehicles using arrows.

[Nicholasli1995]

Hi,
Since no ground truth 3D box is not available on the test set, the arrow and green cuboid are not drawn in fig.2

If you want to add such elements for visualization purposes, you can:

1. draw lines to connect the predicted key points to form a cuboid
2. draw (a) a cuboid center by computing the intersection of lines, (b) a front-face center by computing another intersection and (c) connect the cuboid center and front-face center