gaussians._semantic_feature is 128.

[Lee-JaeWon]

Thank you for your great work. I have some questions.

When using LSeg with the speedup option, the dimension of gaussians._semantic_feature is 128. I want to keep the speedup while ensuring that gaussians._semantic_feature has a dimension of 512 (to match the dimension of the text feature 150*512). Is there any way to achieve this?

I understand that reducing it to 128 is to shorten the rendering time, but the rendered image becomes 512-dimensional during segmentation, which allows it to combine with the text feature.

It would be helpful to get some hints on how to achieve this.

[ShijieZhou-UCLA]

Once you enable speedup, the CNN decoder will achieve this for you. Check here:

feature-3dgs/train.py

Line 52 in 6c570d6

cnn_decoder = CNN_decoder(feature_in_dim, feature_out_dim)

[Lee-JaeWon]

@ShijieZhou-UCLA
Thank you for your reply. Isn't cnn decoder intended for image features? Can it also be used for gaussians._semantic_feature for each Gaussian?

[huanhuanyuan7]

@ShijieZhou-UCLA Thank you for your reply. Isn't cnn decoder intended for image features? Can it also be used for gaussians._semantic_feature for each Gaussian?

In the train.py, from lines106-107, only the feature_map (rendered feature) is used to upsampling for speeding up.
if dataset.speedup: feature_map = cnn_decoder(feature_map)
In addition, I encountered a problem similar to yours.
I adopt SAM to obtain the sematic_feature (dimension (256,48,64)). When using the following code, the CNN_decoder's input dim is 64.

gt_feature_map = viewpoint_cam.semantic_feature.cuda() feature_out_dim = gt_feature_map.shape[0] # speed up if dataset.speedup: feature_in_dim = int(feature_out_dim/4) cnn_decoder = CNN_decoder(feature_in_dim, feature_out_dim)
However, from lines 101-107, this leads to a dimension mismatch error because the feature_map channels dim is 128.
gt_feature_map = viewpoint_cam.semantic_feature.cuda() # (256,48,64) feature_map = F.interpolate(feature_map.unsqueeze(0), size=(gt_feature_map.shape[1], gt_feature_map.shape[2]), mode='bilinear', align_corners=True).squeeze(0) print("feature_map.shape:", feature_map.shape) feature_map = feature_map.unsqueeze(0) # (1,128,48,64) print("unsqueezed feature_map.shape:", feature_map.shape) if dataset.speedup: feature_map = cnn_decoder(feature_map)

[liangyuanZhang1]

@ShijieZhou-UCLA Thank you for your reply. Isn't cnn decoder intended for image features? Can it also be used for gaussians._semantic_feature for each Gaussian?

In the train.py, from lines106-107, only the feature_map (rendered feature) is used to upsampling for speeding up. if dataset.speedup: feature_map = cnn_decoder(feature_map) In addition, I encountered a problem similar to yours. I adopt SAM to obtain the sematic_feature (dimension (256,48,64)). When using the following code, the CNN_decoder's input dim is 64.

gt_feature_map = viewpoint_cam.semantic_feature.cuda() feature_out_dim = gt_feature_map.shape[0] # speed up if dataset.speedup: feature_in_dim = int(feature_out_dim/4) cnn_decoder = CNN_decoder(feature_in_dim, feature_out_dim) However, from lines 101-107, this leads to a dimension mismatch error because the feature_map channels dim is 128. gt_feature_map = viewpoint_cam.semantic_feature.cuda() # (256,48,64) feature_map = F.interpolate(feature_map.unsqueeze(0), size=(gt_feature_map.shape[1], gt_feature_map.shape[2]), mode='bilinear', align_corners=True).squeeze(0) print("feature_map.shape:", feature_map.shape) feature_map = feature_map.unsqueeze(0) # (1,128,48,64) print("unsqueezed feature_map.shape:", feature_map.shape) if dataset.speedup: feature_map = cnn_decoder(feature_map)

I encountered the same problem, I solved it by modifying the dimension of cnndecoder but then I encountered new problems
RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 4 - got [138766, 1, 128] but expected shape compatible with [138766, 1, 64]