Pre-trained caffe model for FCN score evaluation

[FishYuLi]

Hello, could you please tell me the performance of the pre-trained caffe model on the original cityscapes datasets given by this code ?
I follow the instructions in readme.mk to evaluate the label_to_image generator, which means the input of the FCN model is the generated fake image. I wondered how the model downloaded by this code works on the original cityscapes datasets. So I change the parameter --result_dir in ./scripts/eval_cityscapes/evaluate.py to the original val set of cityscapes. But the results are very bad(as follows). Is there anything wrong with this model ? Or is this model already been trained on cityscapes dataset?

Thank you.

[wxh001qq]

sorry, may i ask you how to run the evaluate.py? i found there request labels,and there are many fold that could not found in the cityscape dataset that the download.sh provide

[FishYuLi]

@wxh001qq

1. You need to download the original labels of cityscape dataset (not the given dataset in this code).
2. Change the paths in here to make sure that you can find everything you need.

[tinghuiz]

The pre-trained model does not work well on Cityscapes in the original resolution (1024x2048) as it was trained on 256x256 images that are resized to 1024x2048. The purpose of the resizing was to 1) keep the label maps in the original resolution and 2) avoid the need of changing the standard FCN training code for Cityscapes. To get the 'ground-truth' numbers in the paper, you need to resize the original Cityscapes images to 256x256 before running the evaluation code.

[tychovdo]

This issue seems similar to a problem that I am having. I tried resizing the images to 256x256, but I still get really bad results from the pretrained caffe model.
@FishYuLi : did you find a solution to this problem?

[FishYuLi]

@tychovdo I can reproduce the result in the paper after resizing the images to 256x256. I have read your problem. It seems so strange. I haven't met such problems. Is it possible that anything is wrong?

[tychovdo]

Your ‘bad looking’ segmentations do look very similar to the segmentations that I am getting (mostly “road”). How did you resize the images?
I use

from skimage.misc import imresize
resized = imresize(image, (256, 256, 3))

[FishYuLi]

@tychovdo I just remembered that if you want to run fcn score on the original image, you need to first resize the image to 256x256, and resize it to 1024x2048, and then send it to the segmentation network. Just uncomment this line.
It has been a long time when I met this problem, so I am not very sure. But you may try.

[lkiani]

@wxh001qq

1. You need to download the original labels of cityscape dataset (not the given dataset in this code).
2. Change the paths in here to make sure that you can find everything you need.

I downloaded the labels.py file and placed in content / pix2pix / scripts / helpers / but I still receive the No module named labels message.
thank you

[ZhangCZhen]

@FishYuLi Hello, my evaluation results are the same as yours. In particular,

1. If the images are 1024*2048, the mean pixel accuracy = 0.479361, mean class accuracy = 0.087739, mean class IoU = 0.056188.
2. If the images are 256*256, the mean pixel accuracy = 0.793441, mean class accuracy = 0.237466, mean class IoU = 0.1963344.
   I evaluated using RGB images from real datasets, and I think the accuracy value should be close to 1. What is the result of your reproduction? What is your reproduction process? Thanks a lot!

[FishYuLi]

@ZhangCZhen Please refer to the previous discussion.
I got the same FCN-scores as stated in Pix2pix[1] Table.1. (0.80, 0.26, 0.21 for per-pixel acc. per-class acc. and class IOU). It won't be 1, becuase it is decided by the performance of the FCN model you used to evaluate FCN-score.

[1] Phillip Isola et al. Image-to-Image Translation with Conditional Adversarial Networks.

[ZhangCZhen]

@FishYuLi Thank you sooooooo much for your help!

[syz825211943]

Hello, the link of pre-trained caffe model is unavailable, can you share the model to me? thx! @FishYuLi @ZhangCZhen

[ALISAXXW]

您的邮件已收到，我会尽快回复~~~

[HuXiaokai12138]

Hello, could you please tell me the performance of the pre-trained caffe model on the original cityscapes datasets given by this code ? I follow the instructions in readme.mk to evaluate the label_to_image generator, which means the input of the FCN model is the generated fake image. I wondered how the model downloaded by this code works on the original cityscapes datasets. So I change the parameter --result_dir in ./scripts/eval_cityscapes/evaluate.py to the original val set of cityscapes. But the results are very bad(as follows). Is there anything wrong with this model ? Or is this model already been trained on cityscapes dataset? Thank you.

@FishYuLi I have the same problem, even the numbers are the same as yours This question has been bothering me for a long time, I would love to know how you solved this problem, and I look forward to your reply.

[HuXiaokai12138]

@FishYuLi Hello, my evaluation results are the same as yours. In particular,

1. If the images are 1024*2048, the mean pixel accuracy = 0.479361, mean class accuracy = 0.087739, mean class IoU = 0.056188.
2. If the images are 256*256, the mean pixel accuracy = 0.793441, mean class accuracy = 0.237466, mean class IoU = 0.1963344.
   I evaluated using RGB images from real datasets, and I think the accuracy value should be close to 1. What is the result of your reproduction? What is your reproduction process? Thanks a lot!

@ZhangCZhen I would like to ask, did you change the image to 256×256 only the real picture, or did you change the label map as well?Because I got the same result as you before I changed it.