TBRNet

1. Resources

国内资源链接(密码：e2ww)

1.1 Dataset

• SRD
• ISTD
• ISTD+DA, SRD+DA

1.2 Results

• Results on SRD: TBRNet_SRD
• Results on SRD+DA: TBRNet_SRD_DA
• Results on ISTD: TBRNet_ISTD
• Results on ISTD+DA: TBRNet_ISTD_DA
• Results on ISTD+: TBRNet_ISTD+

Visual comparison results of penumbra removal results on the SRD dataset - (Powered by MulimgViewer)

1.3 Model Weight File for Test

• Model on SRD: TBRNet_SRD.pth
• Model on SRD+DA: TBRNet_SRD_DA.pth
• Model on ISTD: TBRNet_ISTD.pth
• Model on ISTD+DA: TBRNet_ISTD_DA.pth
• Model on ISTD+: TBRNet_ISTD+.pth

1.4 Evaluation Code

1. MAE (i.e., RMSE in paper). Hieu Le reuploaded the evaluation code. Before that, everyone can obtain this code from Jifeng Wang via email. Currently, the MATLAB code is used in most state-of-the-art works for shadow removal.

2. PSNR+SSIM. zhu_aaai2022

2. Environments

ubuntu18.04+cuda10.2+pytorch1.7.1

1. create environments

conda env create -f install.yaml

2. activate environments

conda activate TBRNet

3. Data Processing

For example, generate the dataset list of ISTD:

1. Download:
   • ISTD and SRD
   • USR shadowfree images
   • Syn. Shadow
   • SRD shadow mask (or try our re-uploaded link)
   • train_B_ISTD:

   cp -r ISTD_Dataset_arg/train_B ISTD_Dataset_arg/train_B_ISTD
   cp -r ISTD_Dataset_arg/train_B SRD_Dataset_arg/train_B_ISTD
   

   • VGG19

   cp vgg19-dcbb9e9d.pth ISTD_Dataset_arg/
   cp vgg19-dcbb9e9d.pth SRD_Dataset_arg/
   

2. The data folders should be:

   ISTD_Dataset_arg
       * train
           - train_A # ISTD shadow image
           - train_B # ISTD shadow mask
           - train_C # ISTD shadowfree image
           - shadow_free # USR shadowfree images
           - synC # Syn. shadow
           - train_B_ISTD # ISTD shadow mask
       * test
           - test_A # ISTD shadow image
           - test_B # ISTD shadow mask
           - test_C # ISTD shadowfree image
       * vgg19-dcbb9e9d.pth
   
   SRD_Dataset_arg
       * train #  renaming the original `Train` folder in `SRD`.
           - train_A # SRD shadow image, renaming the original `shadow` folder in `SRD`.
           - train_B # SRD shadow mask
           - train_C # SRD shadowfree image, renaming the original `shadow_free` folder in `SRD`.
           - shadow_free # USR shadowfree images
           - synC # Syn. shadow
           - train_B_ISTD # ISTD shadow mask
       * test #  renaming the original `test_data` folder in `SRD`.
           - train_A # SRD shadow image, renaming the original `shadow` folder in `SRD`.
           - train_B # SRD shadow mask
           - train_C # SRD shadowfree image, renaming the original `shadow_free` folder in `SRD`.
       * vgg19-dcbb9e9d.pth 
   

3. Edit generate_flist_istd.py: (Replace path)

ISTD_path = "/Your_data_storage_path/ISTD_Dataset_arg"

4. Generate Datasets List. (Already contains ISTD+DA.)

conda activate TBRNet
cd script/
python generate_flist_istd.py

5. Edit config_ISTD.yml: (Replace path)

DATA_ROOT: /Your_data_storage_path/ISTD_Dataset_arg

4. Training+Test+Evaluation

4.1 Training+Test+Evaluation

For example, training+test+evaluation on ISTD dataset.

cp config/config_ISTD.yml config.yml 
cp config/run_ISTD.py run.py
conda activate TBRNet
python run.py

4.2 Only Test and Evaluation

For example, test+evaluation on ISTD dataset.

1. Download weight file(TBRNet_ISTD.pth) to pre_train_model/ISTD
2. Copy file

cp config/config_ISTD.yml config.yml 
cp config/run_ISTD.py run.py
mkdir -p checkpoints/ISTD/
cp config.yml checkpoints/ISTD/config.yml
cp pre_train_model/ISTD/TBRNet_ISTD.pth  checkpoints/ISTD/ShadowRemoval.pth

3. Edit run.py. Comment the training code.

    # # pre_train (no data augmentation)
    # MODE = 0
    # print('\nmode-'+str(MODE)+': start pre_training(data augmentation)...\n')
    # for i in range(1):
    #     skip_train = init_config(checkpoints_path, MODE=MODE,
    #                             EVAL_INTERVAL_EPOCH=1, EPOCH=[90,i])
    #     if not skip_train:
    #         main(MODE, config_path)
    # src_path = Path('./pre_train_model') / \
    #     config["SUBJECT_WORD"]/(config["MODEL_NAME"]+'_pre_da.pth')
    # copypth(dest_path, src_path)

    # # train
    # MODE = 2
    # print('\nmode-'+str(MODE)+': start training...\n')
    # for i in range(1):
    #     skip_train = init_config(checkpoints_path, MODE=MODE,
    #                             EVAL_INTERVAL_EPOCH=0.1, EPOCH=[60,i])
    #     if not skip_train:
    #         main(MODE, config_path)
    # src_path = Path('./pre_train_model') / \
    #     config["SUBJECT_WORD"]/(config["MODEL_NAME"]+'_final.pth')
    # copypth(dest_path, src_path)

4. Run

conda activate TBRNet
python run.py

4.3 Show Results

After evaluation, execute the following code to display the final RMSE.

python show_eval_result.py

Output:

running rmse-shadow: xxx, rmse-non-shadow: xxx, rmse-all: xxx # ISRD

This is the evaluation result of python+pytorch, which is only used during training. To get the evaluation results in the paper, you need to run the matlab code.

5. Acknowledgements

Part of the code is based upon:

• https://github.com/nachifur/LLPC
• https://github.com/vinthony/ghost-free-shadow-removal
• https://github.com/knazeri/edge-connect

6. Citation

If you find our work useful in your research, please consider citing:

@ARTICLE{liu2023shadow,
  author={Liu, Jiawei and Wang, Qiang and Fan, Huijie and Tian, Jiandong and Tang, Yandong},
  journal={IEEE Transactions on Neural Networks and Learning Systems}, 
  title={A Shadow Imaging Bilinear Model and Three-Branch Residual Network for Shadow Removal}, 
  year={2023},
  volume={},
  number={},
  pages={1-15},
  doi={10.1109/TNNLS.2023.3290078}
}

7. Contact

Please contact Jiawei Liu if there is any question (liujiawei18@mails.ucas.ac.cn).