NOTE: The inference code and pre-trained models for our work Deep Burst Super-Resolution is now available at https://github.com/goutamgmb/deep-burst-sr.
- Introduction
- Reference
- Dates
- Description
- Track 1 - Synthetic
- Track 2 - Real-world
- Toolkit
- Issues and Questions
- Organizers
- Terms and conditions
- Acknowledgements
Burst Image Super-Resolution Challenge will be held as part of the 6th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop to be held in conjunction with CVPR 2021. The task of this challenge is to generate a denoised, demosaicked, higher-resolution image, given a RAW burst as input. The challenge uses a new dataset and has 2 tracks, namely Track 1: Synthetic and Track 2: Real-world. The top ranked participants in each track will be awarded and all participants are invited to submit a paper describing their solution to the associated NTIRE workshop at CVPR 2021
The provided BurstSR dataset, code, and evaluation protocol for this challenge was introduced in our following paper. Please cite it if you use any of these in your work. Our paper also introduces a new method for Deep Burst Super-Resolution, which you can use as a baseline solution.
Deep Burst Super-Resolution
Goutam Bhat, Martin Danelljan, Luc Van Gool, Radu Timofte
arXiv:2101.10997
[paper]
- 2021.01.26 Release of train and validation data
- 2021.02.01 Validation server online
- 2021.03.15 Final test data release (inputs only)
- 2021.03.20 Test output results submission deadline
- 2021.03.20 Fact sheets and code/executable submission deadline
- 2021.03.22 Preliminary test results released to the participants
- 2021.04.02 Paper submission deadline for entries from the challenge
- 2021.06.15 NTIRE workshop and challenges, results and award ceremony (CVPR 2021, Online)
Given multiple noisy RAW images of a scene, the task in burst super-resolution is to predict a denoised higher-resolution RGB image by combining information from the multiple input frames. Concretely, the method will be provided a burst sequence containing 14 images, where each image contains the RAW sensor data from a bayer filter (RGGB) mosaic. The images in the burst have unknown offsets with respect to each other, and are corrupted by noise. The goal is to exploit the information from the multiple input images to predict a denoised, demosaicked RGB image having a 4 times higher resolution, compared to the input. The challenge will have two tracks, namely 1) Synthetic and 2) Real-world based on the source of the input data. The goal in both the tracks is to reconstruct the original image as well as possible, and not to artificially generate a plausible, visually pleasing image.
In the synthetic track, the input bursts are generated from RGB images using a synthetic data generation pipeline.
Data generation: The input sRGB image is first converted to linear sensor space using an inverse camera pipeline. A LR burst is then generated by applying random translations and rotations, followed by bilinear downsampling. The generated burst is then mosaicked and corrupted by random noise.
Training set: We provide code to generate the synthetic bursts using any image dataset for training. Note that any image dataset except the test split of the Zurich RAW to RGB dataset can be used to generate synthetic bursts for training.
Validation set: The bursts in the validation set have been pre-generated with the data generation code, using the RGB images from the test split of the Zurich RAW to RGB dataset.
If you wish to participate in the Synthetic track, please register for the challenge at the codalab page to get access to the evaluation server and receive email notifications for the challenge.
The methods will be ranked using the fidelity (in terms of PSNR) with the high-resolution ground truth, i.e. the linear sensor space image used to generate the burst. The focus of the challenge is on learning to reconstruct the original high-resolution image, and not the subsequent post-processing. Hence, the PSNR computation will be computed in the linear sensor space, before post-processing steps such as color correction, white-balancing, gamma correction etc.
The results on the validation set can be uploaded on the Codalab server (live now) to obtain the performance measures, as well as a live leaderboard ranking. The results should be uploaded as a ZIP file containing the network predictions for each burst. The predictions must be normalized to the range [0, 2^14] and saved as 16 bit (uint16) png files. Please refer to save_results_synburst_val.py for an example on how to save the results. An example submission file is available here.
The test set is now public. You can download the test set containing 500 synthetic bursts from this link. You can use the dataset class provided in synthetic_burst_test_set.py in the latest commit to load the burst sequences.
For the final submission, you need to submit:
- The predicted outputs for each burst sequence as a zip folder, in the same format as used for uploading results to the codalab validation server (see this for details).
- The code and model files necessary to reproduce your results.
- A factsheet (both PDF and tex files) describing your method. The template for the factsheet is available here.
The results, code, and factsheet should be submitted via the google form
NOTE: Training on the validation split is NOT allowed for test set submissions.
This track deals with the problem of real-world burst super-resolution. For this purpose, we introduce a new dataset BurstSR consisting of real-world bursts.
BurstSR dataset: BurstSR consists of 200 RAW burst sequences, and corresponding high-resolution ground truths (a reference for the paper introducing the dataset will be made available soon). Each burst sequence contains 14 RAW images captured using a handheld smartphone camera. For each burst sequence, we also capture a high-resolution image using a DSLR camera mounted on a tripod to serve as ground truth. We extract 160x160 crops from the bursts to obtain a training set consisting of 5405 crops, and a validation set consisting of 882 crops. A detailed description of the BurstSR dataset is available in the paper "Deep Burst Super-Resolution". Please cite the paper if you use the BurstSR dataset in your work.
Challenges: Since the burst and ground truth images are captured using different cameras, there exists a spatial mis-alignment, as well as color mis-match between the images. Thus, in addition to designing network architectures, developing effective training strategies to utilize mis-aligned training data is a key challenge in this track.
If you wish to participate in the Real-world track, please register for the challenge at the codalab page to receive email notifications for the challenge.
Due to the spatial and color mis-alignments between the input burst and the ground truth, it is difficult to estimate similarity between the network prediction and the ground truth. We introduce AlignedPSNR metric for this purpose. A user study will also be conducted as a complement to determine the final ranking on the test set.
AlignedPSNR: AlignedPSNR first spatially aligns the network prediction to the ground truth, using pixel-wise optical flow estimated using PWC-Net. A linear color mapping between the input burst and the ground truth, modeled as a 3x3 color correction matrix, is then estimated and used to transform the spatially aligned network prediction to the same color space as the ground truth. Finally, PSNR is computed between the spatially aligned and color corrected network prediction and the ground truth. More description of the AlignedPSNR metric is available in the paper "Deep Burst Super-Resolution".
User study: The emphasis of the user study will be on which method can best reconstruct the original high-frequency details. The goal is thus not to generate more pleasing images by modifying the output color space or generating artificial high frequency content not existing in the high-resolution ground truth.
The will be no evaluation server for Track 2. Instead, the ground truth images for the validation set are provided and the methods can be evaluated locally using the provided implementation of AlignedPSNR. Please refer to evaluate_burstsr_val.py script for an example on how to evaluate on BurstSR validation set.
NOTE: The evaluate_burstsr_val.py script computes the AlignedPSNR score in the linear sensor space, before white-balancing or any intensity scaling. Since the RAW sensor values in the captured bursts are generally small (mean of around 0.1 - 0.2), the computed PSNR values are generally very high (> 47), since the maximum signal value in the PSNR computation is still assumed to be 1.0. In the "Deep Burst Super-Resolution" paper, we computed the final scores on the white-balanced image, after scaling the image intensities to be between [0, 1]. Thus the scores computed by evaluate_burstsr_val.py cannot be compared with the scores reported in the paper.
The test set is now public. You can download the test set containing 639 real-world bursts from this link. You can use the dataset class provided in burstsr_test_dataset.py in the latest commit to load the burst sequences.
For the final submission, you need to submit:
- The predicted outputs for each burst sequence as a zip folder. The predictions must be normalized to the range [0, 2^14] and saved as 16 bit (uint16) png files. Please refer to save_results_burstsr_test.py for an example on how to save the results.
- The code and model files necessary to reproduce your results.
- A factsheet (both PDF and tex files) describing your method. The template for the factsheet is available here.
The results, code, and factsheet should be submitted via the google form
NOTE: Training on the validation split is NOT allowed for test set submissions.
We also provide a Python toolkit which includes the necessary data loading and evaluation scripts. The toolkit contains the following modules.
- data_processing: Contains the forward and inverse camera pipeline employed in “Unprocessing images for learned raw denoising”, as well as the script to generate a synthetic burst from a single RGB image.
- datasets: Contains the PyTorch dataset classes useful for the challenge.
- synthetic_burst_train_set provides the SyntheticBurst dataset which generates synthetic bursts using any image dataset.
- zurich_raw2rgb_dataset can be used to load the RGB images Zurich RAW to RGB mapping dataset. This can be used along with SyntheticBurst dataset to generate synthetic bursts for training.
- synthetic_burst_val_set can be used to load the pre-generated synthetic validation set.
- synthetic_burst_test_set can be used to load the pre-generated synthetic test set.
- burstsr_dataset provides the BurstSRDataset class which can be used to load the RAW bursts and high-resolution ground truths for the real-world track.
- burstsr_test_dataset provides the BurstSRTestDataset class which can be used to load the test split in the real-world track 2.
- pwcnet: The code for the optical flow network PWC-Net borrowed from pytorch-pwc. The network weights can be downloaded from here.
- scripts: Includes useful example scripts.
- download_burstsr_dataset can be used to download and unpack the BurstSR dataset.
- test_synthetic_burst provides an example on how to use the SyntheticBurst dataset.
- test_burstsr_dataset provides an example on how to use the BurstSR dataset.
- save_results_synburst_val provides an example on how to save the results on SyntheticBurstVal dataset for submission on the evaluation server.
- save_results_burstsr_test provides an example on how to save the results on the test set for the final submission for Track 2.
- evaluate_burstsr_val provides an example on how to evaluate a method on the BurstSR validation set.
- utils: Contains the AlignedPSNR metric, as well as other utility functions.
Installation: The toolkit requires PyTorch and OpenCV for track 1, and additionally exifread and cupy for track 2. The necessary packages can be installed with anaconda, using the install.sh script.
We provide the following data as part of the challenge.
Synthetic validation set: The official validation set for track 1. The dataset contains 300 synthetic bursts, each containing 14 RAW images. The synthetic bursts are generated from the RGB images from the test split of the Zurich RAW to RGB mapping dataset. The dataset can be downloaded from [here](https://data.vision.ee.ethz.ch/bhatg/syn_burst_val.zip](https://drive.google.com/file/d/1DHu3-_tGSc_8Wwwu6sHFaPtmd9ymd0rZ/view?usp=drive_link).
BurstSR train and validation set: The training and validation set for track 2. The dataset has been split into 10 parts and can be downloaded from here.
Zurich RAW to RGB mapping set: The RGB images from the training split of the Zurich RAW to RGB mapping dataset can be downloaded from here. These RGB images can be used to generate synthetic bursts for training using the SyntheticBurst class.
In case of any questions about the challenge or the toolkit, feel free to open an issue on Github.
- Goutam Bhat (goutam.bhat@vision.ee.ethz.ch)
- Martin Danelljan (martin.danelljan@vision.ee.ethz.ch)
- Radu Timofte (radu.timofte@vision.ee.ethz.ch)
The terms and conditions for participating in the challenge are provided here
The toolkit uses the forward and inverse camera pipeline code from unprocessing, as well as the PWC-Net code from pytorch-pwc.