Deep STORM
Deep-STORM is a deep learning pipeline developed by Nehme et al., see original publication here for single molecule localization microscopy reconstruction which we have adapted for use in ZeroCostDL4Mic. Deep-STORM is an image-to-image reconstruction (does not directly extract localization data), therefore it is capable to directly create a super-resolution image from the raw data.
The demonstrated advantages are:
- it works at higher emitter density than common (Maximum Likelihood Estimation) MLE approaches
- it produces the super-resolved image faster than common MLE approaches
Deep-STORM was originally described in the following papers:
Deep-STORM original code and documentation is freely available on Elias' GitHub page
Please also cite these original papers when using Deep-STORM with our notebook.
Training Deep-STORM is a bit easier compared to other neural networks since it can be trained exclusively using simulated data. So, in our ZeroCostDL4Mic notebook, we implemented a simulator into the notebook in order to enable this directly. However, initially in the published paper, the authors used the simulator from ThunderSTORM so we also enabled that possibility in the notebook (in this case both the localization file and the tiff stack need to be provided).
The simulator will need to emulate as best as possible the type of data that the user intends to reconstruct with this model, so a few parameters need to be provided. A number of parameters are self-explanatory (e.g. pixel size, conversion) but some others need to be estimated by the user from their own experimental dataset. The following will guide the user in estimating these.
-
FOV_sive: It’s typically sufficient to simulate FOV that correspond to 64x64 images, since many patches can be obtained from this image size already.
-
number_of_frames: It’s typically sufficient to simulate 20 frames.
-
The ADC_offset and ReadoutNoise_ADC parameters can be estimated by selecting a representative ROI of the data, thresholding out the areas representing the localizations/signal and obtain the mean and standard deviation of the background. This can be easily done in Fiji, by using the adjusting the Threshold then use the Create selection option and perform a Measurement. The Mean and StdDev measurements will correspond to ADC_offset and ReadoutNoise_ADC respectively.
- emitter_density: this is an important parameter and should be estimated from a region of representative density of the dataset to reconstruct, either by doing a visual estimation or by running a multi-emitter MLE analysis on that small region (and a few representative frames). Check that emitters were identified correctly and estimate the density from the number of localization and the size of that FOV. Important note: the density measured this way may well be an underestimate of the local density to the structure of interest and should therefore be taken as a lower limit of the density to use in simulation.
-
n_photons: similarly to above, the average number of photons in each localization can be estimated from the ME-MLE analysis (provided that the conversion factor was provided properly). One thing to keep in mind is that, when using EMCCD, there is a significant effect of the excess noise of the camera. This will effectively double the Poisson noise contribution, affecting the SNR. One way to take this into account is to do the following: divide the number of photons measured by a factor 2 and multiply the ADC/photon conversion factor by 2 in the simulation. The standard deviation of the number of photons can be similarly estimated from the distribution of photon counts from ThunderSTORM. This allows the network to train on a range of photon counts and make the reconstruction more robust.
-
sigma: The PSF sigma can also be estimated from the MLE estimation. Alternatively, it can also be estimated by the wavelength and the NA of the objective. These can be set using the known values of the experimental conditions. But it’s useful to check with the experimental data. For a Gaussian model:
𝜎 ≈ 0.21 𝜆/𝑁𝐴
where 𝜆 is the wavelength in nm and 𝑁𝐴 is the numerical aperture of the objective used, as described in this paper.
But estimating the PSF sigma with the above equation can be over-optimistic about how narrow the PSF is (ignoring any amount of aberrations) so it’s useful to check the estimate of the PSF size provided (called Sigma and quoted in nm) to what real PSF look like in the user’s data (this can also be estimated in ThunderSTORM). Here, Sigma is the standard deviation of the Gaussian model used to simulate the PSF so it should be compared to the standard deviation of Gaussian fitting to real dataset. In the case of a significant disagreement, the NA or the wavelength of the simulation can be adjusted to match the experimental value. Similarly to other parameters, a variability on sigma can be added by setting sigma_std.
The simulated training dataset can be visualized/saved and downloaded once simulation is complete for assessment that the data appear reasonable to the user.
The next step is to generate training patches from the training dataset. This is performed in section 3.2. For Deep-STORM, 10,000 patches / 100 epochs tend to do a good job. For the batch size, keep a number divisible by 2 (16 or 32 for instance). The higher, the faster it’ll train, but the more GPU memory you’ll need. The number of steps should allow for the whole set of patches to be seen for each epoch therefore: a good rule of thumb for steps_per_epochs is to use (Total number of patches used for training)/(batch size) (This calculation will be automated in future versions of the notebook).
To train Deep-STORM in Google Colab:
| Network | Link to example training and test dataset | Direct link to notebook in Colab |
|---|---|---|
| Deep-STORM | here |  |
or:
-
Download our streamlined ZeroCostDL4Mic notebooks
-
Open Google Colab
-
Once the notebook is open, follow the instructions.