Deep STORM
Deep-STORM is a deep learning pipeline developed by Nehme et al., see original publication here for single molecule localization microscopy reconstruction which we have adapted for use in ZeroCostDL4Mic. Deep-STORM is an image-to-image reconstruction (does not directly extract localization data), therefore it is capable to directly create a super-resolution image from the raw data.
The demonstrated advantages are:
- it works at higher emitter density than common (Maximum Likelihood Estimation) MLE approaches
- it produces the super-resolved image faster than common MLE approaches
- training is performed on simulated data, which is much faster than acquiring a large amount of training data
Deep-STORM was originally described in the following papers:
Deep-STORM original code and documentation is freely available on Elias' GitHub page
Please also cite these original papers when using Deep-STORM with our notebook.
Training Deep-STORM is a bit easier compared to other neural networks since it can be trained exclusively using simulated data. So, in our ZeroCostDL4Mic notebook, we implemented a simulator into the notebook in order to enable this directly. However, initially in the published paper, the authors used the simulator from ThunderSTORM so we also enabled that possibility in the notebook (in this case both the localization file and the tiff stack need to be provided).
The simulator will need to emulate as best as possible the type of data that the user intends to reconstruct with this model, so a few parameters need to be provided. A number of parameters are self-explanatory (e.g. pixel size, conversion) but some others need to be estimated by the user from their own experimental dataset. The following will guide the user in estimating these.
-
FOV_size: It’s typically sufficient to simulate FOV that correspond to 64x64 images, since many patches can be obtained from this image size already.
-
number_of_frames: It’s typically sufficient to simulate 20 frames.
-
The ADC_offset and ReadoutNoise_ADC parameters can be estimated by selecting a representative ROI of the data, thresholding out the areas representing the localizations/signal and obtain the mean and standard deviation of the background. This can be easily done in Fiji, by using the adjusting the Threshold then use the Create selection option and perform a Measurement. The Mean and StdDev measurements will correspond to ADC_offset and ReadoutNoise_ADC respectively. If these parameters are missing in the results table, they can be added using the Set Measurement option in Fiji.
- emitter_density: this is an important parameter and should be estimated from a region of representative density of the dataset to reconstruct, either by doing a visual estimation or by running a multi-emitter MLE analysis (e.g. using ThunderSTORM) on that small region (and a few representative frames). Check that emitters were identified correctly and estimate the density from the number of localization and the size of that FOV. Important note: the density measured this way may well be an underestimate of the local density of the structure of interest and should therefore be taken as a lower limit of the density to use in simulation.
-
n_photons: similarly to above, the average number of photons in each localization can be estimated from the ME-MLE analysis (provided that the conversion factor was provided properly). One thing to keep in mind is that, when using EMCCD, there is a significant effect of the excess noise of the camera. This will effectively double the Poisson noise contribution, affecting the SNR. In the simulator provided by ThunderSTORM, this can be directly set within the camera settings. For the simulator that we provide, this can be taken into account by doing the following: divide the number of photons measured by a factor 2 and multiply the ADC/photon conversion factor by 2 in the simulation. The standard deviation of the number of photons can be similarly estimated from the distribution of photon counts from ThunderSTORM. This allows the network to train on a range of photon counts and make the reconstruction more robust.
-
sigma: The PSF sigma can also be estimated from the MLE estimation. Alternatively, it can also be estimated by the wavelength and the NA of the objective. These can be set using the known values of the experimental conditions. But it’s useful to check with the experimental data. For a Gaussian model:
𝜎 ≈ 0.21 𝜆/𝑁𝐴
where 𝜆 is the wavelength in nm and 𝑁𝐴 is the numerical aperture of the objective used, as described in this paper.
But estimating the PSF sigma with the above equation can be over-optimistic about how narrow the PSF is (ignoring any amount of aberrations) so it’s useful to check the estimate of the PSF size provided (called Sigma and quoted in nm) to what real PSF look like in the user’s data (this can also be estimated in ThunderSTORM). Here, Sigma is the standard deviation of the Gaussian model used to simulate the PSF so it should be compared to the standard deviation of Gaussian fitting to real dataset. In the case of a significant disagreement, the Sigma parameter can be adjusted to match the experimental value. Similarly to other parameters, a variability on sigma can be added by setting sigma_std.
The simulated training dataset can be visualized/saved and downloaded once simulation is complete for assessment that the data appear reasonable to the user.
The next step is to generate training patches from the training dataset. This is performed in section 3.2. For Deep-STORM, 10,000 patches / 100 epochs tend to do a good job. For the batch size, keep a number divisible by 2 (16 or 32 for instance). The higher, the faster it’ll train, but the more GPU memory you’ll need. The number of steps should allow for the whole set of patches to be seen for each epoch. Therefore, a good rule of thumb for steps_per_epochs is to use (Total number of patches used for training)/(batch size) (This calculation will be automated in future versions of the notebook).
In section 6 of the notebook, you can finally use the model that you just trained to reconstruct super-resolved images from unseen high-density SMLM data. Along with the super-resolved image itself, the notebook will also extract single-molecule localizations coordinates using a Center of Gravity (CoG) estimator. This is especially useful as localization lists allow for post-processing, significantly improving image quality and providing access to downstream analysis approaches. This includes:
- Drift correction (can be performed directly in the notebook!)
- Filtering of localizations (e.g. spatiotemporal linking of localizations)
- Point pattern analysis (e.g. cluster analysis algorithms such as DBSCAN
First, you have to set the required parameters such as the data/results folder, pixel size or model to be used. To avoid running into RAM limitation issues, the time series are split into subsets, where the batch size represents the number of imaging frames that are transferred into the RAM for inference. This allows you to work with large field of views or long time series!
Finally, you have to set post-processing parameters in order to extract the localization coordinates. Deep-STORM hereby assigns a confidence value to each localization, which describes the likelihood for the predicted localization to be a ‘true’ localization. Higher values (closer to 1) hence represent a higher likelihood. The threshold sets the lower limit of the confidence value, meaning that you will get less localizations when setting this parameter to higher values. This value is also provided as a separate column in the localization file, therefore also allowing you to filter your localization dataset at a later time point (e.g. using ThunderSTORM).
The figure below illustrates the power of coordinate-based post processing. The dataset is a DNA-PAINT measurement of a cell labelled for β-tubulin consisting of only 2,000 camera frames! The Deep-STORM reconstruction without post-processing looks heavily impaired. Merging (linking) of localizations after the drift correction removes prominent spots in the image that are introduced by long-binding imager strands or a high label density at crossings. Filtering according to the confidence value is somehow comparable to filtering by localization precision, as often performed in conventional SMLM analysis pipelines.
To train Deep-STORM in Google Colab:
| Network | Link to example training and test dataset | Direct link to notebook in Colab |
|---|---|---|
| Deep-STORM | here |  |
or:
-
Download our streamlined ZeroCostDL4Mic notebooks
-
Open Google Colab
-
Once the notebook is open, follow the instructions.