CC BY
10
Super-resolution microscopy based on interpolation and wide spectrum de-noising
T. Cheng1, T. Chenchen1 1 Guangxi University of Science and Technology, 545006, P.R. China, Liuzhou, Chengzhong District, Avenue Donghuan, 268
Abstract
In the conventional single-molecule localizations and super-resolution microscopy, the pixel size of a raw image is approximately equal to the standard deviation of the point spread function. Such a raw image is referred to herein as a conventional raw image, based on which better single molecule localization effect and efficiency can be achieved. It is found that both interpolation and de-noising can effectively improve the Signal to Noise Ratio of the conventional raw image. The conventional raw image, the de-noised, the interpolated and the de-noised interpolated are compared and analyzed and compressed sensing is used for super-resolution reconstruction. The simulation results show that both the highest Signal to Noise Ratio and the best super-resolution reconstruction can be obtained by de-noising the interpolated conventional raw image. This method also renders the best super-resolution reconstruction and minimum gradient in the real experiment. De-noising the interpolated conventional raw image is an effective method to improve the super-resolution microscopy.
Keywords: super-resolution microscopy; interpolation; de-noising; point spread function; compressed sensing.
Citation: Cheng T, Chenchen T. Super-resolution microscopy based on interpolation and wide spectrum de-noising. Computer Optics 2023; 47(4): 614-619. DOI: 10.18287/2412-6179-CO-1272.
Introduction
Optical microscopy imaging has several advantages such as magnification, real-time visualization and noninvasive observation, making optical microscope a basic tool for cell research. Owing to the diffraction phenomenon, the point light source forms a diffuse spot (i.e., an Airy disc) on the focal plane, which is the point spread function (PSF) of the microscope. The diffraction limit restricts an optical microscope to resolve the subcellular organization and cell organelles. However, super-resolution microscopy (SRM) imaging based on fluorescent probes can observe the inside of cells and subcellular life activities. It breaks the diffraction limit and increases the imaging resolution [1 - 6].
Super-resolution microscopy (SRM) techniques include single-molecule localization microscopy (SMLM), stimulated emission depletion microscopy (STED), structured illumination microscopy (SIM) and superresolution optical fluctuation imaging (SOFI), etc., among which SMLM achieves the highest resolution. SMLM methods include (fluorescence) photoactivated localization microscopy ((F)PALM), stochastic optical reconstruction microscopy (STORM), and DNA-based point accumulation for imaging in nanoscale topography (DNA-PAINT), etc[1 - 5, 7 - 9]. Since thousands frames of raw images in the same field of view are acquired for single molecule localization in SMLM, the temporal resolution of SMLM is very poor. Fluorescent molecules (fluorophores or emitters) must be sparsely activated in SMLM. If PSFs of some fluorescent molecules overlap, these fluorescent molecules will not be efficiently localized [3, 4, 10 - 14].
Compressed sensing (CS) can reconstruct the raw image with high density of fluorescent molecules. PSFs of fluorescent molecules can overlap in CS, greatly reducing the number of frames of the raw image required to reconstruct the final super-resolution image. Thus the temporal resolution is much better than SMLM [13 - 16]. The CS mathematical model between a raw image and a super-resolution image is shown in Eq. (1). The vectors, y and x consist of columnwise concatenations of the raw image and the superresolution image (i.e., pixelated original image) respectively. The measurement matrix A is subject to the PSF of the microscope. The acquired raw image corresponds to the ith column of A if only one molecule emits fluoroscopic photons at the position index i of x [13 - 16].
min||x ||o s.t. y = Ax, (1)
where xe RN, ye RM, Ae RMxN, M < N, M and N are natural numbers. x is a vector containing N elements. y is a vector containing M elements. A is a matrix of size M x N. min||x||o is a objective function. y = Ax is a constraint function. If the objective function is min||x||1 rather than min||x||o, Eq. (1) is transformed to a convex optimization problem.
In conventional SMLMs, if the pixel size of the raw image is approximately equal to the standard deviation (s.d.) of the PSF, good and fast localization can be achieved. The existing research results show that interpolation can also improve the CS reconstruction [13, 14, 17]. Interpolation based on the bicubic method can make the pixel size of the raw image become half of the original's [14].
Noise in the raw image is inevitable. Especially in low light environment, noise can almost drown the signal.
The noises of a raw image mainly include a shot noise following a Poisson distribution, a readout noise following a Gaussian distribution and a background. The reconstruction of CS is also restricted by noise [4, 12, 17 - 19]. Large noise can even lead to reconstruction failure. Wide spectrum denoising (WSD) is a SRM de-noising algorithm for various random noises, which helps improve the reconstruction of CS [18, 20]. Hereinafter, the raw image whose pixel size is approximately equal to the s.d is referred to as a conventional raw image (CR). CI is the interpolated CR. CW and CIW respectively represent the de-noised CR and CI with WSD.
The denoising effect and CS reconstruction of CIW are studied in this paper, and they also compared with those of CR, CW and CI. It is found that both interpolation and de-noising can effectively improve the Signal to Noise Ratio (SNR) of the CR and the highest SNR can be obtained by de-noising the CI. The simulation and real experimental results show that the best super-resolution reconstruction can be obtained by de-noising the CI. Thus de-noising the CI is an effective method to improve the super-resolution microscopy.
1. Wide spectrum de-noising and quantitative evaluation
WSD can effectively remove random noise such as Poisson and Gaussian noise from very low density to ultra-high density fluorescent molecular distribution scenarios. If the measurement matrix A is operated by orthogo-nalization and normalization, the measurement matrix Ao can be obtained. Row orthogonal normalization is an operation in the matrix theory. It makes the rows of the matrix completely orthogonal. Moreover, it makes the 2-norm of each row of the matrix equal to 1. Through Ao and A, the operator matrix T which is equivalent to the row orthogonal normalization operation can be obtained, where T = AOAT (AAT) -1. Therefore, y = Ax can be equivalently converted to Ty = TAx. If a singular value decomposition is applied to T, T = USVT. S is a diagonal sparse matrix that is composed of singular values of T. VT is a transpose matrix of V. Therefore, Ty = TAx can be equivalently converted to SVTy = SVTAx. Let z = SVTy. z's elements greater than cri are set to cri. The new z is represented by zwsd. The threshold value cri is the largest absolute values of elements of z between the indices M*0.9 and M*0.95. The de-noised raw image is yWSD, yWSD = T -1UyWSD [18, 20].
SNR can indicate the overall quality of the de-noised raw image and the reconstructed super-resolution image [14]. Structural similarity index measure (SSIM) can measure the similarity between two images. The SSIM values range between 0 and 1. If the two images are identical, the value of SSIM is equal to 1[10, 21].
SNR = 20 X log l0
(2)
where x is a true signal, xe RN; ||||2 is the norm of a vector; and x^ denotes the reconstructed signal corresponding x.
SSIM (x, y ) =
(2|i x | y + c0(2a xy + c2)
(|2 +| y + c,
1)( + ay + c2^
(3)
where |x and Ox represent the mean and the variance of image x, respectively. Oxy represents the covariance of image x and y. c = (k\L)2 and c2 = kL)2. L is the dynamic range of pixel values. k\ = 0.01, k2 = 0.03.
2. Interpolation and de-noising of simulated conventional raw images
In real experiments, an inverted optical microscope with a 100* oil immersion objective lens (N.A. 1.40) was used. Tubulin of HeLa cells was stained with Alexa-647. The fluorescence wavelength was 670 nm. The fluorescent lights were acquired using an EMCCD. Twenty CR frames were collected. The PSF was a Gaussian function. The full width at half maximum (FWHM) of the PSF was 239.286 nm. The same parameters were used in simulation experiments.
To be distinguished from the pixels of the CR, the pixels of the super-resolution image were referred to as grids. The super-resolution image's grid was 1/8 of the pixel size of the CR. If the size of the CR was 7 * 7 pixels, it was located in the middle of the 64 * 64 grid superresolution image. The pixel size of the CI was 1/2 of the pixel size of the CR.
To evaluate the de-noising performance of WSD for different molecular densities and pixel sizes for CR and CI, simulated CRs with known real molecular positions were generated. The simulation randomly placed K molecules in a 64 * 64 grid region. K represented the sparsity in CS. K ranged from 1 to 16, with corresponding molecular densities from1.869 to 30 |m2 The grid size was 11.43 nm. The effective pixel size of the CR, 45.714 nm, matched the pixel size of the microscope setup and was roughly equal to half of the s.d. of the PSF of the setup. The size of the CR was 7 * 7 pixels in the middle of the 64 * 64 grid super-resolution image. The simulation was for a photon number of 3,000 per molecule and a uniform background of 16 photons per pixel. Poisson noise and Gaussian noise (Gaussian noise variances of 0.01 and 0.0025) were added to each CR frame. The size of the CI was 13*13 pixels.
Fig. 1 is mean SNRs of 500 frames of CR and CI before and after de-noising. For each K, the simulation was repeated 500 times as shown in Fig. 1. CI is the CI. CW and CIW respectively represent the CW and CI with WSD.
The de-noising performance analysis curves are shown in Fig. 1a and b. Both interpolation and de-noising can effectively improve the SNR of the CR. The highest SNR can be obtained by de-noising the CI. In Fig. 1a for the high noise environment (Gaussian noise with a variance of 0.01), the SNR of the CI is lower than that of the CW. In Fig. 1b for the low noise environment (Gaussian
noise with a variance of 0.0025), the SNR of the CI is almost the same as that of the CW.
Molecular density (|m~2)
-16
a)
24
Sparsity K
Molecular density (|m-2) 5 10 15 20 25
30
24
20
b)
8
Sparsity K
Fig. 1. Comparison of mean SNRs based on 500 simulated CRs before and after interpolation and de-noising. The simulation is for a photon number of3,000 per molecule and a background of 64 photons per pixel. Poisson noise and Gaussian noise are added to each frame of the CR. The y axis is labeled as SNR (dB). The x axis is labeled as both molecular density and signal sparsity. a) Gaussian noise variance of 0.01, b) Gaussian noise variance of0.0025
Figs. 2 and 3 are one simulation of Fig. 1a at K equal to 4. Fig. 2a is the true super-resolution image containing four fluorescence molecules. Fig. 2b is the noisefree CR corresponding to Fig. 2a. Fig. 2c is the noisefree CI.
a)
c)
Fig. 2. Comparison of the true super-resolution image, the corresponding simulated noisefree CR and the corresponding
simulated noisefree CI. a) 4 molecules, b) noisefree CR, c) noisefree CI
Fig. 3a is the CR after adding background, Poisson noise and Gaussian noise with variance of 0.01 to Fig. 2b. Fig. 3b is the CW. Fig. 3c and d are respectively interpolated Fig 3a and de-noised Fig. 3c. SNRs of the CR and CI after de-noising is improved by 2.485 dB and 1.814 dB respectively.
3. Simulation data analysis
To evaluate the CS reconstruction for CIW, 20 frames of simulated super-resolution image with known true molecular positions and their corresponding CR frames were generated. The simulation randomly placed molecules in a grid region. The molecular density was 6.304 |m -2.
Fig. 4a is a CR frame without noise. Fig. 4b is a CI frame without noise. Fig. 4c is the true superresolution image.
a) m m m b) ^m ^m C) m^ ^^m dj
Fig. 3. Comparison of the CR (Gaussian noise with a variance of 0.01), the CW, the CI, and the CIW. The numbers indicate SNR (dB) of the images a) CR 14.421 dB, b) CW 16.906 dB, c) CI 16.366 dB, d) CIW 18.18dB
24
12
8
0
16
12
8
0
4
12
16
a) iihh b) m C) \
Fig. 4. Comparison of the simulated noisefree CRs, the corresponding simulated noisefree CI, and the true super-resolution image,
Scale bars: 1 pm. a) noisefree CR, b) noisefree CI, c) True Super
Raw
a)
c,^^^^^^ j,g)h)\ Fig. 5. Comparison of the simulated (interpolated) CR (Gaussian noise with a variance of 0.01) and the corresponding superresolution images before and after de-noising. The upper row is one frame of CR and CI before and after de-noising. The lower row is the final super-resolution images corresponding to the upper row. The number pairs indicate SSIM and SNR (dB) of the images. Scale bars: 1 jm. a) CR (0.312, 7.059 dB), b) CW (0.452, 11.397 dB), c) CI (0.25, 8.615 dB), d) CIW (0.325, 10.791 dB),
e) CR, f) CW(0.574, -2.093 dB), g) CI, h) CIW(0.671, -2.021 dB)
Raw
a)
giHBHBa h)\
Fig. 6. Comparison of the simulated (interpolated) CR (Gaussian noise with a variance of0.0025) and the corresponding superresolution images before and after de-noising. The upper row is one frame of CR and CI before and after de-noising. The second row is the final super-resolution images corresponding to the upper row. The number pairs indicate SSIM and SNR (dB) of the images. Scale bars: 1 jm. a) CR (0.45, 12.261 dB), b) CW (0.547, 15.86 dB), c) CI (0.373, 13.873 dB), d) CIW (0.433, 15.977 dB), e) CR, f) CW (0.637, -1.76 dB), g) CI (0.779, -1.68 dB), h) CIW (0.796, -1.411 dB)
Fig. 5a is one CR frame after adding background, Poisson noise and Gaussian noise with variance of 0.01 to Fig. 4a. Fig. 5b is the CW frame corresponding to Fig. 5a. Fig. 5c and d respectively are the CI and the CIW frames. SNRs of Fig. 5b and d increased by 4.338 dB and 2.176 dB than Fig. 5a and c. SSIMs of Fig. 5b and d increased by 0.14 and 0.075 than Fig. 5a and c.
The lower row of Fig. 5 is the final super-resolution image reconstructed by CS. SSIM and SNR of Fig. 5h reached the maximum, respectively 0.671 and - 2.021 dB. The cell microtubule structure is clearer and thinner. SSIM and SNR increased by 0.097 and 0.072 dB respectively compared with Fig. 5f. The CS reconstruction of Fig. 5e and g failed. No valid microtubule structures can be seen. Although some scattered structures can be seen in Fig. 5g, the result is too poor to be useful. The microtubule structure of the rectangle area in Fig. 5h is cleaner than that
in Fig. 5f All three microtubules are clearly identifiable in Fig. 5h. However, only two microtubules can be seen clearly in Fig. 5f. Therefore, the CS reconstruction of Fig. 5h is the best.
Fig. 6a is one CR frame after adding background, Poisson noise and Gaussian noise with variance of 0.0025 to Fig. 4a. SNRs of Figs. 6b and d respectively increased by 3.599 dB and 2.104 dB than Figs. 6a and c. SSIMs of Figs. 6b and d respectively increased by 0.097 and 0.06 than Figs. 6a and c.
The second row of Fig. 6 is the final super-resolution image reconstructed by CS. SSIM and SNR of Fig. 6(h) reached the maximum, respectively 0796 and - 1.411 dB. The cell microtubule structure is clearer and thinner. SSIM and SNR increased by 0.159 and 0.349 dB respectively compared with Fig. 6f. SSIM and SNR are better than Fig. 6g, too. The CS reconstruction of Fig. 6e failed.
No valid microtubule structures can be seen. The area between microtubules in the rectangular area in Fig. 6h has almost no discrete points, which is cleaner than that in Fig. 6f and g. Therefore, the CS reconstruction of Fig. 6h is the best.
4. Real experimental data analysis
Fig. 7 shows the real experimental CR and CI before and after de-noising and the corresponding CS reconstruction results. Fig. 7a - d of the upper row is CR and CI frames before and after de-noising. Since the real raw image cannot be known, the sum of the absolute values of the horizontal and vertical gradients of the raw
image is used as an indicator of measuring the de-noising effect. The gradients of CR, CW, CI, and CIW were 7.458x105, 7.146x105, 4.25x105 and 3.974x105 respectively. Through comparison of the images, it is obvious that the raw image after denoising is smoother than that before denoising, indicating prominent de-noising effects.
Comparing the CS reconstruction results of CR, CW, CI and CIW in the lower row, we can see noticeably improved spatial resolutions of CI and CIW and clearer, thinner cell microtubule structure. The super-resolution reconstruction of Fig. 7f is the worst, and Fig. 7g is the second-worst.
Raw
a)
Fig. 7. Comparison of the experimental (interpolated) CR and the corresponding final super-resolution images before and after denoising. (a)-(d) of the upper row is one frame of the experimental CR and CI before and after de-noising. The number indicate gradient of the images. (f)-(i) of the lower row is corresponding final super-resolution images before and after denoising, respectively. (e) and (i) is plots of photon-count profiles obtained by measurements made along the red lines in (h)-(i). The maximum wave peak is 4300. Scale bars: 1 m a) CR 7.458*105, b) CW 7.146*105, c) CI 4.25*105, d) CIW3.974*105, e) CI-profile, f) CR,
g) CW, h) CI, i) CIW, j) CIW- profile
The peaks in Fig. 7e and j correspond to the photon-count profiles at the red line in Fig. 7h and i. The highest peak value is 4300. The peak in Fig. 7j is thinner and less than Fig. 7e. Therefore, the super-resolution reconstruction of Fig. 7i is the best.
Conclusion
In the conventional single-molecule localizations and super-resolution microscopy, the pixel size of a raw image is approximately equal to the standard deviation of the point spread function. Both interpolation and de-noising can effectively improve the SNR of the CR. It is found that the highest SNR and minimum gradient can be obtained by de-noising the CI. In the high noise environment, the SNR of the CI is lower than that of the CW and the CI cannot achieve effective super-resolution reconstruction, while in the low noise environment, the SNR of the CI is almost the same as that of the CW. The simulation and real experimental results show that the best super-resolution reconstruction can be obtained by de-noising the CI. In conclusion, de-noising the CI is an effective method to improve the super-resolution microscopy.
Acknowledgements
The work was funded by Guangxi National Natural Science Foundation (2022GXNSFAA035593), National Natural Science Foundation of China (81660296, 41461082).
References
[1] Gabitto MI, Marie-Nellie H, Pakman A, Pataki A, Darzacq X, Jordan MI. A Bayesian nonparametric approach to
super-resolution single-molecule localization. Ann Appl Stat 2021; 15(4): 1742-1766. DOI: 10.1214/21-AOAS1441.
[2] Nevskyi O, Tsukanov R, Gregor I, Karedla N, Enderlein J. Fluorescence polarization filtering for accurate single molecule localization. APL Photon 2020; 5(6): 061302. DOI: 10.1063/5.0009904.
[3] Costello I, Cox S. Analysing errors in single-molecule localisation microscopy. Int J Biochem Cell Biol 2021; 27(2): 105931. DOI: 10.1016/j.biocel.2021.105931.
[4] Rimoli CV, Valades-Cruz CA, Curcio V, Mavrakis M, Brasselet S. 4polar-STORM polarized super-resolution imaging of actin filament organization in cells. Nat Commun 2022; 13(1): 301. DOI: 10.1038/s41467-022-27966-w.
[5] Kwon J, Elgawish MS, Shim SH. Bleaching-resistant super-resolution fluorescence microscopy. Adv Sci 2022; 9(9): 2101917. DOI: 10.1002/advs.202101817.
[6] Henriques R, Lelek M, Fornasiero EF, Valtorta F, Zimmer C, Mhlanga MM. QuickPALM: 3D real-time photoactivation nanoscopy image processing in ImageJ. Nat Methods 2010; 7(5): 339-340. DOI: 10.1038/nmeth0510-339.
[7] Kozma E, Kele P. Fluorogenic probes for super-resolution microscopy. Org Biomol Chem 2019; 15(17): 215-233. DOI: 10.1039/c8ob02711k.
[8] Chung J, Jeong U, Jeong D, Go S, Kim D. Development of a new approach for low-laser-power super-resolution fluorescence imaging. Anal Chem 2021; 101(94): 618-627. DOI: 10.1021/acs.analchem.1c01047.
[9] Jeong D, Kim D. Super-resolution fluorescence microscopy-based single-molecule spectroscopy. Bulletin of the Korean Chemical Society 2022; 43(12): 316-327. DOI: 10.1002/bkcs.12471.
[10] Roa C, Le VND, Mahendroo M, Saytashev I, RamellaRoman JC. Auto-detection of cervical collagen and elastin in Mueller matrix polarimetry microscopic images using. Biomed Opt Express 2021; 12(4): 2236-2249. DOI: 10.1364/BOE.420079.
[11] Holden SJ, Uphoff S, Kapanidis AN. DAOSTORM: an algorithm for high-density super-resolution microscopy. Nat Methods 2011; 8(4): 279-280. DOI: 10.1038/nmeth0411-279.
[12] Junhong M, Cedric V, Hagai K, Lina C, Nicolas O, Seamus H, Suliana M, Chul YJ, Michael U. FALCON: fast and unbiased reconstruction of high-density superresolution microscopy data. Sci Rep 2014; 4(4): 4577. DOI: 10.1038/srep04577.
[13] Zhu L, Zhang W, Elnatan D, Huang B. Faster STORM using compressed sensing. Nat Methods 2012; 9(7): 721723. DOI: 10.1038/nmeth.1978.
[14] Cheng T, Chen DN, Yu B, Niu HB. Reconstruction of super-resolution STORM images using compressed sensing based on low-resolution raw images and interpolation. Biomed Opt Express 2017; 8(5): 2445-2457. DOI: 10.1364/B0E.8.002445.
[15] Arjoune Y, Kaabouch N, Ghazi HE, Tamtaoui A. A performance comparison of measurement matrices in compressive sensing. Int J Commun Syst 2018; 31(2): e3576. DOI: 10.1002/dac.3576.
[16] Calisesi G, Ghezzi A, Ancora D, D'Andrea C, Valentini G, Farina A, Bassi A. Compressed sensing in fluorescence microscopy. Prog Biophys Mol Biol 2022; 60(6): 66-80. DOI: 10.1016/j.pbiomolbio.2021.06.004.
[17] Thompson RE, Larson DR, Webb WW. Precise nanometer localization analysis for individual fluorescent probes. Biophys J 2002; 43(82): 2775-2783. DOI: 10.1016/s0006-3495(02)75618-x.
[18] Cheng T, Chen DN, Li H. Wide spectrum denoising (WSD) for superresolution microscopy imaging using compressed sensing and a high-resolution camera. J Phys Conf Ser 2020; 1651: 012177. DOI: 10.1088/17426596/1651/1/012177.
[19] Beier HT, Ibey BL. Experimental comparison of the highspeed imaging performance of an EM-CCD and sCMOS camera in a dynamic live-cell imaging test case. PLoS ONE 2014; 18(9): e84614. DOI: 10.1371/journal.pone.0084614.
[20] Cheng T. Wide spectrum denoising method for microscopic images. US Patent 16845110 of July 2, 2022.
[21] Lee G, Oh JW, Her NG, Jeong WK. DeepHCS ++ : Bright-field to fluorescence microscopy image conversion using multi-task learning with adversarial losses for labelfree high-content screening. Med Image Anal 2021; 70: 101995. DOI: 10.1016/j.media.2021.101995.
Authors' information
Tao Cheng, Doctor of Technical Sciences, Professor, Professor of School of Mechanical and Automotive Engineering of Guangxi University of Science and Technology. Research interests: medical imaging and image processing. He is also the corresponding author of this paper. E-mail: ctnp@J63.com .
Chenchen Tao, Master's student of School of Foreign Studies of Guangxi University of Science and Technology. Research interests: microscopy and image processing. She is also the corresponding author of this paper. E-mail:
taochenchen621@,126.com .
Received January 6, 2023. The final version - February 20, 2023.
