CC BY
32INTERNATIONAL SCIENTIFIC AND TECHNICAL CONFERENCE "DIGITAL TECHNOLOGIES: PROBLEMS AND SOLUTIONS OF PRACTICAL IMPLEMENTATION IN THE SPHERES" APRIL 27-28, 2023
THEME: WORKING PRINCIPLES OF MULTI-FRAME IMAGE SUPERRESOLUTION
Mokhirjon Rikhsivoev1, Ahmed Yusupov2, Shohruh Begmatov3, Mukhriddin Arabboev4,
Khabibullo Nosirov5, Saidakmal Saydiakbarov6, Sardor Vakhkhobov7,Zukhriddin
Khamidjonov8
1,2Electronics and Radiotechnics Department,3,4 Television and Radio Broadcasting Systems
Department, 5Dean of Radio and Mobile Communications faculty, telecommunication
Technologies faculty, 7Television Technologies faculty, 8Computer Engineering faculty, 1-8 Tashkent University of Information Technologies named after Muhammad al-Khwarizmi,
Uzbekistan https://doi.org/10.5281/zenodo.7856110
Abstract. The subject of study in image processing has advanced significantly in recent years because to the strong rise in interest in image super-resolution. The quality of images can be improved using a variety of techniques.
In this research, the operating principles and image processing stages of one multi frame picture super-resolution approach were discussed. A sequence of low-resolution (LR) photos are combined to create a high-resolution (HR) image using the multi-frame image super-resolution (SR) technique. Recent years have seen a rise in interest in the issue of multi-frame SR reconstruction as it is necessary or desired in many actual applications. This group of algorithms often builds the association between the recorded LR pictures and the unidentified reconstructed HR image estimations using a linear observation model[1].
Keywords: Convolution neural network (CNN), Multi-frame super-resolution (MFSR), Single-frame super-resolution (SFSR)
1. Introduction
Multi-frame image Super-resolution is the practice of improving an image's resolution by merging many frames of the same picture taken at various points in time. A picture with a higher resolution than any single frame is produced by aligning and combining the frames. In contrast to
single-frame super-resolution techniques, the approach uses the variations between the frames to maintain fine features and eliminate noise.
The main goal of designing an algorithm for multi-frame super-resolution is to determine the optimal method for achieving alignment between successive video frames. Information from many nearby low-resolution frames may be combined using convolutional neural networks (CNN)
and motion compensation techniques, according to recent study. The majority of modern multiframe super-resolution algorithms employ a network topology that combines convolution with multi-frame fusion, with the input being the fusion of many video frames. This method provides the rebuilt picture with precise detailed details while ensuring the continuity of neighboring frames[2].
Multi-frame super-resolution works by combining multiple low-resolution frames of the
same scene to produce a single high-resolution image. The steps involved are Image acquisition, Image alignment, Frame combination image, Super-resolution reconstruction and last is Output.
The specific method used for combining the frames, such as averaging or weighted averaging, and the algorithm used for super-resolution reconstruction can vary depending on the
INTERNATIONAL SCIENTIFIC AND TECHNICAL CONFERENCE "DIGITAL TECHNOLOGIES: PROBLEMS AND SOLUTIONS OF PRACTICAL IMPLEMENTATION IN THE SPHERES" APRIL 27-28, 2023
application and the desired result. For example, deep learning-based methods can be used to learn a mapping from low-resolution to high-resolution images, leading to more sophisticated and effective results.
Figure-1. Process of super-resolution[3].
2. Working principles of MFSR
Image acquisition. Multiple frames of the same scene are captured at different times. The process of capturing an image using a camera or other imaging device and it involves capturing light that has reflected off a scene and converting it into an electronic representation, such as a digital image.
In the context of multi-frame image super-resolution, image acquisition involves capturing multiple frames of the same scene at different times. The frames are then used to create a highresolution image by combining the information from each frame.
Ficture-2. Image acquisition for image processing[4] The quality of the images captured during the image acquisition stage has a significant impact on the final result of the super-resolution process. Factors such as exposure time, aperture size, ISO, and focus can all affect the quality of the captured images and should be carefully controlled to ensure the best possible result.
Image alignment. Image alignment refers to the process of registering multiple images so that corresponding pixels in each image correspond to the same physical location in the scene. This is a crucial step in multi-frame image super-resolution, as it ensures that the frames can be combined effectively to produce a high-resolution image.
Image alignment can be performed in a number of ways, including feature-based alignment, optical flow-based alignment, and homography-based alignment. The specific method used will depend on the nature of the images and the desired result.
Feature-based alignment involves detecting and matching distinctive features in each image, such as corners or edges, and using these to align the images. Optical flow-based alignment involves estimating the motion of each pixel between the frames and using this information to
245
INTERNATIONAL SCIENTIFIC AND TECHNICAL CONFERENCE "DIGITAL TECHNOLOGIES: PROBLEMS AND SOLUTIONS OF PRACTICAL IMPLEMENTATION IN THE SPHERES" APRIL 27-28, 2023
align the images. Homography-based alignment involves finding a mathematical transformation that maps each pixel in one image to the corresponding pixel in another image.
Once the images have been aligned, they can be combined to form a high-resolution image that preserves the detail and reduces the noise present in each individual frame.
Figure-3. Process of Image alignment [5] Frame combination. The aligned frames are combined to form a high-resolution image. The goal of frame combination is to preserve the detail present in each frame and reduce the noise present in the individual frames.
There are several methods for combining frames, including simple averaging, weighted averaging, and more sophisticated methods such as maximum likelihood estimation. In simple averaging, each pixel in the final image is simply the average of the corresponding pixels in each frame. In weighted averaging, each pixel in the final image is a weighted average of the corresponding pixels in each frame, with the weights determined based on factors such as sharpness or noise level.
Maximum likelihood estimation is a more sophisticated method that takes into account the underlying noise model of the images and uses this information to determine the weights used in the averaging process. This can lead to a more accurate result and improved preservation of detail compared to simple or weighted averaging.
The specific method used for frame combination will depend on the nature of the images and the desired result. The choice of method can have a significant impact on the quality of the final result, and it is important to choose a method that is appropriate for the specific application and data.
INTERNATIONAL SCIENTIFIC AND TECHNICAL CONFERENCE "DIGITAL TECHNOLOGIES: PROBLEMS AND SOLUTIONS OF PRACTICAL IMPLEMENTATION IN THE SPHERES" APRIL 27-28, 2023
Figure-4. Super resolution combination framework[6]
Super-resolution reconstruction. The high-resolution image is then processed to further enhance its resolution, often using a single-frame super-resolution algorithm.
In the context of multi-frame image super-resolution, super-resolution reconstruction is performed on the high-resolution image created by combining multiple frames.
Super-resolution reconstruction algorithms can take a number of forms, including singleframe super-resolution algorithms and deep learning-based methods. Single-frame superresolution algorithms typically involve using mathematical models and optimization techniques to estimate the high-resolution image from the low-resolution input.
Deep learning-based methods involve training a neural network on a large dataset of highresolution and low-resolution images, with the goal of learning a mapping from low-resolution to high-resolution images. This can lead to more sophisticated and effective results, as the neural network can learn to capture complex patterns and relationships in the data.
The goal of super-resolution reconstruction is to produce a high-resolution image that is superior to the original low-resolution input. This can involve enhancing the detail and reducing the noise present in the image, as well as improving the overall visual quality of the image. The specific algorithm used for super-resolution reconstruction will depend on the nature of the images and the desired result, and can have a significant impact on the quality of the final result.
Figure-5. Framework of single-remote-sensing-image super-resolution (SR)
reconstruction [7]
Output. The output of multi-frame image super-resolution is a high-resolution image that is created by combining multiple low-resolution frames. The goal of the process is to produce an output image that is superior in terms of resolution, detail, and visual quality compared to the individual low-resolution frames.
The output image can be used for a wide range of applications, including scientific imaging, security and surveillance, and visual arts. The specific properties of the output image, such as resolution and noise level, will depend on the specific method used for image acquisition, alignment, frame combination, and super-resolution reconstruction.
In general, the output of multi-frame image super-resolution is a high-resolution image that preserves the detail present in each frame and reduces the noise present in the individual frames.
INTERNATIONAL SCIENTIFIC AND TECHNICAL CONFERENCE "DIGITAL TECHNOLOGIES: PROBLEMS AND SOLUTIONS OF PRACTICAL IMPLEMENTATION IN THE SPHERES" APRIL 27-28, 2023
The specific quality of the output will depend on factors such as the quality of the images captured during the image acquisition stage, the accuracy of the image alignment and frame combination, and the effectiveness of the super-resolution reconstruction algorithm.
Ground Truth Ground Truth (crop) ADMM VNG DeepJoint FlexISP
Figure-6. Output of multi frame image super-resolution (SR) reconstruction^]
CONCLUSION
In conclusion, multi-frame picture super-resolution is an effective method for boosting an image's resolution by combining a number of low-quality photos. Multi-frame super-resolution involves four main steps: image acquisition, image alignment, frame combination, and superresolution reconstruction. During image acquisition, multiple low-resolution images are captured of the same scene. Image alignment is then used to register the images and correct for any differences in viewpoint or camera settings.
The aligned images are then combined using a weighted average, a simple average, or other techniques to create a high-resolution image with more information than any of the individual low-resolution images. Finally, super-resolution reconstruction is used to increase the resolution of the combined image even further.
REFERENCES
1. X. L. Li, Y. Hu, X. Gao, D. Tao, and B. Ning, "A multi -frame image super-resolution method," Signal Processing, vol. 90, no. 2, pp. 405-414, 2010, doi: 10.1016/j.sigpro.2009.05.028.
2. C. Guo, J. Jiang, Q. Wang, and G. Chen, "Multi-Frame Super-Resolution Algorithm Based on Attention Mechanism," in 2021 6th International Conference on Signal and Image Processing, ICSIP 2021, 2021, pp. 431-435. doi: 10.1109/ICSIP52628.2021.9688891.
3. J. A. Kennedy, O. Israel, A. Frenke, R. Bar-Shalom, and H. Azhari, "Improved image fusion in PET/CT using hybrid image reconstruction and super-resolution," Int. J. Biomed. Imaging, vol. 2007, no. April 2014, 2007, doi: 10.1155/2007/46846.
4. V. K. Mishra, S. Kumar, and N. Shukla, "Image Acquisition and Techniques to Perform Image Acquisition," SAMRIDDHI A J. Phys. Sci. Eng. Technol., vol. 9, no. 01, pp. 21-24, 2017, doi: 10.18090/samriddhi.v9i01.8333.
INTERNATIONAL SCIENTIFIC AND TECHNICAL CONFERENCE "DIGITAL TECHNOLOGIES: PROBLEMS AND SOLUTIONS OF PRACTICAL IMPLEMENTATION IN THE SPHERES" APRIL 27-28, 2023
5. T. Tsujikawa et al., "Quantitative Multiplex Immunohistochemistry Reveals Myeloid-Inflamed Tumor-Immune Complexity Associated with Poor Prognosis," Cell Rep., vol. 19, no. 1, pp. 203-217, 2017, doi: 10.1016/j.celrep.2017.03.037.
6. J. Jameson, S. Abdullah, N. Ghazali, and N. A. Zamani, "Multiple Frames Combination Versus Single Frame Super Resolution Methods for CCTV Forensic Interpretation," J. Inf. Assur. Secur., vol. 8, no. 5, pp. 230-239, 2013.
7. H. Zhu et al., "Super-resolution reconstruction and its application based on multilevel main structure and detail boosting," Remote Sens., vol. 10, no. 12, 2018, doi: 10.3390/rs10122065.
8. B. Wronski et al., "Handheld Multi-Frame Super-Resolution," vol. 38, no. 4, pp. 1-18, 2019.
