CC BY
75
Gradient-based technique for image structural analysis and applications
D. G. Asatryan1,2 'Russian-Armenian University, Armenia, Yerevan, 2Institute for Informatics and Automation Problems of National Academy of Sciences of Armenia, Armenia, Yerevan
Abstract
This paper is devoted to application of gradients field characteristics in selected problems of image intellectual analysis and processing. To analyse the properties and structure of an image several approaches and models based on the use of the gradients field characteristics, are proposed. In this paper, models based on Weibull distribution are considered, an image dominant direction estimation algorithm using the parameters of scattering ellipse of gradients field components is proposed, and a similarity measure of two images with arbitrary dimensions and orientation is proposed. Some examples of applications of these models for estimation of blur and structuredness of an image, for the quality assessment of resizing and rotating algorithms, as well as for detection of a specified object on the image delivered by an unmanned aerial vehicle, are given.
Keywords: Image gradient field, Weibull distribution, similarity measure, dominant orientation, blur estimation, video stream analyse.
Citation: Asatryan DG. Gradient-based technique for image structural analysis and applications. Computer Optics 2019; 43(2): 245-250. DOI: 10.18287/2412-6179-2019-43-2-245-250.
Acknowledgments: This work was supported by the State Committee of Science of the Ministry of Education and Science of the Republic of Armenia, under the research project No. 15T-1B274 and by Russian-Armenian University under the research project funded by the Ministry of Education and Science of the Russian Federation.
Introduction
In last two decades, image processing literature has introduced new image processing techniques that are aimed at obtaining results that are better aligned with the perception of human visual system (HVS) than with traditional methods using standard-mean-square criteria. The basis for the development of such methods was the understanding that HVS extracts structural (substantive) information from an image and it is being well adapted for this purpose. In particular, in the papers of American professor A. Bovik and his colleagues, measures for assessing an image quality based on the using of its structural properties are proposed, and the numerous publications of these authors and their followers clearly demonstrate the effectiveness of proposed approaches [1 - 3].
The fruitfulness of above philosophy has prompted researchers to look for other measures that use the structural properties of an image to expand the range of problems to be solved, and in some cases, to improve the results obtained.
Moreover, there were created image databases, intended for evaluation of visual quality assessment metrics of interest. For instance, the database described in [4] contains large number of test images by many types of distortions with different levels. There were many types of distortions among them which are representing the structural changes. Then Mean opinion scores for this database have been collected by performing subjective experiments with volunteers and given in the paper. This information allows the use of the design database as a fundamental tool for assessing the effectiveness of visual quality.
A comprehensive review of the literature on this topic deserves a separate study and is beyond the scope of this paper. However, it is well known that a huge number of methods are devoted to algorithms for deter-
mining various structural elements of an image (edges and boundaries, contours, corners etc.) which somehow or other are using for the creating of quality assessment measures. Many of these measures rely on the use of gradients of pixel intensities so the technique based on image gradient properties are topical and needs to be more widely applied in the practice to analyse of an image structure (see, for example, [5]).
The present work is also based on the use of information delivered by gradients, treating it as a characteristic of the structure of an image.
Basic mathematical models
Let's consider the basic mathematical models used in the paper.
Image gradients field. Mathematically, the gradient of a function of two variables for each point of an image is a two-dimensional vector, the components of which are horizontal and vertical derivatives of the brightness of an image. The brightness function for an image is known only at discrete points [6].
Let a halftone image I be given, the intensity matrix elements of which take values I(m, n)e{0, 1, ..., 255}, m = 0, 1, ..., M- 1, n = 0, 1, ..., N- 1. The components of image gradients field are usually determined using the Sobel operator 6[4]. The use of the Sobel operator to calculate the vertical GV and horizontal GH components of the gradients field is reduced to the convolution of a local neighborhood of size 3*3 of each image point with the elements of the matrix denoted by A, with masks of the same dimension denoted by SV and SH, respectively,
' a a2 a3 N
A = a4 a5 a6
y a7 a8 a9,
Sv =
(-1 - 2 -1Ï 0 0 0
v 121 y
Sh =
(-1 0 1Ï -2 0 2 -1 0 1
As a result, the following linear combinations of matrix elements are obtained,
GV = -ax - 2a2 - a3 + a7 + 2a8 + a9, GH = -ax - 2a4 - a7 + a3 + 2a6 + a9.
(1) (2)
and for corresponding points of the image gradient magnitude M is calculated by formula as follows
M = JG
2 + G 2
V + ^H
(3)
The set of pairs (GV, GH) calculated for the whole image I is called gradients field of the image.
We consider the gradients field as an object that in a certain sense characterizes the structure of an image. In doing so, we proceed from the thesis that a number of properties of the gradients field are accessible to the perception of HVS when visualizing the image and extracting the necessary information from it. To illustrate what has been said, we give a few simple examples with calculated magnitudes of the gradients for the entire image and, observing certain requirements, visualize the corresponding matrix (see figure 1).
Fig. 1. Examples with visualized gradient magnitude We see that by these images it can be easy recognized the original images. Moreover, the basic structural elements and objects are well distinguishable and can be a part of description of the original image. Thus, when extracting information from an image, the HVS makes substantial use of the informativeness of the gradients field. By the way, this circumstance served as the basis for carrying out a huge number of scientific works in the world on image processing using gradients field (it is enough to mention the work on the definition of edges, borders and contours).
In this paper, a range of problems is expanded by using certain structural properties of gradients field. These properties, models and applications are considered below.
Scattering ellipse of the gradients field. By |H, |V, cth, ctv denote the mean, standard deviations of the gradient components GV, GH, by pHV the correlation coefficient between components. The family of scattering ellipses are determined by formula as follows
1
(gH - M"H )2 2pHV (gH - M"H )(gv - Vv )
1 -pHv
(gv -Vv )2
g2h
Gh Gv
(4)
= C2
where C is a constant.
The principal axis of the ellipse (4) coincides with the orthogonal regression line [7], and the slope of the principal axis is given by formula as follows
tga =-
a h Gv PHV
gH
-GV -
^(g2h -gv )2
(5)
+ 4GH a2 pHv
Formulas (4) and (5) we consider as an estimate of the "dominant orientation" or the "dominant direction" of the image [8]. This formula has proved useful in a number of processing problems, in particular, in image registration problems.
Figure 2 shows several images with estimates of dominant direction (note the agreement of the estimates to HVS perception results).
. yrfKESiS'"
■ . x>
A'.' '■ ./
v-r. •/ -A
/7>->V
Fig. 2. Image examples with the results of estimating the dominant orientation: a= 65° (a), a= 18° (b), a= 136° (c)
Models based on the distribution of gradient magnitude. Obviously, in general case, it is not entirely correct to talk about a certain distribution of the pixel intensities of the entire image, because any image is mostly semantic and often contains deterministic details. However, formulas (1) - (2) allow, based on an approximate implementation of the central limit theorem, with at least some reservations for small neighborhoods of each pixel, to propose models that give approximate solutions of certain problems. Proceeding from this observation, for gradient magnitude distribution we take the two-parameter Weibull distribution density function [9]. Note that in the
literature on image processing, the Weibull distribution, the generalized Gaussian distribution and other distributions from the exponential family are often used to describe the gradient magnitude.
Weibull distribution density function is given by formula as follows
f ( x; X, n) =
U x
n-1
exp
x > 0,
(6)
where n > 0 is the shape parameter, X > 0 is the scale parameter.
Estimation of parameters n and X can be carried out by the method of moments using the matrix M. In particular, the shape parameter is estimated, solving with respect to n the equation
— +1 =
|2 r2
r(l + 2/n ) (1 +1/n )
(7)
where r(-) is the Gamma function, and p., ct2 are sample mean and variance calculated by elements of the matrix M. For the numerical solution of equation (7), the method of dividing the segment in half is used. Similarly, the scale parameter is estimated. However, since the right-hand side of equation (7) depends monotonically on the parameter n, in the problems of comparative analysis one can use the values of the left-hand side without solving equation (7).
Images similarity assessment measure. The similarity (proximity) of two images can be estimated as the corre-
sponding empirical Weibull distributions are constructed, using the samples of gradients magnitudes of the images being compared. However, instead of nonparametric fitting criterion, we apply a less precise but simple measure based on the proximity of estimates of Weibull distribution parameters, which is determined by following formula
min(ni, n2) min(Xi, X2)
W2 =-
max(n1, n2 ) max(X1, X 2 )
0 < W2 < 1.
(8)
This measure proved to be convenient and very useful in the problems of analyzing and processing of large arrays of images. In particular, the measure (8) is robust under rotations, shifts and cropping of images, if the structural information is not great distorted under these transformations [8].
Examples. Figure 3 shows the reduced copies (a), (d). (e) and (f) of originals of chosen images and their transformations with sizes shown under images. Image (b) was obtained by rotating the image (a) by 45°, image (c) was obtained by increasing the (a) 1.5 times, images (d) and (e) was obtained by a small cropping of an original, and the image of (g) was obtained by resizing, rotating and cropping the image (f) simultaneously.
Table 1 shows the results of similarity estimating of corresponding images using known measure Peak signal-to-noise ratio (PSNR) and the measure W2 proposed above. We see the preference of similarity measure W2 relative to PSNR which is valuable only for images with the same sizes.
Fig. 3. Examples of images and their transformations: 236*236 (a, b); 354*354 (c); 511*511 (d, e); 256*256 (f); 240*240 (g) Table 1. Results of similarity estimation for pairs of images from figure 3
(a) and (b) (a) and (c) (b) and (c) (d) and (e) (f) and (g)
PSNR, dB 12.5 - - 18.5 13.5
W 2 0.97 0.94 0.96 0.998 0.97
2
The above robustness property of proposed similarity measure relative to the scaling and rotation of an image allows us to successfully carry out research in various applied problems. In particular, this measure can be used for classification of textures [10], at processing of video sequences filmed by moving camera [11] etc.
Applications
Image blur estimation. Estimating the degree of blur is an important step in improving image quality. In the literature, many approaches, criteria and algorithms for estimating the degree of blurring are proposed [12-14], particularly using the properties of the image gradients field. Our recent studies have shown that the value of the statistical estimate of shape parameter n of Weibull distribution can be used as a blur measure. In [15], on simulated blur data of various images, it is shown that the value of the parameter estimate increases monotonically as the degree of blurring increases. For example, Figure 4
shows an image and its blurred patterns, with the corresponding values of parameter n.
It is also shown in [15] that in general the parameter n can be interpreted as a measure for image structuredness.
Estimating the quality of an image resizing and rotation algorithms. Execution of any transformation of an image with purposes of resizing or rotating causes a distortion of the source information, the degree of which depends on the used transformation method. Judging by the available information on the Internet, the quality of transformed image is usually estimated visually, which is not acceptable because of the ambiguity visual perception of different people. In the literature also are described methods for investigating the quality of scaling algorithms based on aligning the size of the transformed image with the original one so that it can be compared by pointwise methods (see, for example, [16]). But obvious that in this case the interpolation error occures twice which hampers to estimate the real quality of used method.
RäLT
- >C- - - -----------
»55555555SSSSSH
xxxyyvyyvvvi
yvxxxxv lKAXXJÖÖC« w vi w KXA/wwWW^
HQqckVOQOÖOOOv
_
n = 0.57
f| = 0.73 n = 0.93 n = 1.41
Fig. 4. Examples of blurred images
f| = 1.91
Estimating the quality of an image resizing and rotation algorithms. Execution of any transformation of an image with purposes of resizing or rotating causes a distortion of the source information, the degree of which depends on the used transformation method. Judging by the available information on the Internet, the quality of transformed image is usually estimated visually, which is not acceptable because of the ambiguity visual perception of different people. In the literature also are described methods for investigating the quality of scaling algorithms based on aligning the size of the transformed image with the original one so that it can be compared by pointwise methods (see, for example, [16]). But obvious that in this case the interpolation error occures twice which hampers to estimate the real quality of used method.
To solve this problem it is important to apply methods that work effectively in the case of images with different sizes and orientations, without any additional transformations. The above proposed approach allows evaluating the quality of algorithms and software tools of an image resizing and rotating without involving additional transformations.
W2 1.0
0.8
0.6
0.4
0.2
1
< / VA., J <
< Y g 4 •o C
/' // i
r h .........o -1 - ?
---A- 3 ----*■■ 4
0.5
1.0
1.5
2.0 ß
Fig. 5. Dependence of similarity measure W between a scaled (with coefficient ¡3) image and the original one on ¡3 using four popular interpolation algorithms
Here we give the preliminary results to show the usefulness of using the proposed technique. Figure 5 shows the dependence of similarity measure W2 for a scaled (with coefficient ¡) image and the original one using four popular interpolation algorithms, namely
1 - Interpolation by S-splines;
2 - Bilinear interpolation;
3 - Bicubic interpolation;
4 - Interpolation by the Lanczos method.
The obtained results indicate the possibility of choosing the most suitable scaling method for an image of in-
terest at 3 < 1 or 3 > 1. It is interesting to note that the preferred method of interpolation depends on the reduction or increase in the size of the image.
Note that a similar investigation technique has been developed for rotation algorithms [8]. In the same paper, the concept of "backlash" for image transformation algorithms is introduced.
Unfortunately, a misprint has crept into the text of the paper [8] for dominant direction of an image, but all the rest of results in the paper are correct. The true formula is shown in (5) of the present paper.
Intellectual analysis of a video stream. Using the similarity measure of images proposed above, a technique for analyzing the video streams has been developed with the aim of detecting in them hidden regularities of changing of the story shown in the video. The analysis procedure includes the following steps:
Step 1. Segmentation of the video sequence by finding the jump points of the similarity measure for images of neighboring frames. Memorization of the Weibull distribution parameters (n, X) for each segment of the processed frame;
Step 2. Search for the regularities in the scattering pattern of parameters (n, X) in each segment by regression analysis methods.
In [17], the proposed procedure is described in detail, and in figure 6 there are shown two examples of the scattering of parameters of segments that have different patterns of change in their content. It is also shown a regression line calculated from these data. We see that in the first example there is a significant spread of parameter values, which can be interpreted as evidence of a chaotic change in the content of frames in the segment. In the second example, we observe a certain tendency for a fairly smooth change in the content. These examples show the possibility of a meaningful, albeit very approximate intellectual analysis of the video sequence by formal methods.
Note that the above analysis procedure also allows the speeding up significantly the process of indexing the video sequence, because instead of re-analyzing the information on the entire frame, it is sufficient to use the estimated values of two parameters (n, X) which already were found at the segmentation stage.
Searching out an object by UAVimaging using a template. Using the above procedures for estimating the similarity and dominant orientation of images, it is possible to solve even more complex problems. Consider, for example, a problem of searching out an object by video filming
of unmanned aerial vehicle (UAV) using the preliminarily given template. The complexity of this problem is due to the presence of numerous factors affecting the quality and content of images on video frames. The main factors that constantly act on the process of video filming are the evolutions of UAV, turbulence of the atmosphere, pres-
27
26
25
24
23
22
21
20
19
s°
« 0 0 0 o
«8 o a IP o 0
% o o 9 c 0
c < <0D?P o °c %
o oc <9o o
o o r
ence of winds, vibrations of UAV board, as well as external intentional and unintentional interference and distortions of the real scene image. Therefore, it is necessary to apply algorithms that can operate with the images of different sizes and orientations and are sufficiently stable with respect to distortions caused by mentioned factors.
16.5 16.0 15.5 15.0 14.5 14.0 13.5
13.0 12.5
o
o o____
0 •"o
o____ o Oo
0 6 1 ____
o <îp o/ Of ' o
o c 0
0.58 0.62 0.66 0.70 0.74 0.78 0.82 0.512 0.516 0.520 0.524 0.528 0.532 0.536
Fig. 6. Scattering diagrams and corresponding regression lines. Dependence is insignificant (left); dependence is significant (right)
In the literature suggested a number of approaches and methods related to the problem of search, detection, and tracking of a road and / or objects on it [18]. The use of a structure similarity measure for tracking has proposed in [19] and shown that the tracking procedure is robust to illumination and contrast changes. .
In the present paper, following to our pursuit to show the availability of gradient-based technique we propose consistently using described above estimates of similarity and dominant directions of images in the current and previous frames when making a decision on adjusting the movement of UAV. Investigations using modelling with UAV application show that such a procedure allows solving the problem of finding an object with an acceptable quality [11].
Consider a fragment of mentioned investigation. Suppose that on a filmed video by UAV or Google map high-
lighted an object or region of interest (ROI), as shown in fig.7a. Imagine a situation where, for solving new problems, it is necessary to detect the same object from the video shooting performed by UAV, using only the previously shot image of the area of interest (fig. 7b) as a sample (assuming that the information from the navigation equipment is not available). The object is searched by comparing this sample with the image obtained by scanning the image of the current frame using a sliding window. Since the dimensions and orientation of the compared images do not necessarily coincide, we apply the measure of similarity of W2 proposed above and select the image area of the frame with the maximum value of this measure. In fig. 7c an example of a found section is shown (the sizes of shown images are changed for more convenient illustration).
Fig. 7. Detection of an object using the template by UAV imagery: Choice of ROI (a); Template for ROI (b); Found the similar region to template (c)
It is necessary to pay attention to the difference of the sought and found objects, since it is impossible to provide their identity due to the uncertainty of the UAV traffic parameters.
Conclusions
The paper presents the main mathematical models based on using of the gradients distribution which characterize the structural properties of an image. Some prob-
lems of image processing applications using proposed mathematical models are considered. In particular, we propose methods and procedures for image blur determination by Weibull distribution shape parameter estimate using the sample of gradients magnitudes, a method for evaluating the quality of algorithms and means of image scaling and rotation, an approach to intellectual analysis of the video streams, the possibility of searching for an object by an unmanned aerial vehicle video filming using a given template and the proposed measure of image similarity, as well as the algorithm for estimating the dominant direction of the images of video sequence. The examined examples of different applications show the effectiveness of proposed approach, procedures and algorithms based on using of an image gradients field.
References
[1] Wang Z, Bovik AC. A universal image quality index. IEEE Signal Process Lett 2002; 9(3): 81-84.
[2] Wang Z, Bovik AC, Lu L. Why is image quality assessment so difficult? Proc IEEE Int Conf Acoustics, Speech, and Signal Processing 2002; 4: 3313-3316.
[3] Wang Z, Bovik AC. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 2004; 13(4): 600-612.
[4] Ponomarenko N, Carli C, Lukin V, Egiazarian K, Astola J, Battisti F. Color image database for evaluation of image quality metrics. Proc International Workshop on Multimedia Signal Processing 2008: 403-408.
[5] Xue W, Zhang L, Mou X, Bovik AC. Gradient magnitude similarity deviation: A highly efficient perceptual image quality index. IEEE Trans on Image Processing 2014; 23(2): 684-695. DOI: 10.1109/TIP.2013.2293423.
[6] Gonzales RC, Woods RE. Digital image processing. 2nd Ed. Prentice Hall, 2002.
[7] Cramér H. Mathematical methods of statistics. Princeton: Princeton University Press; 1991.
[8] Asatryan DG, Egiazarian KO, Kurkchiyan VV. Orientation estimation with applications to image analysis and registration. Information Theories and Applications 2010; 17(4): 303-311.
[9] Asatryan DG, Egiazarian KO. Quality assessment measure based on image structural properties. Proc International Workshop on Local and Non-Local Approximation in Image Processing 2009: 70-73.
[10] Asatryan DG, Kurkchiyan VV, Kharatyan LR. Method for texture analysis and classification [In Russian]. Computer Optics 2014; 38(3): 574-579.
[11] Asatryan DG, Hovsepyan SM, Kurkchiyan VV. Road tracking from UAV imagery using gradient information. Information Technologies & Knowledge 2016; 10(2); 191199.
[12] Koik, BT, Haidi I. A literature survey on blur detection algorithms for digital imaging. AIMS '13 Proc 2013 1st International Conference on Artificial Intelligence, Modelling and Simulation 2013: 272-277.
[13] Garg V, Manchanda M. A survey on image blurring. International Journal of Engineering Applied and Management Sciences Paradigms 2014; 15(1): 2320-6608.
[14] Singh D, Sahu RK. A survey on various image deblurring techniques. International Journal of Advanced Research in Computer and Communication Engineering 2013; 2(12): 4736-4739.
[15] Asatryan DG. Image blur estimation using gradient field analysis [In Russian]. Computer Optics 2017; 41(6): 957962. DOI: 10.18287/2412-6179-2017-41-6-957-962.
[16] Pankaj SP, Paresh V. Virparia image quality comparison using PSNR and UIQI for image interpolation algorithms. International Journal of Innovative Research in Computer and Communication Engineering 2016; 4(12): 2167921687. DOI: 10.15680/IJIRCCE.2016. 0412056.
[17] Asatryan DG, Zakaryan MK. Novel approach to content-based video indexing and retrieval by using a measure of structural similarity of frames. Information Content and Processing 2015; 2(1): 71-81.
[18] Athanesious JJ, Suresh, P. Systematic survey on object tracking methods in video. International Journal of Advanced Research in Computer Engineering and Technology 2012; 1(8): 242-247.
[19] Loza A, Mihaylova L, Bull D, Canagarajah N. Structural similarity-based object tracking in multimodality surveillance videos. Machine Vision and Applications 2009; 20(2): 71-83.
Author's information
David G. Asatryan (b. 1940). Doctor of sciences (Engineering), professor, he works as a Head of Center for Critical Technologies of the Russian-Armenian (Slavonic) University and as a leading scientific worker at the Institute for Problems of Informatics and Automation of National Academy of Sciences of Armenia. Research interest: signal and image processing. E-mail: dasat@ipia.sci.am .
Received June 13, 2018. The final version - December 19, 2018.
