CC BY
84
UDC 62-50
V. Y. STEPANOV, E. A. HVITKO
ALGORITHM FOR ALERTING THE UNMANNED AERIAL VEHICLE OPERATOR BASED ON THE IMAGE POTENTIAL OBSTACLE BORDERS DETECTION ON THE FLIGHT TRAJECTORY USING
THE OPEN CV LIBRARY
Belarusian National Technical University
The article describes several modifications of the obstacle detection algorithm on the flight trajectory of an unmanned aerial vehicle (UAV). First, we give an illustration of the problems on the example of the relief image on topographic maps and the essence of the relief image by horizontal contours. Then the conclusion is made about necessity of development of algorithm and software, which can help the operator of the UAV in deciding on necessary trajectory changes of UAV, since, for example, guided solely by the method image of the terrain or another similar method in the planning of the UAV trajectory as preliminary preparation for the flight, however, such methods are fairly static and are not suitable in such situations as, for example, detection of unexpected obstacles. Further, the article describes the process of developing several modifications of the algorithm that solves the above problem and generates a notification of the UAV operator. It is also described depending on the flight speed of the UAV and natural conditions; it is possible to choose the most appropriate modification of the algorithm, which has different sensitivity to potential obstacles. The conclusion shows the settings screen and results of the software work and the principle of visual notification of the UAV operator with the proposal direction of the flight around detected obstacles.
Keywords: unmanned aerial vehicle, mathematical modeling, software, algorithm for finding obstacles on the flight trajectory, notification of the UAV operator.
Introduction
In today's world, received fairly widespread the unmanned aerial vehicles (UAV), which have a variety of purposes and often require using in conditions far from ideal. Thus, the problem of detecting obstacles on the flight trajectory, especially those that have arisen unexpectedly, since the design of UAV if bump with almost any kind of obstacle do not allow it to remain fully operational.
Therefore, it is necessary to design and implement an algorithm for detection potential obstacles to the flight of the UAV, to develop software that uses streaming image acquisition from the camera.
The main part
To illustrate the problem, consider, for example, the image of relief on topographic maps, which gives a complete and sufficiently detailed idea of the irregularities of the earth's surface, their shape and relative position, heights of the
terrain points, the prevailing steepness and length of the slopes. Let's consider the essence of the relief image by horizontal contours (figure 1).
Horizontal contour is a closed line representing a contour of irregularities on the map, all points of which are located on the ground at the same height above sea level. Contours can be represented as lines obtained as a result of the cross-section of the terrain by level surfaces, that is, surfaces parallel to the water level in the oceans [1].
A
Figure 1. Image of relief by horizontal contours on topographic maps
However, such images are quite static and are not suitable in such situations as, for example, the detection of an unexpectedly appeared obstacle, and as mentioned earlier, in the case of an unmanned aerial vehicle, a bump with even small objects will almost certainly have catastrophic consequences for UAV. For this reason in this article the most attention will be paid to the detection of dynamically appearing obstacles due to potential obstacle borders detection on the image and reaction of system after this.
Hence, the algorithm, step by step:
1) sensors reading (modeling);
2) get image from camera (in article using Open CV);
3) assignment of safe (green), potentially dangerous (yellow) and dangerous (red) zones on the image for the purpose of detection of potentially dangerous objects and filtration of borders of the objects which are not dangerous (being aside from a trajectory of flight - in a green zone);
4) convert the obtained image into a grayscale (shades of gray);
5) detection potential obstacles on the image;
6) alerting the UAV operator (displaying of a message and playback of an audio signal) about the potential obstacle detection (subject to fulfilment of previous step), indicating the most secure direction of the overflight of the obstacle (left or right and up or down) based on the detection of the area obstacle and its location relative to the UAV trajectory [2-5].
For greater clarity, we present the most significant steps of the described algorithm in the form of flowchart. Hence, step 2 of the algorithm (get information from the camera of an unmanned aerial vehicle) is shown on figure 2.
As the device receiving the video stream, simulating the UAV camera, were used Logitech camera C110, with the following main characteristics:
- number of megapixels: 0.3;
- resolution (photo): 640x480;
- resolution (video): 640x480;
- interpolated resolution (video): 1024x768;
- interpolated resolution in megapixels (photo): 1.3 Mpix;
- maximum frequency: 30 Hz.
Then, to research the different variants of sensitivity to the borders of the image objects, consider two variants of borders detection in images (steps from the 3rd to the 5th of algorithm), prog-
Figure 2. Receiving video stream from Web-camera with image conversion using Open CV
rammed in the unsafe context of the C# programming language, to accelerate the image processing.
1) method of the image binarization by threshold (figure 3, where illustrate the preparation procedures for analyzing the original image and figure 6, where illustrate the obstacle detection procedure) as the most sensitive to detection the obstacles of those considered, therefore it can be used only in conditions close to ideal [6-8].
Note: since memory is a linear structure, the entire image will be represented in computer memory not like a matrix, and not like we imagine the image in a life (each pixel consists of three colors: red, green and blue), but the entire image will be represented as a linear structure, where each pixel is represented as three values (without transparency - an additional alpha channel) - figure 4.
It should also be noted that the step 4 is obtained by applying the known color ratio coefficients of the RGB model of the main image) and
Figure 3. - Image preparation for analysis
Figure 4. Representation of images: a) logical; b) physical
at the physical level is part of the step 5 of the algorithm.
The 3rd step is the division of the image into logical sectors-zones (the size of the sectors selected empirically and, for example, the red zone is equal to the fourth part of the image, because this size is sufficient for the flight of the UAV between the objects placed in the yellow zone on the left and right or up and down, if the analysis is conducted from a distance of approximately 5 x b, where: b - the width of the UAV, with the characteristics of the camera similar to those presented earlier) [9-15].
This algorithm step most fully reflects the essence of the development of our algorithm as a whole, which consists in separating the processing logic of potentially dangerous zones based on calculating the areas of obstacles in these zones with automated camera calibration and taking into account the flight speed of the UAV, without the need for additional measurement equipment.
The process of camera calibration and zones visual representation is shown on figure 5.
A breakdown into zones flowchart is presented below (figure 6):
Figure 5. The process of camera calibration and zones visual representation
Figure 6. A breakdown into zones flowchart
Figure 7. The key features of Kirsch method in comparison with the method of binarization by threshold
2) the Kirsch method. To prevent of duplication elements in flowcharts, will be presented only the key features, in comparison with the method of binarization by threshold value [15]. The Kirsch method has an average level of sensitivity to obstacles. Algorithm fragments are shown on figures 7.
The last step of the algorithm is to alert (using developed software) the operator of the UAV which controls the flight about potentially dangerous situation and possible bump (if an obstacle was detected) with the displaying of an alert message and playback of an audio signal to attract the attention of the operator.
The operator is also informing about the most secure direction of the overflight of the obstacle, according to the area of the obstacle in the yellow and red zones (right (left) or up (down)).
The developed software has two tabs: «Settings» and «Simulation», three control buttons «Start» - to run the simulation based on the settings, «Stop» - to stop the modeling process and «Save log» - to generate a report involves the setting values and other working events of software to a text file [16-18].
The «Settings» tab is shown on figure 8: According the figure above the «Settings» tab involves the next settings:
Figure 8. «Settings» tab
- the flight speed of an unmanned aerial vehicle - this parameter determines the power of noise filtering during searching for obstacles;
- flight time can be limited if necessary, but by default - simulation continues until stopped;
- next three parameters - set to zero - are not used in this version of the software;
- threshold of insensitivity - the coefficient that determines what the total area occupied by a potential obstacle (obstacles) is considered noise;
- threshold of sensitivity to the increase of speed - the greater the speed, the more algorithm sensitivity to obstacles, because with speed increasing the maneuvering should do as soon as possible;
- data sampling rate - algorithm trigger frequency in seconds;
- method for detecting borders on the image -a drop-down list that allows you to select the desired method with different sensitivity, because the choice of method should vary depending on the experiment conditions. Possible options: Bi-narization by threshold, Kirsch method, Laplace method, Roberts method, Sobel method, Wallace method and auto selection.
Tab «Simulation» is divided into four logical zones:
- top left - display streaming video from the camera (original image, before the analysis);
- top right - display the result of the analysis with a visual representation of zones described in the third step of the algorithm;
- alerting zone (for display alert message during detection of a potential obstacle);
- part of the software log (for display exceptions of the hardware component working or failures in the software itself).
Alert situation where was obstacle detected is shown on figure 9:
Flight in a normal mode is shown on figure 10:
Figure 9 shows a result of execution the algorithm, where the borders of the potential obstacles detect by using of Laplace method. All options are shown in figure 8 (drop-down list with choosing of method for detecting borders on the image) As shown on the figure, if the UAV was directed to an object that potentially threatens it with a bump, the operator receives a visual alert (together with an audio signal) with indicating the direction in which it is proposal to direct the UAV.
Figure 10 shows the situation of simulation the flight of UAV in non-ideal conditions (in conditions of high clouds of the autumn sky) in order to determine the number of false alerts and establish the algorithm's insensitivity factor.
It should be noted the most important stage of the software execution is using of an obstacle de-
^ MairWîndpw ^
Настройка Моделирование 1
1
Iе
1 !!! ALERT !1! (it is recomemded to qo around an obstacle on the RIGHT and DOWN) |
Figure 9. Alert situation where was obstacle detected
Figure 10. Flight in a normal mode
tection algorithm based on the programmed processing logic of the zones potentially dangerous for flight (for convenience, they are highlighted in color and are called respectively), and the operator notification subsystem is easily replaceable and is intended to demonstrate the algorithm execution. In the future, due to the low cohesion and modularity of the developed solutions, the operator alert subsystem can be easily replaced with
the sending control impacts subsystem to the UAV.
Conclusion
In this paper, the image of relief on topographic maps was raised. Than the conclusion was made about the need to detect and respond to objects around UAV, the presence of which, nothing was known in advance.
The algorithm for detection of potential obstacles to the flight of UAV was designed and implemented. Next, software that get video stream from the camera using the Open CV library, that used above mentioned algorithm in several modifications, and provides visual and audio alert for the user (UAV operator) about need to reaction to the potential dangerous situation. Also, software can propose of a possible direction of flight
around the obstacle, where, according to the software, less chance of a bumping with a potential obstacle.
As a further development of this theme, it can be proposed to take into account the distance to the potential obstacle according the flight speed of the UAV, to be able to estimate the time range available to the operator for decision-making about control of the UAV.
REFERENCES
1. lektsii.org [Electronic resource]. - Access mode: https://lektsii.org/4-12821.html. - Date of access: 10.09.2018.
2. Recog.ru - Распознавание образов для программистов [Electronic resource]. - Access mode: http://blog.vidikon. com/?p=228. - Date of access: 15.10.2018.
3. RoboCraft [Electronic resource]. - Access mode: http://robocraft.ru/blog/computervision/264.html. - Date of access: 17.10.2018.
4. Habr [Electronic resource]. - Access mode: https://habrahabr.ru/post/208090/. - Date of access: 18.10.2018
5. Методы компьютерной обработки изображений / под ред. В.А. Сойфера. - Москва: Физматлит, 2003. - 784 с.
6. OpenCV - библиотека компьютерного зрения с открытым исходным кодом [Electronic resource]. - Access mode: http://locv.ru. - Date of access: 18.10.2018
7. Старовойтов В. В., Голуб Ю. И. Цифровые изображения: от получения до обработки - Минск: ОИПИ НАН Беларуси, 2014. - 202 с.
8. Приоров, А. Л. Цифровая обработка изображений: учеб. пособие / А. Л. Приоров, И. В. Апальков, В. В. Хря-щев. - Ярославль: ЯрГУ, 2007. - 235 с.
9. Фисенко, В. Т. Компьютерная обработка и распознавание изображений: учеб. пособие / В.Т. Фисенко, Т.Ю. Фисенко. - Санкт-Петербург: СПбГУ ИТМО, 2008. - 192 с.
10. Шапиро, Л. Компьютерное зрение / Л. Шапиро, Дж. Стокман: пер. с англ. - Москва: БИНОМ. 2006. - 752 с.
11. Гонсалес Р., Вудс Р. Цифровая обработка изображений. Издание 3-е, исправленное и дополненное. - Москва: Техносфера, 2012. - 1104 с.
12. Методы компьютерной обработки изображений / под ред. В.А. Сойфера. - М.: Физматлит, 2003. - 784 с.
13. Цифровая обработка изображений в информационных системах: учеб. пособие // И. С. Грузман [и др.]. -Новосибирск : Изд-во НГТУ, 2002. - 352 c.
14. Прэтт, У Цифровая обработка изображений: пер. с англ. / У Прэтт. - М.: Мир, 1982. - Кн. 2. - 480 с.
15. Стариченко, Б. Е. Теоретические основы информатики : учеб. пособие для вузов. - 2-е изд. - М.: Горячая линия - Телеком, 2003.
16. Шилдт Г. C# 4.0: полное руководство.: Пер. с англ. - М.: ООО «И.Д. Вильямс», 2011. - 1056 с.: ил. - Парал. тит. англ.
17. Троелсен Э. Язык программирования C# 5.0 и платформа .NET 4.5, 6-е изд.: Пер. с англ. - М.: ООО «И. Д. Вильямс»., 2013. - 1312 с.: ил. - Парал. тит. англ.
18. Натан А. WPF 4. Подробное руководство. - Пер. с англ. - СПБ.: Символ-Плюс, 2011. - 880 с., ил.
Поступила После доработки Принята к печати
19.07.2018 15.10.2018 30.11.2018
Степанов В. Ю., Хвитько Е. А.
АЛГОРИТМ ОПОВЕЩЕНИЯ ОПЕРАТОРА БЕСПИЛОТНОГО ЛЕТАТЕЛЬНОГО АППАРАТА НА ОСНОВЕ ВЫДЕЛЕНИЯ НА ИЗОБРАЖЕНИИ ГРАНИЦ ПОТЕНЦИАЛЬНЫХ ПРЕПЯТСТВИЙ ДЛЯ ПОЛЁТА С ИСПОЛЬЗОВАНИЕМ БИБЛИОТЕКИ OPEN CV
Белорусский национальный технический университет
В статье даётся описание нескольких вариантов алгоритма обнаружения препятствий на траектории движения беспилотного летательного аппарата (БЛА). Вначале даётся иллюстрация проблематики на примере изображения рельефа на топографических картах и сущность изображения рельефа горизонталями. Затем делается вывод о необходимости разработки алгоритма и программного обеспечения, которые могут помочь оператору БЛА в принятии решения о необходимости изменения траектории БЛА, т.к., например, руководствуясь исключительно
методом изображения рельефа горизонталями или другим подобным методом при планировании траектории БЛА можно в качестве предварительной подготовки к полёту, однако такие методы являются достаточно статичными и не подходят в таких ситуациях, как, например, распознавание неожиданно появившегося препятствия. Далее в статье описан процесс разработки нескольких модификаций алгоритма, решающего поставленную выше задачу и производящего оповещение оператора БЛА. Также описано, что в зависимости от скорости полёта БЛА и условий его эксплуатации, есть возможность выбора наиболее подходящей модификации алгоритма, имеющую разную чувствительность к потенциальным препятствиям. В заключение приводятся настройки программы оповещения и результаты работы, где показан принцип визуального оповещения оператора БЛА с указанием рекомендуемого направления облёта беспилотным летательным аппаратом зафиксированного препятствия.
Ключевые слова: беспилотный летательный аппарат, математическое моделирование, программное обеспечение, алгоритм обнаружения препятствий на траектории полёта, оповещение оператора БЛА.
Stepanov V. Y. Received the Software Engineer and M.S.C. degree in field of «Information systems and technologies» from the Belorussian National Technical University in 2013 and 2014 respectively. He is currently working toward in PhD degree in the field of «Digital signal processing and control» including, parametric optimization and programming. His research interests include IT, programming and robust control.
E-mail: vladimir.stepanov@bntu.by
Hvitko E. A. Master Student of the department «Information Systems and Technologies» of the International Institute of Distance Education BNTU. Lecturer of the Department «Software Computing and Automated Systems» of the faculty of «Information Technologies and Robotics». His current research interests include control of unmanned aerial objects.
E-mail: evgeni.hvitko@bntu.by
