CC BY
17
МАТЕМАТИЧНЕ МОДЕЛЮВАННЯ Ф1ЗИЧНИХI ТЕХНОЛОГ1ЧНИХ ПРОЦЕС1ВIТЕХН1ЧНИХ СИСТЕМ
UDC 002.53:004.89; 61:007
O.Yu. MIESHKOV, О.О. NOVIKOV
Kherson National Technical University
MATHEMATICAL MODEL OF HUMAN VOICE FOR THE TASK OF PERSONAL IDENTIFICATION AND HUMAN CONDITION ANALYSIS
In this work, the human voice mathematical model for the task of personal identification and human condition analysis is developed. The specified model is based on human voice signal characteristics - voice fundamental frequency and the structure of amplitude distribution in time domain. These characteristics are considered to be influenced by human anthropometry, gender and age parameters. Personal identification is performed in two-dimensional space based on the specified characteristics. For the task of human condition analysis, two types of human voice standards are developed using multiple regression method and clustering of speakers database by the KNN-graph technology. Experimental research has shown satisfied results of the work of developed algorithms. The developed system is implemented as script-files for open source applied mathematics package SciLab 5.5.2.
Keywords: human voice, personal identification, human condition analysis, multiple regression method, human voice standard, database clustering, authors' algorithms.
о.ю. мешков, о.о. новжов
Херсонський нацюнальний техшчний ушверситет
МАТЕМАТИЧНА МОДЕЛЬ ЛЮДСЬКОГО ГОЛОСУ ДЛЯ ЗАДАЧ1 1ДЕНТИФ1КАЦП ЛЮДИНИ
ТА АНАЛ1ЗУ II Ф1ЗИЧНОГО СТАНУ
У po6omi розроблено математичну модель голосу людини для 3ada4i iдентифiкацii та анал1зу стану людини. Дана модель базуеться на характеристиках голосового сигналу людини - основнт частотi голосу та структурi розподту амплтуди у часовому просторi. Вважаеться, що на ц характеристики впливають антропометрiя людини, статевi та ei^ei осо6ливостi. Iдентифiкацiя осо6истостi виконуеться у двомiрному просторi на базис вказаних характеристик. Для задачi аналiзу стану людини розроблено два типи еталотв голосового сигналу людини з використанням методу множинноi регресп та кластеризаци бази даних дикторiв за технологiею KNN-графу. Експериментальне до^дження показало задовiльнi результати роботи розроблених алгоритмiв. Розроблена система реалiзована у формi скрипт-файлiв для вiдкритого пакету прикладних математичних программ SciLab 5.5.2.
Ключовi слова: голос людини, iдентифiкацiя осо6истостi, аналiз стану людини, метод множинно '1' регресп, еталон голосу людини, кластериза^я бази даних, авторськ алгоритми.
А.Ю. МЕШКОВ, А.А. НОВИКОВ
Херсонский национальный технический университет
МАТЕМАТИЧЕСКАЯ МОДЕЛЬ ГОЛОСА ЧЕЛОВЕКА ДЛЯ ЗАДАЧИ ИДЕНТИФИКАЦИИ ЧЕЛОВЕКА И АНАЛИЗА ЕГО ФИЗИЧЕСКОГО СОСТОЯНИЯ
В работе разработана математическая модель голоса человека для задачи идентификации и анализа состояния человека. Данная модель базируется на характеристиках голоса человека -основной частоте голоса и структуре распределения амплитуды во временном пространстве. Считается, что на эти характеристики влияют антропометрия человека, половые и возрастные особенности. Идентификация личности выполняется в двумерном пространстве на базисе указанных характеристик. Для задачи анализа состояния человека разработаны два типа эталонов голосового сигнала человека с использованием метода множественной регрессии и кластеризации базы данных дикторов по технологии KNN-графа. Экспериментальное исследование показало удовлетворительные результаты работы разработанных алгоритмов. Разработанная система реализована в форме скрипт-файлов для открытого пакета прикладных математических программ SciLab 5.5.2.
Ключевые слова: голос человека, идентификация личности, анализ состояния человека, метод множественной регрессии, эталон голосу человека, кластеризация базы данных, авторские алгоритмы.
MATEMATHHHE MO^EHWBAHHH OBHHHHXI TEXHOnOrmHHX nPOUECIB I texhivhhx chctem
Problem statement
One of the main tasks of modern diagnostics is to provide technologies that are able to perform human condition analysis with a minimal influence on human organism. Such technologies are usually called non-invasive. Most of these technologies are based on different human organism characteristics. Nevertheless, none of them gives a comprehensive assessment of human condition and their accuracy is not always sufficient.
Along with these technologies, there are a lot of personal identification methods that also take into account different criteria concerning human physiology. These methods analyze some certain characteristics or signals that are generated by human organism. According to the results of this analysis, they make a decision that is needed for the system. Such technologies are used in automatic recognition systems, access systems etc.
Very often in these systems, human voice signal is used as a main decision criterion. It can be analyzed with a great variety of characteristics that are consequently used for the task of analysis. At the same time it is generally known that any changes in human organism to some extent is displayed on human voice and this signal changes in this case. In our opinion according to these changes, it is possible to determine the nature of changes that take place in human condition in general.
It is a known fact that human voice signal is individual for every single person. In spite of this fact, we are able to build an automatic system for human identification on the first stage and human condition analysis on the second. Development of such a system is very relevant.
Reference analysis
Human voice is a type of complex acoustic signal that is generated by the human vocal apparatus. The main organs that take place in voice production are lungs, larynx with vocal folds, articulation apparatus (mouth, teeth etc.). The whole voice formation process is coordinated by nervous system [1].
Sounds of human voice can be divided into two large groups - vowels and consonants. For the task of personal identification vowels are usually used. The reason for this is that the formation of vowels involve much more organs and systems of human organism. In this case, these sounds are more sensitive to changes in human conditions. According to this fact a set of technologies for diagnostics of certain human organism disorders are built today [2], [3], [4].
Each vowel sound can be characterized by certain set of characteristics. Amplitude, frequency and spectral features of human vocal apparatus define these characteristics. According to anthropometric, physiologic and other features of human organism human voice signal is formed individually for every certain person. On this base, a lot of technologies for personal identification or verification are built today.
There is a set of standard methods that are used for the task of identification by human voice analysis. Frist of all human voice signal must be submitted in the form of certain characteristics. For this purpose, mel-frequency cepstral coefficients, hidden Markov models, linear prediction methods are used. For speakers classification methods of cluster analysis are used [5], [6].
Vowel sounds of human voice can be divided into a set of consecutive segments. When considering these segments in time domain it can be noticed that they can be described as oscillations with the same structure and time duration. Such quasi-periodic oscillations are called frames [1].
Each frame of human voice vowel sound has its own characteristics such as time duration, frequency, amplitude time distribution, spectrum etc. Some of these characteristics can be defined manually in time domain. But to define a set of certain voice characteristics we have to transfer this signal into frequency domain using Fourier transforms.
One of the main human voice signal characteristics in time domain is fundamental frequency or F0. It can be defined as a frequency of vocal folds oscillations that usually is equal to the frequency of each single frame. There are a lot of researches and investigations that determine the main factors that influence frequency of their oscillations. Usually these investigations concern defining correlations between these factors and human anthropometry [7], [8], [9], [10], [11].
In most cases, researches do not pay much attention to voice signal amplitude distribution in time domain in these works. They prefer to analyze signals in frequency domain, pointing to greater efficiency and simplicity of these types of analysis [5]. At the same time, some researches analyze human voice signal in time domain [6]. In our opinion, analysis of human voice signal in time domain can give a comprehensive assessment of human voice state in complex with analysis in frequency domain. Therefore, time domain characteristics must be taken into account.
The purpose of research
The purpose of this research is to combine different methods of human voice signal analysis into a system that will be able to provide human condition analysis by analysis of its voice. In spite of the fact that the developed system will deal with a great amount of speakers who have different voices it is desirable to organize the work of this system in two modes. First, this system must identify the speaker among others by the entered voice signal. For this task, some special methods of voice signal analysis must be used.
MATEMATHHHE MO^EHWBAHHH OBHHHHXI TEXHOnOrmHHX nPOUECIB I texhivhhx chctem
Second, after identification the developed system on the base of previous recordings of speakers voice or certain statistical models must examine the condition of entered voice signal and, consequently, human organism condition.
Description of research
According to the reference analysis, human voice analysis must be provided on the basis of a set of certain characteristics extracted from this signal. In this research, we referred to these characteristics the follows ones:
- fundamental frequency and its distribution in time domain;
- amplitude distribution in time domain.
Both these parameters were analyzed on the basis of human voice recordings (vowel /A/ in Ukrainian pronunciation). Recordings were performed in *wav format (Sample Rate - 22050 Hz, Bit Depth - 16 bits, Channels - Mono) both for male and female speakers.
First, the developed system have to preprocess the entire acoustic material. Using authors' algorithm the developed system extract the vocalized segment from the speech flow. This is performed by cutting out the voice attack at start and voice damping at finish. On the next stage, the developed system defines the primary value of fundamental frequency. For this purpose voice signal have been exposed to cepstral transform:
cepstr(t) = i//t(log|//t(s(t))|), (1)
where s(t) - input signal in time domain
cepstr(t) - cepstr of the input signal in time domain;
fft, if ft - fast Fourier transform and inverse fast Fourier transform respectively.
The local maximum of the cepstr of the signal in the range of human voice corresponds to the primary value of human voice frame duration. According to this value, the developed system evaluates fundamental frequency primary value.
After that extracted vocalized segment is divided into separate frames. For this task, according to the fundamental frequency primary value the average frame duration is defined. Then zero crossing points of signal in the areas corresponding to the frame duration are defined. Duration of each frame may differ from the average value. As a result, a massive of separate frames is formed for every single recording of each speaker.
Each frame has a common structure, but the same time every single frame has its own duration. That is why some local peaks of amplitude distribution in time domain can have different time localization. For the task of amplitude distribution in time domain analysis, each separate frame must be scaled to the certainly defined duration. This allows aligning corresponded peaks.
The massive of scaled frames in combination with the frequencies of each frame is saved in the special database into a general directory and into a personal directory of a certain speaker. This has been implemented to fill the database in two ways. First, the general directory of the database have to include recordings of each speaker for the task of personal identification. In addition, the general directory is used to form a human voice standard that will take into account statistics of human voice signal of different speakers. Second, personal directory of the database is used to form the individual human voice standard that takes into account only recordings of voice of the certain speaker. In addition, the persons' data includes age, weight, height and gender parameters, which are used in the identification and voice analysis procedures.
The developed system have to analyze human condition by voice signal. However, first, it must identify the person some way. Of course, to begin working with the developed system user can enter his name/surname or some special code to work with his own directory. But for the task of automation of the system it has been built on the two-step technology.
The first step implies personal identification of the user. For the task of identification the general directory of the database is used. Person performs voice signal recording with the same parameters as the database signals. After that the incoming signal is processed by the same algorithms as the database signals. As a result, the system receives the massive of scaled frames of the incoming signal and defines fundamental frequency value of each frame.
These two parameters must be compared with database ones. Fundamental frequency value can be compared with the same parameter for each speaker in the database. Fundamental frequency deviation can be defined as follows:
afub =ft- fb, (2)
where Fit Fd - fundamental frequency value of input and database signal respectively.
В1СНИКХНТУ№ 1(56), 2016р. МАТЕМАТИЧНЕМОДЕЛЮВАННЯФ1ЗИЧНИХI _ТЕХНОЛОГ1ЧНИХ ПРОЦЕС1ВIТЕХН1ЧНИХ СИСТЕМ
However, amplitude distribution in time domain includes a very big massive of data. This fact makes the comparison of signals rather difficult.
In references [12] the normalized signal structure standard deviation coefficient is defined as:
(3)
Ki,b ~ ~
^Mj-Y^At
where Yifj, Ybj - amplitudes of j-th count of input and database signal respectively; T - frame period (duration); At - time intervals between counts.
After defining fundamental frequency deviation and normalized signal structure standard deviation coefficient two-dimensional space of speakers is built. The basis for this space are values of specified characteristics. Each voice signal is represented as a point with a specified coordinates in this space. Then the developed system calculates the distance from the input signal to each signal available in the database. The distance between signals is defined as Euclidean distance, taken with the weight coefficients:
I--(4)
di,b = J(0F X AF?b +a)KX AK?b,
where (oF,o)K - weight coefficients of fundamental frequency deviation and normalized signal structure
standard deviation coefficient respectively.
The presence of weight coefficients can be explained by the necessity of consideration the importance of the factor. For some speakers fundamental frequency plays more important role than amplitude structure and for some it will be vice versa. Database signal that will have a minimal distance to input signal is considered to belong to this speaker. If the distance exceeds the minimal threshold dmin, the system decides the speaker is out of base.
After the person has passed the identification procedure, implemented in the developed system, it is able to perform the analysis of the input voice signal. For this task in the developed system authors' mathematical model of the chosen human voice characteristics is implemented.
In our opinion, human voice fundamental frequency depends on human anthropometry, gender, age and many other factors. At the same time, some researchers notice that there is no correlation between some of these parameters [13]. For example, emotional conditions, season features, time of day when recording has been performed are also important parameters but on this stage of our research, we are not able to submit correlations between these features and human voice characteristics.
For this reason, we have chosen the set of primary factors that mostly influence human voice signal:
- persons' gender;
- persons' age;
- persons' height;
- persons' weight.
In spite of the fact, that primary factors must be taken into account in combination we propose to use three-factor mathematical model, which is as follows:
F0 = cPX Aaf X Hhr X WWF, (5)
where F0 - fundamental frequency, Hz;
A - persons' age, complete years;
H - persons' height, cm;
W - persons' weight, kg;
cP, aF, hF, wF - frequency factor coefficients, which are determined using the multiple regression method.
Input signal amplitude distribution in time domain cannot be described in whole as a mathematical expression because of the big amount of data shown before. However, we consider that amplitude of each count can be described using the model, similar to fundamental frequency one:
Yt = cY,i X Aavi X Hhvi X Wwn, (6)
MATEMATHHHE MO^EHWBAHHH OBHHHHXI TEXHQnOrmHHX nPOUECIB I TEXHIVHHX chctem
where Vj - amplitude of the i-th count of human voice signal;
a - persons' age, complete years;
H - persons' height, cm;
W - persons' weight, kg;
cF, aF, hP, wF - amplitude factor coefficients, which are determined using the multiple regression method.
Multiple regression method for determining frequency and amplitude factor coefficients is used in spite of the fact, that human voice is a dynamic signal, so it changes in time. Changes in human voice signal may occur both in the single vowel between frames, and between vowels recorded in different time. We consider that these changes primarily are due to the changes in human anthropometry.
To determine values of specified characteristics the developed system possesses two human voice standards. Group human voice standard involves using acoustic material of different speakers available in the general directory of the database. To form this type of standard the developed system primarily select a set of speakers, whose voice signal is the closest to the input one. This procedure is performed the same way as the identification. Two-dimensional space of speakers is based on the same parameters (fundamental frequency value and the structure of amplitude distribution in time domain). The difference is that this time the developed system divides the whole database into clusters. This division is based on the KNN-graph technology, where K-parameter is settled equal to 1.
Algorithm for this task works on the following script. First, the system defines distances between every signal available in the database, including the input one. The threshold value for the distance is settled as an average value of the distance. Then, the system defines the closest neighbor-signal for every signal. After that, it takes the input signal as a starting point and select its neighbor-signal. This neighbor-signal becomes the starting point and the cycle repeats. Cluster is considered to be formed if all the neighbor-signals that are tied to the input signal directly or after several starting point changes are taken from the database. After defining all needed signals the system processes them with the authors' algorithms for frequency and amplitude factor coefficients definition. These coefficients unequivocally define the signal, which is considered to be the group human voice standard for this speaker. The result is saved into a personal directory of a speaker defined during the identification procedure.
Individual human voice standard involves using acoustic material of a certain speaker and takes into account dynamics of a certain human voice. To form this type of standard the developed system uses the results of personal identification on the first step. After identification, the system selects a speakers' personal directory and defines values of frequency and amplitude factor coefficients using selected data. These results are also saved into a personal directory, but this time as an individual standard.
To perform human condition analysis the developed system ask a user to fill the form with his/her anthropometric parameters (age, weight, height) and to choose gender. This is made to form a standard human voice signal based on the pre-defined factor coefficients. The system calculates the standard value of the fundamental frequency and the amplitude of every count of the signal.
Comparison of the input signal and the standard one is based on the deviation criteria for fundamental frequency and the structure of amplitude distribution in time domain. These criteria are defined during the following experimental research for both parameters. Deviations of the respective parameters are defined by formulae (2-3). If one of deviations is considered to be out of the defined range, human voice signal and, as a result, human condition is out of norm. Only in the situation when both parameters have deviations less, then the threshold value, human condition is considered to be normal.
For the task of concrete analysis or diagnostics of some diseases, the described type of analysis is not enough. For the specified task, the developed system have to include such s called 'disease standards'. It means that in the system there must be a special database formed with human voice signals that certainly has a specified deviation in human condition. During the analysis of these signals, the system will be able to define some features that appears in human voice signal with the appearance of a certain deviation in health. With the presence of these features in human voice signal defined in analysis process, the system will be able to make a decision of presence or absence of a specified deviation in human health. Research on this way is performed actually.
In general, the developed system is implemented as script-files for open source applied mathematics package SciLab 5.5.2. This system has been tested on the base speakers set (60 female, 70 male recordings). During the pilot study weight coefficients values (wF = 0,67; (oK = 0,33) and threshold (dmin = 0,2) for the task of identification have been defined. In addition, for these speakers two types of standards have been built.
The developed system have been tested on the test speaker set (30 female, 30 male voice recordings). A number of speakers of the test speakers test have been taken from the base one; some of them have not been introduced to it. Test procedures showed the absolute accuracy in speakers' identification for the short-time period (one week). However, experimental research on long-time period has shown that some basic speakers'
МАТЕМАТИЧНЕ МОДЕЛЮВАННЯ Ф1ЗИЧНИХI ТЕХНОЛОГ1ЧНИХ ПРОЦЕС1ВIТЕХН1ЧНИХ СИСТЕМ
signals ceases to be relevant for the task of personal identification. This fact, in our opinion, proofs that human voice signal is very dynamic, so it can be used for the task of dynamic and real-time human condition analysis. However, the same time for the task of personal identification human voice standards must be updated after a certain period of about one-two weeks. Research works on this subject are performed now.
The developed system can be used as a complex product for the tasks of both personal identification and complex human condition analysis. At the same time, the first stage responsible for the identification tasks can be used as a standalone program. This system can be used in many ways in everyday life. For example, if a person wants to monitor his/her health condition at home, this system will cope with the specified task. It also can be used in medical facilities as a recommended or clarifying diagnostic system. This system also can show a great benefit being used by persons whose profession involves voice activities (actors, singers, teachers etc.). At the same time, it is widely known that human voice can change in some time before the respective changes will appear in human organism. In spite of this fact, we hope that our system can make a prediction of human health condition. This mode will be useful for persons who have to monitor their condition in real time for a long period directly while performing their duties (drivers, rescuers, divers etc.).
Conclusions
1. Human condition analysis and personal identification are the important tasks for today. To perform the specified procedures a great amount of technologies can be used.
2. In this work, authors propose to use for these tasks authors algorithms of human voice analysis. These algorithms are implemented in a form of script-files for open source applied mathematics package SciLab 5.5.2. In general, developed algorithms are arranged as a complex system that allows to perform both personal identification and human condition analysis based on human voice analysis.
3. These procedures are based on extraction the following characteristics of human voice signal: fundamental frequency value and the structure of amplitude distribution in time domain. For the task of specified characteristics extraction, authors have developed a special algorithm for division human voice signal into separate frames based on the cepstral transform and zero crossing clarifying procedure. The results are saved in a special database.
4. For the task of personal identification, the input voice is preprocessed using the developed algorithms. Comparison of signals is performed in two-dimensional space based on the extracted characteristics. The database signal and the speaker respectively is identified as the closest signal to the input one in this space.
5. For the task of human condition analysis authors form two types of human voice standards - group and individual. Group standard involves dividing the general speakers' database using KNN-graph technology. This type of standard is based on acoustic material of different speakers and takes into account voice signals features of each speaker, taken to the working cluster. Individual standard takes into account only acoustic recordings of a certain person, which has been identified during the identification procedure. Both types of standards are formed by the multiple regression method based on the human anthropometry, gender and age parameters.
6. Comparison of human voice signals for the task of human condition analysis is reformed on the basis of the same human voice signal characteristics and by the same technology. If the deviation of one of the specified characteristics exceeds the defined threshold the signal and human condition respectively considers to be out of norm. Only in the case of both characteristics deviation does not exceeds the defined threshold human condition is defined as normal.
7. The developed system has passed the testing on the real human voice recordings and has shown satisfied results both in the identification mode and in condition analysis. However, during the experimental research identification procedure for some speakers failed in long-time period while working satisfied in short-time period. This fact, in our opinion, shows the dynamical character of human voice signal. In spite of this fact, human voices that are available in the systems' database must be updated after a short period. Research work in this field continues for now.
8. The developed system can be applied in many ways of everyday life as mentioned in the article.
References
1. Mieshkov O.Y, Novikov O.O. Development of Universal Program Complex for Human Condition
Analysis, Based on the Analysis of Human Voice // Theoretical and Applied Aspects of Cybernetics.
Proceedings of the 4th International Scientific Conference of Students and Young Scientists - Kyiv:
Bukrek, 2014. - P. 294-305.
2. Чорна Л.Б. Стохастична модель голосового сигналу для 3aAa4i дiагностики ритмши серця
людини: автореф. дис. канд. техн. наук: 01.05.02 / Л.Б. Чорна; Терноп. держ. техн. ун-т iM.
1.Пулюя. - Т., 1999. - 19 с.
3. Lorraine O Ramig, Cynthia Fox and Shimon Sapir, Speech treatment for Parkinson's disease, Expert
Rev. Neurotherapeutics vol. 8, no. 2, pp. 299-311, 2008.
МАТЕМАТИЧНЕ МОДЕЛЮВАННЯ Ф1ЗИЧНИХI ТЕХНОЛОГ1ЧНИХ ПРОЦЕС1ВIТЕХН1ЧНИХ СИСТЕМ
4. A. Tsanas, M. Little, P. McSharry, J. Spielman, and L. Ramig, Novel speech signal processing algorithms for high-accuracy classification of Parkinson's disease," IEEE Transactions on Biomedical Engineering, vol. 59, no. 5, pp. 1264-1271, 2012.
5. Первушин Е.А. Обзор основных методов распознавания дикторов. // Математические структуры и моделирование, № 24, С. 41-54, 2011.
6. Алгоритм извлечения признаков речевого сигнала во временной области для задачи распознавания дикторов // Е.А. Первушин, Д.Н. Лавров, Вестник Омского государственного университета / 2011, №2, с. 182-185.
7. S. Bhatia, Engineering Biomaterials for Regenerative Medicine: Novel Technologies for Clinical Applications. Springer, 2011.
8. J. Filho, E. de Melo, D. Tsuji, C. de Giacomo Carneiro, and L. Sennes, Length of the human vocal folds: proposal of mathematical equations as a function of gender and body height, The Annals of Otology, Rhinology & Laryngology, vol. 114, no. 5, pp. 390-392, 2005.
9. J. Kreiman and D. Sidtis, Foundations of Voice Studies: An Interdisciplinary Approach to Voice Production and Perception. Wiley, 2011.
10. W. Fitch and J. Giedd, Morphology and development of the human vocal tract: a study using magnetic resonance imaging, Journal of Acoustic Society of America, vol. 106, no. 3, pp. 1511-1522, 1999.
11. Понизов А.Г. Устройство и методика формирования тестовых акустических сигналов эквивалентных камертону для оценки качества слуха: автореф. дис. на соискание ученой степени кандидата технических наук: спец. 05.13.05 - Элементы и устройства вычислительной техники и систем управления / Понизов А.Г. - Томск, 2012. - 20 с.
12. I. Titze, Vocal fold mass is not a useful quantity for describing F0 in vocalization, Journal of Speech Language and Hearing Research, vol. 54, no. 2, pp. 520-522, 2011.
