CC BY
20
ВЕСТНИК ТОМСКОГО ГОСУДАРСТВЕННОГО УНИВЕРСИТЕТА
2021 Управление, вычислительная техника и информатика № 55
МАТЕМАТИЧЕСКОЕ МОДЕЛИРОВАНИЕ
УДК 519.6
DOI: 10.17223/19988605/55/2
I.A. Brokarev, M.P. Farkhadov, S.V. Vaskovskii RECURRENT NEURAL NETWORKS TO ANALYZE THE QUALITY OF NATURAL GAS
Comparative analysis of various neural network models was carried out for natural gas quality analysis. Based on the results of such analysis, it was concluded that recurrent neural networks are main statistical models in this problem. This paper considers a recurrent neural network with a more complex architecture. The neural network with gated recurrent unit is used in the discussed task in particular. The comparison of the main recurrent neural network models (simple recurrent neural network, recurrent neural network with long short-term memory, recurrent neural network with gated recurrent unit) is shown. Models accuracy characteristics are shown for analyzing the models performance.
Keywords: recurrent neural networks; natural gas quality analysis; gated recurrent unit.
The natural gas quality analysis is an important task for the gas industry. Slight fluctuations of natural gas composition and energy characteristics can lead to unexpected difficulties in calculating its cost indicators. Currently, a wide variety of different natural gas analysis systems are developed. Moreover, many alternative systems that are based on the correlation methods are under development. The possibility to analyze gas quality in real time is the most significant benefit of this class of systems in comparison with systems based on the traditional gas chromatography methods. However, systems that are commonly used in gas industry have a number of drawbacks: expensive specialized equipment, significant amount of time of the analysis, the necessity of regular instrumentation calibration and checkout.
Various statistical models are used in correlation methods because of high complexity of solving the task with traditional computational methods. The choice of statistical model for the gas quality determination is made by heuristic methods in most cases due to the lack of a general algorithm. That is why comparative analysis of statistical models for the discussed task is an urgent problem that should be solved for reaching the required goal.
This paper provides a proposition of using recurrent neural network with gated recurrent unit for natural gas composition determination. The model differs from previously used neural network in the form of multilayer perceptron and simple recurrent neural network. The main difference of the proposed system from previous papers [1, 2] is usage of more complex neural network architecture (network with gated recurrent unit).
1. Proposed recurrent neural network
Based on the results of correlation analysis and neural network models investigated in previous works [3], we choose the speed of sound, thermal conductivity coefficient and concentration of carbon dioxide as the model input parameters. As the output parameters, we choose the concentrations of the components included in the investigated gas mixture.
The Levenberg-Marquardt algorithm was chosen to train the model. This algorithm optimizes parameters of nonlinear regression models. Note that in this algorithm, the optimization criterion is the root-mean-
square error of the model on a training sample. The basic idea of the algorithm is as follows: to minimize the error locally, the initial values of the parameters are approximated. Let a regression sample be a set of pairs of an independent variable X and a dependent variable Y, and the regression model is a continuously differenti-able function F. It is necessary to find the value of the parameter vector W, where the error function Fe reaches its local minimum:
N
Fe = X (Y " F(X ,W)). (1)
i=\
On the first iteration of the algorithm, the initial vector of parameters W0 is specified. On each following iteration, the vector is replaced by the vector Wo + AW. To estimate the increment AW, the following approximation of F is used:
F(Wo + W, X) - F(W0, X) = JW, (2)
where J is the Jacobian of F.
The increment AW at the minimum of Fe is zero. So, to find the subsequent value of the increment AW, it is necessary to set the vector of partial derivatives of Fe over W to zero. Then we differentiate this expression over W and set the partial derivative to zero. After all transformations, we get the expression for AW:
W = (JT J)-1 JT (Y - F(W)). (3)
For this algorithm, the condition number of the matrix is important; the number shows how close the matrix is to the partial rank matrix (for square matrices it shows how close the matrix is to degeneracy). Since the condition number of the matrix JT J is equal to the squared condition number of the matrix J, the matrix JT J may turn out to be degenerate. For this reason, in this algorithm we introduced a regularization parameter X; the parameter is greater than or equal to zero. This parameter is selected on each iteration of the algorithm. Given the regularization parameter, expression for AW takes the following form:
W = (JT J + 1 E)- JT (Y - F(W)), (4)
where E is a unity matrix.
There is a modification of this method, where the regularization parameter is multiplied by the matrix D, a diagonal matrix with the elements that coincide with the diagonal elements of the matrix JT J. This approach is used to reduce the effect of the regularization parameter on the AW value.
It is necessary, however, to note that this algorithm converges slower if the step is constant, which is a disadvantage. The problem is solved by introducing a coefficient K; the coefficient determines the step length and makes the method converge faster. Given the two amendments described, the expression for AW takes the following form:
W = K (JT J + 1E)- JT (Y - F(W)) . (5)
The value of the vector W at the last iteration of the algorithm is the target. It is reached either if the calculated increment AW is less than the specified value, or if the error function Fe is less than the specified value for the vector W.
To solve this problem, we use a recurrent neural network with a gated recurrent unit (GRU) [4] to model our system. Gated recurrent units are a specialized gate mechanism for recurrent neural networks. The structure of a gated recurrent unit is shown on Figure 1. The structure of a neural network with this block is similar to the previously presented simple recurrent network.
The main differences between neural networks with GRU and neural networks with LSTM are the number of gates and maintenance of cell states. Unlike neural networks with GRU, neural networks with LSTM have three gates (input, forget, output) and maintains an internal memory cell state, which makes it more flexible, but less efficient memory and time wise. However, since both of these networks are great at addressing the vanishing gradient problem required for efficiently tracking long term dependencies. When deciding between these two variants, it is recommended that firstly using neural networks with LSTM, since it has more parameters and is more flexible, followed by neural networks with GRU. If there are no sizable differences between the performance of this two types of RNN, then the simpler and more efficient neural networks with GRU are used.
Fig. 1. The architecture of gated recurrent unit
Recurrent neural networks (RNN) is a class of neural networks that can use their internal memory when processing input data [5]. The functioning of this class of neural networks is based on the use of previous network state to calculate the current one. A recurrent network can be considered as several copies of the same network, each of which transfers information to a subsequent copy. Currently, there are a large number of architectures of recurrent neural networks. Taking into account the computational difficulties encountered in developing this class of neural networks, it was proposed to consider a simple recurrent neural network first. The hidden elements have links directed back to the input layer in such type of network. This allows to take into account the previous state of the network during training. Mathematically, the process of saving information about the previous training step is as follows: at each i-th training step, the output value of the RNN hidden layer hi is calculated taking into account the output value of the hidden layer in the previous step.
The proposed model with gated recurrent unit is similar to recurrent neural network with long short-term memory [6]. Long short-term memory (LSTM) is a special type of architecture of recurrent neural networks, that is capable of learning long-term dependencies. A more complex method is used to calculate both the output value of the hidden layer and the output value of the network as a whole in neural networks with a similar architecture. This method involves use of so-called gates. A gate is a special unit in LSTM architecture, that is implemented as a logistic function and operation of elementwise multiplication (Hadamard's product). The logistic function layer shows how much of the information coming from a particular unit should be transmitted further along the network. This layer returns values in the range from zero (information does not go further along the network structure at all) to one (information completely goes further along the network structure). There are three such gates in traditional LTSM architecture: a forget gate, an input gate and an output gate. The sigmoid function is often used as a logistic function for gates.
This neural network architecture includes an update gate, a reset gate and a current memory container. Both gates are functions of the input vector and the state of the hidden layer on the previous step. The update gate determines the amount of information from the previous state of the hidden layer that should affect its current state. The reset gate determines the part of in-formation from the previous state that will not be taken into account when we calculate the current state. The current memory container is used to store the relevant information from the previous state and depends on the reset gate.
Let us investigate how the gated recurrent unit operates. When the vector xi is fed to the block, the vector is multiplied by the vector of weights Wz of the update gate. The vector of the previous state
of the hidden layer h-1, that stores information about the previous i - 1 steps, is multiplied by the vector of weights Uz of the update gate. Then the two products are added and fed to the sigmoidal activation function o. As a result, the output value of the update gate zi is calculated as:
Z =G(WzXi + Uh-i) . (6)
The formula to calculate the output value of the reset gate ri is similar to the formula for the update gate, except for the weight vectors Wr and Ur
r =o(WrXi + Urhi-1). (7)
We use the value of the reset gate to calculate the current value of the memory container. First, the corresponding vectors of the weights W and U. Multiply the input vector and the previous state vector of the hidden layer Then the Hadamard product of the state reset filter value and the Uh-1 value is calculated. This is necessary to determine the amount of information from the previous steps we do not consider. Then, we sum up the two previously obtained terms and apply the nonlinear hyperbolic tangent activation function (tanh) to obtain the value of the current memory container He
H = tanh(WXi + rUh-i). (8)
If the value of the reset gate is close to one, then the memory container saves information from the previous hidden state. If the value of the reset gate is close to zero, then the memory container does not take into account the previous hidden state.
At the final stage of the block, the vector hi is calculated; the vector contains the information about the current state of the hidden layer and is transmitted to the next layer of the neural network. For this procedure, we use an update gate; the filter determines how much information to take from the current memory container and from the previous state of the hidden layer. To do this, the Hadamard product of the update gate value and the vector of the previous state of the hidden layer, and the Hadamard product of the value of the current memory container and the value (1 - zi) are calculated. Finally, we sum up the results of the two products to get the block output value:
h = zh-i+ (1 - Zi )H . (9)
If the update gate value is close to one, then the current memory container is not taken into account, and the block output equals the previous value of the hidden layer. If the update gate value is close to zero, then the previous state of the hidden layer is not taken into account, and the block output depends only on the current memory container.
The main advantage of a managed recurrent block is the ability to store and filter information using update and reset gates. Such system has no vanishing gradients as the model stores the relevant information and passes the information on to the next time steps of the network.
2. Testing of neural network model
Based on the results of this study, we obtain a neural network model with a gated recurrent unit to analyze the quality of natural gas by measured physical parameters. To determine if the proposed neural network model is adequate, we studied a test sample of data. The results of composition determination and accuracy of the proposed model are shown in Table 1. Maximum absolute deviation (MaxAD), mean absolute deviation (MAD), mean square error (MSE) and determination coefficient (R2) are determined.
The ranges of gas mixtures for the test sample were as follows: 85-100% for methane, 0-5% for propane, nitrogen and carbon dioxide. We took the concentration of carbon dioxide to be zero, since this parameter is an input for the model and is considered a known value.
Table 1
Accuracy of RNN model for determination of the natural gas composition
Component MaxAD, % MAD, % MSE R2
Methane 0.73 0.33 0.32 0.9999
Propane 0.21 0.12 0.13 0.9999
Nitrogen 0.19 0.08 0.10 0.9999
Table 2
Comparison of models accuracy characteristics for determination of the natural gas composition on testing set
Component Characteristic Model
ANN RNN LSTM GRU
Methane MaxAD, % 0.823 0.767 0.783 0.781
MAD, % 0.456 0.381 0.356 0.343
MSE 0.581 0.491 0.451 0.401
R2 0.998 0.9998 0.9998 0.9999
Propane MaxAD, % 0.312 0.271 0.281 0.256
MAD, % 0.161 0.183 0.171 0.128
MSE 0.341 0.291 0.289 0.195
R2 0.998 0.9999 0.9999 0.9999
Nitrogen MaxAD, % 0.512 0.331 0.305 0.245
MAD, % 0.267 0.181 0.179 0.115
MSE 0.356 0.201 0.197 0.191
R2 0.998 0.9998 0.9998 0.9999
The comparison of models accuracy characteristics for determination of the natural gas composition on training and testing set is shown in Table 2. The fact that the statistical model can show good results on a training set, but a high error on a testing set is taken into account.
3. Using the proposed model in natural gas quality determination system
The main structure of the proposed natural gas quality determination system is shown in Figure 2. The system consists of three main blocks. It is proposed to use commercially available and relatively inexpensive sensors for natural gas physical parameters measurements to obtain necessary measurement data. That data are input data of the proposed system. The aim of the system is to determine target natural gas quality parameters using input measurement data.
Fig. 2. Structure of the proposed natural gas quality determination system
The main advantages of the proposed system are possibility to operate in real time due to the usage of neural network model, the broad list of output quality parameters and the joint usage of the system with traditional gas chromatographs. The proposed recurrent neural network is located in the first block (gas composition determination). It is the main block of the designed system. It contains the majority of the proposed system features. The main goal of the proposed system is gas energy parameters determination. To calculate the energy parameters of the gas under study, NIST REFPROP software is used. The target energy parameters for the discussed task are volumetric superior calorific value and Wobbe index. These parameters along with partial gas composition and relative density are considered to be final gas quality parameters that system should determine. To calculate the quality parameters, the GERG-2008 gas state equation was used at standard temperature and pressure conditions. The amount of output parameters can be decreased to simplify the calculations or increased by adding volumetric inferior calorific value in special cases. The next step involves energy parameters accuracy check that occurs in the corresponding block. The calculated in previous block gas quality parameters are compared with reference data. Any data obtained from traditional natural gas analyzers, e.g. gas chromatographs, can be used as the reference data.
The volumetric calorific value determined using the proposed recurrent neural network is shown on Figure 3. The determination of target energy characteristics is conducted in the block of the designed gas quality determination system. That block is located after the gas composition determination block where the proposed neural network model determines the desired gas composition.
Volumetric superior calorific value
0.04 r
0 20 40 60 80 100 120 140 160 180 200
Composition number
Fig. 3. The accuracy of determination of volumetric calorific value designed gas quality determination system
Theoretical values of natural gas energy parameters was used as reference data. The volumetric superior calorific value and Wobbe index were calculated using determined gas composition and compared with reference data. The maximum absolute error of gas quality parameters determination is less than the allowable error that is equal to 0.1 MJ/m3. The allowable error is permissible deviation of gas quality parameters determination for the first accuracy class according to current regulatory document [7].
Conclusion
Proposed neural network model with a gated recurrent unit determines the quality of natural gas by the values of the gas measured parameters. Our model is adequate to determine accurately the composition of the gas for the specified problem. The simulation results showed that the maximum absolute error in determining the gas quality parameters is less than the permissible error value determined by the current regulatory document.
Another advantage of the proposed gas quality analysis system is possibility to operate in real time due to the usage of recurrent neural network model.
Acknowledgments. The authors are grateful to the international cooperation projects BRISK II TA and Erasmus + 2017-1-SE01-KA107-034292 Staff Mobility between the universities for the opportunity to conduct this study.
REFERENCES
1. Vaskovskii, S.V. & Brokarev, I.A. (2020) Comparative analysis of statistical models for the task of natural gas composition analysis.
Journal of Information Technologies and Computing Systems. 1. pp. 34-43. DOI: 10.14357/20718632200104
2. Brokarev, I.A. & Vaskovskii, S.V. (2020) Gas Quality Determination Using Neural Network Model-based System. Proceedings of
the 2nd International Workshop on Stochastic Modeling and Applied Research of Technology (SMARTY 2020). pp. 113-128.
3. Dörr, H., Koturbash, T. & Kutcherov, V. (2019) Review of impacts of gas qualities with regard to quality determination and energy
metering of natural gas. Meas. Sci. Technol. 30(2). Article ID 022001.
4. Chung, J., Gulcehre, C., Cho, K. & Bengio, Y. (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling.
arXiv preprint arXiv :1412.3555.
5. Callan, R. (1999) The Essence of Neural Networks (The Essence of Computing Series). Prentice Hall Europe.
6. Hochreiter, S. & Schmidhuber, J. (1997) Long short-term memory. Neural computation. 9(8). pp. 1735-1780.
7. ISO 15971:2008. (2008) Natural gas - Measurement of properties - Calorific value and Wobbe Index. International Organization for Standardization. 2008. [Online] Available from: https://www.iso.org/standard/44867.html (Accessed: 2nd March 2021).
Received: March 2, 2021
Brokarev I.A., Farkhadov M.P., Vaskovskii S.V. (2021) RECURRENT NEURAL NETWORKS TO ANALYZE THE QUALITY OF NATURAL GAS. Vestnik Tomskogo gosudarstvennogo universiteta. Upravlenie, vychislitelnaja tehnika i informatika [Tomsk State University Journal of Control and Computer Science]. 55. pp. 11-17
DOI: 10.17223/19988605/55/2
Брокарев И.А., Фархадов М.П., Васьковский С.В. РЕКУРРЕНТНЫЕ НЕЙРОННЫЕ СЕТИ ДЛЯ АНАЛИЗА КАЧЕСТВА ПРИРОДНОГО ГАЗА. Вестник Томского государственного университета. Управление, вычислительная техника и информатика. 2021. № 56. С. 11-17
В статье рассматривается рекуррентная нейронная сеть с управляемым рекуррентным блоком. Показано сравнение основных моделей рекуррентных нейронных сетей (простая рекуррентная нейронная сеть, рекуррентная нейронная сеть с долгой кратковременной памятью, рекуррентная нейронная сеть с управляемым рекуррентным блоком). Приведены точностные характеристики нейросетевых моделей для анализа качества природного газа.
Ключевые слова: рекуррентные нейронные сети; анализ качества природного газа; управляемый рекуррентный блок.
BROKAREV Ivan Andreevich (Post-graduate Student, Assistant, National University of Oil and Gas «Gubkin University», Moscow,
Russian Federation).
E-mail: brokarev.i@gubkin.ru
FARKHADOV Mais Pashaevich (Doctor of Technical Sciences, V.A. Trapeznikov Institute of Control Sciences of Russian Academy of Sciences, Moscow, Russian Federation). E-mail: mais@ipu.ru
VASKOVSKII Sergei Vladimirovich (Candidate of Technical Sciences, V.A. Trapeznikov Institute of Control Sciences of Russian Academy of Sciences, Moscow, Russian Federation). E-mail: v63v@yandex.ru
