CC BY
7
https://doi.org/10.29013/AJT-21-9.10-8-16
Bratskyi Vadim Olegovych, master, the Faculty of Information Systems National University of Food Technologies, Kyiv, Ukraine E-mail: vadymbratskyi@gmail.com Myakshylo Olena, PhD, Associate Professor Department of Information Systems National University of Food Technologies, Kyiv, Ukraine
E-mail: mem2004@ukr.net
EXPERT SYSTEM OF DIAGNOSIS OF ERRORS IN THE OPERATION OF REMOTE NODES OF THE DISTRIBUTED SYSTEM OF THE BANK
Abstract. Due to log files coming to the main office from the bank's branches, analysts have the opportunity to receive information about the correct operations of customers, as well as failures in the software and hardware of the nodes. The developed expert system, under the control of the LogHelper program, allows to build clear and exact inquiries for the analysis of log files, automatically find and offer the decision for correction of the found errors, classify unknown errors.
Keywords: log files, expert diagnostic system, MongoDB.
I. Formulation of the problem
In distributed systems, in particular in banking systems, failures during important transactions lead to a loss of bank profits and a decrease in customer confidence. The problem was a huge number of log files from remote offices, which had to be sorted by date, to find the place where something was supposed to indicate the reason for failure, and it took a lot of time and nerves. At that time, it was decided to write a software product that will help to analyze log files faster and more efficiently, which is necessary both for testing the bank's IT system and for its technical support. Subsequently, this program became the basis of the diagnostic expert system. When developing a diagnostic expert system, it is obvious that two aspects are important:
1. The diagnostic expert system should provide a quick search in the log file of messages about failures in the system, identify the causes of failures and suggest ways to solve the problem. This problem can be represented as an objective function:
V n Tft +Ym Ljtj + Tn + min, (1)
¿—a=1 i i ¿—¡ )=1 ; ; n p '
where l. - ith log file object, n - the number of objects in the log file, 1 <i< n,
t. - the time of viewing the ith log file object, l. - jth log file object, in which an error message was found, m- the number of objects in the log file in which errors were found, 1 < j < m
t. - the time of processing the jth object of the log file, identifying the cause of the failure and the formation of proposals for its elimination,
T - the time of transmission of the log file with messages about ways to eliminate errors,
Tc - time for correction errors sent in the log file. The diagnostic system cannot affect the response time to the client (T) and the error correction time (Tc). The system should reduce the time to find the causes of failures and the time to find solutions to eliminate them.
Restrictions have to be imposed on the components of the objective function.
EL v, < t , (2)
where - T - time to view the log file without the use of programs;
EI, j < T,. (3)
where - Tg -time for searching a solution using Google and other search engines.
2. The developed system should provide service for identifying errors which are not in the knowledge base, ie, should be constantly updated with new knowledge.
Accumulation of statistical information from log files allows to select a set of N-states of normal system operation, Z-states corresponding to failures of various kinds. Y is a set of states that have not yet been determined. Then,
X - N u Z uY, (4)
where X - the set of all possible states of the bank s IT system.
It was supposed that X can have a finite number of states x^, X3 • • • x. . » » x, which correspond to both the correct functioning of the system and work with the presence of deviations, p p2... p.... pn- the probabilities of the system to be in states xx,x ,... x.... xn,respectively. Then the entropy (a measure of uncertainty) of such a system will be calculated by the formula:
H(X ) = -£>& pi, (5)
where ^pi = 1
For a distributed bank system consisting of s independent systems, the total entropy will be the sum of the entropies of all systems:
H (X,,..., X ) = Ç=, H (Xk ). (6)
In the case when the bank's branches operate IT systems identical in terms of hardware and software
Table 1.- Comparison table
they can be expected to have the same indicators of correct operation by their nature, as well as failures and errors. In this case, it can be assumed that formula (3) will turn as follows:
H(X1V..,X) - s• H(Xk). (7)
As can be seen from formula (7), the tasks of control and diagnostics are to reduce the entropy of a typical node system by detecting known errors and identifying unrecognized situations in the operation of nodes (remote offices) of the banking system.
This goal can be achieved by studying and analyzing set of messages (log files) which come from remote nodes from the bank's branches to the central office or a special diagnostic center.
That is,
H(X) = -LlpPibg2 Pi ^ minH, (8)
Provided Epi = 1.
According to formula (8), the entropy will decrease due to the reduction of the set Z, which consists of unrecognized situations recorded in the log files.
To solve these problems, the most suitable is expert system, based on the knowledge base, which can store the contents of log files, quickly process them, find the right solution and have a mechanism to replenish the knowledge base.
There is a number of software products at the market which perform log file processing functions. The study and comparison of existing parsers of log files, presented in [3], showed their unsuitability for solving the tasks. The results of the comparison of four systems, which are designed to solve search problems in log files, according to the specified criteria, are shown in (Table 1).
of Parsers of log objects
Log Parser 2.2 Logstash Web Log Explorer Web Log Storming Tasks
1 2 3 4 5
- - + + Interface
+ + + - Support for various formats
- - - - Failure sustainability, impact on resources
1 2 3 4 5
+ - - - The presence of a database
- - + + Clarity in settings
- - - - Data analysis
As it can be seen from (Table 1), none of these programs provides data analysis, which is essential for the diagnostic system.
II. Selection of database for expert system and log file structure development
As it was shown in [1; 2; 7], the non-relational MongoDB database has a number of advantages over relational databases, namely:
- is convenient for storing documents of variable length;
- scales much better than a relational database, which is important when processing very large amounts of data;
- is effective for analytical data processing.
Log file structure model
Since it is possible to specify the format oflog files, their structure, it was proposed to save all failures registered in the log files ofthe system in the JSON document format, which is used in the MongoDB database.
An object in a NoSQL MongoDB database is a document which has its own ID and attributes. The number of attributes is not regulated, but the total volume of the document cannot exceed 16 Kv. Documents are collected in such way, which is a prototype table in a relational database. Communication among documents is provided by means of identifiers. In view of the above, the document in the MongoDB database can be described as a tuple of the form: D ={< f < f: e, f2: e2..., /„: e, f + 1: ^ f + 2: ^ fn + l: d >}, (9)
wheref -id of the document;
f ...f - document attributes;
e,... e - atomic values of document attributes;
1 n '
d ...dl- links to other documents.
Formula (9) corresponds to the case where the parent document contains references to child documents through their id.
There is another case of providing a link among documents, when the parent document contains a subsidiary document with its ID and attributes.
Then the model of the document as a database object will look like:
D ={< f0: e0< f1: e1, f2: e2", U en, < fn + 1: d0, fn + 2 : edl,..fn + l: edl >>}, (10)
Ae f0: e0 -id of the document;
f ...f - document attributes;
e, . e - atomic values of document attributes;
1n
fn + 1: d0 - id of subsidiary document; ed ...edl- atomic values of subsidiary document attributes.
The structure of the log file object as a MongoDB document
To control the correct operation of each node of the bank's corporate system, the log file is created every day, in which all user actions are recorded. The software product is a client-server, and therefore records everything that the client sent to the server and that the client received from the server. The structure of the log file object was considered as the following example:
«=> BarsScheduler at 5/8/2016 7:17:45 AM: from 10.7.73.12; RequestType = POST; ContentType = application/json; charset=»UTF-8»;
Input = {"sessionid":"slxctezudkzyv0di3ymysxyd ","method":"CloseSession","params": null,"message_ id":"BARS-MESS-6735248"}
<= BarsScheduler at 5/8/2016 7:17:45 AM: to 10.7.73.12
Output = {"status":"OK","RESULT":{"sessionId": "slxctezudkzyv0di3ymysxyd"},"message_id":"BARS-MESS-6735248","responce_id":"1","current_ timestamp":"2016-05-08T07:17:45"}»
The following parts can be selected in the structure of the log file object:
1) Date and information about the request.
- Server title, date and time of the received request (BarsScheduler at 5/8/2016 7:17:45 AM);
- IP- address from which the request came (from 10.7.73.12);
- Request type (RequestType = POST);
- Content type (ContentType = application/ json);
- Unicode conversion format (charset="UTF-8");
2) Request body.
- Input = {...};
3) Date and information about the answer.
- Server title, date and time of sending a response (BarsScheduler at 5/8/2016 7:17:45 AM);
- IP - addresses to which the response is sent (to 10.7.73.12);
4) Response body.
- Output = {...}.
The description of the structure of the log file in the form of a tuple, according to formula (9) was defined as:
D = F F < P-: e, P{: e^ P3: e, P4: e4 > ^ F < < Pv eXi P2 : e^ P3 : e^ Pr e4 >, where
1. F0 - ID of the document
2. F1 - Date and information about the request;
3. F2 - Request body
4. F3 - Date and information about the answer;
5. F4 - Response body;
6. P - P - document attributes;
1n
7. e,... e - atomic values of document attributes.
1n
The structure of the log file object with attached documents
A model with attached documents was also developed to save parser-processed log files. The view of the Log class presented in C # language is shown
in (Fig. 1).
class Log {
public Objectld Id. { get; set; } public DateTiiue ReguestDate { get; set; } public BsonDocuiuent Reguest { get; set; } public BsonDocuiuent Response { get; set; } public DateTiir.e Re3ponseDate { get; 3et; } public Dbjectld LogFileld { get; set; } public string Messageld { get; set; }
}
Figure 1. Description of the Log class in C# language
Ehe type of parent document with attached subsidiary documents was considered as a tuple, according to the formula (10):
D = F F2 < P1:: ei> P2 : e2> P3 : № : PS
>, F3 < P1 : e, P2 : [ah en] p3 : e3, pA : e, p5 : e5 >, F,
F5< Pi : d > F6}
where
1. F0 - Id;
2. F1 - Request date
1. F2 - Request document;
2. F3 - Response document;
3. F4 - Response date;
4. F - Document link;
5. F6 - Id a message that belongs to one request and one response.
6. P - P - document attributes;
1n
7. a1 - an - array attributes;
8. e,...e - atomic values of document attributes;
1n
9. d1...dl - links to other documents. Therefore, a log file structure model adapted for
storage in MongoDB in JSON format was proposed.
IV. Processing and analysis of log files in the diagnostic expert system
For easy storage, processing and analysis of log files, a two-tier server architecture was chosen. The database and knowledge base were stored on a separate server.
The object model of diagnostic expert system is sists of objects (collections): Log, Error, StatusError, shown in (fig. 2). As for the object model, it con- Known Error, Answer, Log Query, Query Config.
Figure 2. Object model of system data
In the Error collection all errors unknown to the knowledge base are saved, which will be further processed by the administrator, expert analyst or programmer; In the StatusError collection the statuses for the processed errors by means ofwhich there is a classification of errors are saved; In the KnownError collection all known to the knowledge base errors which were processed by the administrator, analyst or programmer are saved;
In the Answer collection variants of the solutions for the processed errors and errors which are in the KnownError collection are saved;
In the LogQuery collection filter options are saved, by properties, to find the desired log objects;
In the QueryConfig collection the filtering options already selected by users are saved.
The diagnostic expert system works under the control of the LogHelper software product. The description and detailed algorithm of the program are given in [4]. The structure of the expert system is shown in (Figure 3), which consists of: the database designed to store log files received and processed by the parser; knowledge base, which contains the knowledge of experts about known errors and ways to eliminate them; a mechanism of logical conclusions that allows to identify errors with those saved in the knowledge base, a module of knowledge acquisition, which provides interaction of the system with the expert and a module of advice and explanations, which is the interface between the user and the system.
System administrator, expert analyst and programmer-developer can be users of the system in the process of testing a distributed system.
Figure 3. The structure of the expert system
After starting the LogHelper program, the system offers to select log files for analysis, or the user can select from the top menu bar the section «work with log objects», where it is possible to:
1. Construct queries on log objects;
2. View the results of the query;
3. Consider all known and unknown, for the system, errors;
4. Process the unknown error and set the status for it;
5. Suggest existing solutions to the user.
The mechanism of logical conclusions
The mechanism of logical conclusions provides a search by complete coincidence or by key words, for information about the error in the log file object processed by the parser (hereinafter - the error). If such information is present in the knowledge base, then the mechanism of logical conclusions transfers control of the module of advice and explanations. If not, the inference mechanism compares the objects processed by the parser with already existing messages similar to the found error, counts the number of similar errors and saves it in the database for further processing.
Knowledge acquisition module
Processing of unknown errors. When analyzing log files, the system checks the generated Logs object for the error property and determines whether the error is known to the system. The check is based on data on known errors from the knowledge base. If the system recognizes the error as unknown, it will be saved in the database and in the future processed by the analyst. Expert Analyst has the opportunity to process the error by setting the status and option or solutions (Fig. 4). In the background it is seen that the error message identifies 'Not found' and indicates the number of errors of this type. The system attracts attention of the analyst-expert to an unknown error, indicating the frequency of its occurrence. The active window lists the events close to the unknown error and their status.
When specifying a status for an unknown error, it is possible to create a new status or select an existing one. With the help of the status it is easy to classify errors and filter data for further analysis of the correctness of the distributed system of the bank.
Figure 4. Interface for determining the status of an unknown error
The module of advice and explanations - the offer of options of decisions for the user
Known errors. This section displays all errors handled by experts which are known to the system and are stored in the knowledge base. The known error, as a result of the logical inference mechanism, (full match search or by keywords) is displayed with the following information:
• Error status, which describes its code and title;
• The answer or solution of this problem;
• Number of similar errors found.
The found information is provided to the user through the user interface (Fig. 5).
Figure 5. Administrator user interface with solution offer
When the processing of log files is completed, the expert system displays information about the result of processing - this is the number of created objects and errors found, as well as suggestions for their elimination. The user has the opportunity to see the errors which are known to the system and view the solutions offered by the system.
When processing a log file, the LogQuery object is expanded with new properties found during the analysis of the log object. The user has the opportunity to select the desired property and add it to the QueryConfig collection for further quick search for the specified parameters.
A certificate of ownership of the computer program Log Helper was received [5].
IV. Efficiency of implementation of expert diagnostic system
One of the tasks of the study was to evaluate the effectiveness of the proposed method and expert system. The effectiveness of the expert system, as proposed in [6] was considered on the example of determining the payback and return (return of lost profits) from investment in the developed project. Profitability of the project - ROI (return on investment) can be calculated by the formula:
ROI = net return on investment / amount of expenses (investments)
As sources of economic effect (ROI) from the introduction of the expert system, the amount of returned benefit from the successful conduct of transactions in the distributed information system of the bank was considered. The amount of lost profit from the failure of transactions for the settlement period (month) was calculated by the formula (11)
, (11)
where x- the volume of the bank's provision of the jth type of services;
P. - profit from the provision of the jth type of services;
J - transactions which failed as a result of a system error and the benefits of which were lost;
It was assumed that formula (11) corresponded to the lost benefit which can be recovered as a result of the operation of the diagnostic expert system.
The objective function of recovering lost profits through the use of an expert system will be as follows:
F(j) - C + Co+ Cc) ^ max' (12)
where
Ca - depreciation deductions from the cost of the expert system,
Cv - monthly costs for the operation of the expert system
Cc - monthly expenses for maintenance of the expert system.
As a result, the conclusion was, that it was necessary to assess the return on investment in the developed expert system by making profit from transactions which failed, but were quickly corrected and did not lead to lost profits by either the bank or the client. A few examples of solutions which can be sources of ROI:
1. Monitoring of failures, data on which are already available in the expert system of the bank:
- identification of the cause of failure;
- search for recommendations (solutions) to eliminate the problem;
- transmission of the message to the node of the distributed system.
The solution was providing up-to-date information on the nature of the failure or error and ways to eliminate them. Since the selected DBMS MongoDB has a high speed, the client receives feedback from the system in minutes, in contrast to the manual processing of arrays of log files.
2. Monitoring of failures, data on which are missing in the expert system:
- providing the system administrator with a list of similar errors that have already been described in the knowledge base;
- replenishment of the knowledge base with new knowledge about possible errors, which reduces the entropy of the system.
The solution was replenishment of the knowledge base, which enhances the benefits of implementing an expert system, described in paragraph 1.
3. Monitoring of successful transactions:
- accumulation of data on successful transactions in the knowledge base;
- collection of statistical data based on the calculation of successful transactions;
The solution was use of the accumulated information to analyze the efficiency of the bank's branches.
The result of the implementation of the system will be an increase in the bank's income by reducing losses on individual transactions of the bank by prompt detection of errors.
V. Conclusions
As a result of the study it was found that log files are very powerful tool for analyzing the performance of remote nodes in a distributed system. Using an expert system, run by LogHelper, allowed to build clear and accurate queries for the analysis of log files, create a classification of errors. The system provides a learning process and has knowledge of known and unknown errors.
With the help of a knowledge base, the expert system has the ability to automatically find and suggest solutions to found errors. All these measures will help to increase the reliability of the distributed system and thus increase the efficiency of the bank.
References:
1. Бэнкер К. Mongo DB в действии / Кайл Бэнкер- М: ДМК Пресс, 2016.- 394 с.
2. Брацький В. О. «Досл!дження особливостей застосування реляцшних i не реляцшних баз даних на прикладi Sql Server Та Mongodb» / В. О. Брацький, О.М. М'якшило: К.- Науковi пращ НУХТ,-Том 22.- № 5. 2016.- С. 15-24.
3. Брацький В. О. Порiвняння методу обробки i анализу log-файлiв у формата JSON з кнуючими pi-шеннями / Брацький В.О., М>якшило О. М. Науковi пращ НУХТ.- Т. 24.- № 3. 2018.
4. Брацький В. О. «Досл^ження та розробка методу обробки log-файлiв у розподгленш шформацшнш системi з використанням нерелящйно!' бази даних MongoDB"/ В. О. Брацький, О.М. М'якшило. Науковi пращ НУХТ.- Т. 24.- № 1. 2018.- С. 17-25.
5. Брацький В. О., М'якшило О. М. СвЦоцтво про реестращю авторського права на твip комп'ютерна програма «Аналiз log файлiв» № 95824 вЦ 6.02.2020, видано Мшктерством розвитку економжи, тоpгiвлi та с1льського господарства Украши.
6. Мойсеенко I. П. 1нвестування [Текст]: навч. поаб. / I. П. Мойсеенко - К.: 2006.- 490 с.
7. Фаулер М. NoSQL: Новая методология разработки нереляционных баз данных / М. Фаулер, Пра-модкумар Дж. Садаладж - М.: «Вильямс», 2013.-192 с.
