CC BY
139
UDC 004.9 : 004.62 https://doi.org/ 10.35546/kntu2078-4481.2019.3.12
I. DYCHKA
National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"
ORCID: 0000-0002-3446-3076 Ye. SULEMA
National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"
ORCID: 0000-0001-7871-9806 I. BUKHTIIAROV
National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"
ORCID: 0000-0002-0541-5251
DIGITAL TWIN INFORMATION TECHNOLOGY FOR BIOMEDICAL DATA COMPLEX REPRESENTATION AND PROCESSING
The paper presents an information technology of digital twin for implementation in healthcare, in particular in e-health and m-health applications. The primary objective of this research is to develop a concept of digital twin information technology for medical decision support systems. The second objective is to analyse various medical data formats and to develop an approach to synchronization of multimodal medical data. The approach proposed in the paper will enable aggregation of multimodal data sequences obtained from a wide range of medical diagnostic equipment with the purpose of a patient's digital twin creation. The paper presents an analysis of data synchronization possibility and data representation formats for both single-channel and multi-channel biological signals, results of such investigations as blood tests, ultrasound research, magnetic resonance imaging etc.
Digital twin technology will enable development of a new generation of medical decision support systems. A digital twin of a patient is a synchronized and aggregated multimodal data set obtained from a wide range of diagnostic medical equipment which is continuously updated and based on a personalized semantic modal of a patient. Since data are obtained from different medical devices and tools in various formats which directly do not fit for data synchronization and aggregation, the format of a file-wrapper that enables storing time characteristics of medical investigations (time stamps) in an evident form. It allows us to simplify a procedure of multimodal data aggregation while creating and continuous updating the digital twin of a patient. The process of digital twin forming includes the following stages: receiving of original data files in a device format (sonographic device, MRI scanner, electrocardiograph etc.), analysis of data and their time stamps, transformation of the original file to the format of a file-wrapper, data synchronization and aggregation, representation of multimodal data in a digital twin format for further storing and processing.
Keywords: digital twin, multimodal data, data synchronization.
1.А. ДИЧКА
Кшвський полггехшчний шститут iM. 1горя Сжорського
ORCID: 0000-0002-3446-3076 е.е. СУЛЕМА
Кшвський полггехшчний шститут iM. 1горя Сжорського
ORCID: 0000-0001-7871-9806 Ю.В. БУХТ1ЯРОВ
Кшвський полггехшчний шститут iM. 1горя Сжорського
ORCID: 0000-0002-0541-5251
ШФОРМАЦШНА ТЕХНОЛОГ1Я ЦИФРОВИХ ДВ1ЙНИК1В ДЛЯ КОМПЛЕКСНОГО ПОДАННЯ ТА ОБРОБЛЕННЯ Б1ОМЕДИЧНИХ ДАНИХ
У cmammi запропоновано тформацшну технологiю цифрових двшниюв для використання у галузi охорони здоров'я, зокрема, у телемедицит. Основною метою цього до^дження е розроблення концепцИ тформацшног технологи цифрових двшниюв для систем тдтримки прийняття лтарських ршень. Другою метою у даному дослiдженi е анализ ргзноматтних форматiв медичних даних та розроблення пiдходу до синхронгзацИ мультимодальних медичних даних. Запропонований у статтi пiдхiд зробить можливою агрега^ю по^довностей мультимодальних даних, отриманих з широкого спектра медичного дiагностичного обладнання з метою створення «цифрового двтника» пацiента. У статтi проаналгзована можливiсть синхронгзаци та формати подання таких даних, як одноканальнi та багатоканальнi бiологiчнi сигнали, результати лабораторних тестiв, ультразвукового до^дження, магнтно-резонансно'1 томографа тощо.
Технологгя цифрових двшнитв зробить можливим створення нового поколтня медичних систем пгдтримки прийняття ршення. Цифровий двтник пацгента являе собою синхрон1зовану та агреговану сукуптсть мультимодальних даних, отриманих з широкого спектру д1агностичного медичного обладнання, яка постшно оновлюеться та Трунтуеться на персотфжовант семантичнт модел1 патента. Осюльки дат отримуються з медичних пристрогв та приладдя у р1зноман1тних форматах, як безпосередньо не придатш для синхронгзацИ та агрегацИ, у статтг запропоновано формат файлу-обгортки, який дозволяе зберггати часовi характеристики медичних досл1джень (часовi мтки) у явному виглядг, що дозволить спростити процедуру агрегацИ мультимодальних даних при створен та посттному оновленнi цифрового двтника пацiента. Процес формування цифрового двтника включае таю етапи: отримання первинних файлiв даних у форматi пристрою (апарат УЗД, МРТ, елекmрокардiограф тощо), аналiз даних та iх часових мток, перетворення первинного файлу у формат файлу-обгортки, синхронiзацiя та агрегацiя даних, подання мультимодальних даних у формаmi цифрового двтника для подальшого збереження та оброблення.
Ключовi слова: цифровий двтник, мульmимодальнi дат, синхронiзацiя даних.
И.А. ДИЧКА
Киевский политехнический институт им. Игоря Сикорского
ORCID: 0000-0002-3446-3076 Е.С. СУЛЕМА
Киевский политехнический институт им. Игоря Сикорского
ORCID: 0000-0001-7871-9806 Ю.В. БУХТИЯРОВ
Киевский политехнический институт им. Игоря Сикорского
ORCID: 0000-0002-0541-5251
ИНФОРМАЦИОННАЯ ТЕХНОЛОГИЯ ЦИФРОВЫХ ДВОЙНИКОВ ДЛЯ КОМПЛЕКСНОГО ПРЕДСТАВЛЕНИЯ И ОБРОБОТКИ БИОМЕДИЦИНСКИХ ДАННЫХ
В статье предложена информационная технология цифровых двойников для использования в сфере охраны здоровья, в частности, в телемедицине. Основной целью данного исследования является разработка концепции информационной технологии цифровых двойников для систем поддержки принятия врачебных решений. Второй целью в этом исследовании является анализ разнообразных форматов медицинских данных и разработка подхода к решению задачи синхронизации мультимодальных медицинских данных. Предложенный в статье подход позволит агрегировать последовательности мультимодальных данных, полученных с широкого спектра медицинского диагностического оборудования с целью создания «цифрового двойника» пациента. В статье проанализирована возможность синхронизации и форматы представления таких данных, как одноканальные и многоканальные биологические сигналы, результаты лабораторных тестов, ультразвукового исследования, магнитно-резонансной томографии и другие данные.
Технология цифровых двойников сделает возможным создание нового поколения медицинских систем поддержки принятия решения. Цифровой двойник пациента представляет собой синхронизированную и агрегированную совокупность мультимодальных данных, полученных с широкого спектра диагностического медицинского оборудования, которая постоянно обновляется и основывается на персонифицированной семантической модели пациента. Поскольку данные поступают с медицинских приборов и устройств в разнообразных форматах, которые непосредственно не пригодны для синхронизации и агрегации, в статье предложен формат файла-обертки, который позволяет сохранять временные характеристики медицинских исследований (временные метки) в явном виде, что позволит упростить процедуру агрегации мультимодальных данных при создании и постоянном обновлении цифрового двойника пациента. Процесс формирования цифрового двойника включает такие этапы: получение первичных файлов данных в формате устройства (аппарат УЗИ, МРТ, электрокардиограф и т.д.), анализ данных и их временных меток, преобразование первичных файлов в формат файла-обертки, синхронизация и агрегация данных, представление мультимодальных данных в формате цифрового двойника для дальнейшего хранения и обработки.
Ключевые слова: цифровой двойник, мультимодальные данные, синхронизация данных.
Problem Statement
Digital twin technology is a new approach to representing and processing a dynamic digital model of a physical object or process, its past, current and future state and behaviour. According to [1], a digital twin is a set of virtual information constructs that fully describes a potential or actual physical manufactured product from the micro atomic level to the macro geometrical level. It can be important to create a digital twin of a certain physical object to model its possible behaviour, critical states in a future, etc. Digital twin technology was
developed for optimization of business capability in industries, but the key idea of this technology can be applied to other fields. In particular, it can be useful in health care applications, especially in e-health and m-health. Implementation of digital twin information technology into healthcare enables creating a next generation of medical decision support systems based on synchronized and aggregated medical data about a patient's health status. Such advanced medical decision support system will allow a medical doctor to see the overall picture of the patient's health status over time, to understand better dependences between treatment events and treatment results (including side effects on other organs), to search for similar cases happened to the patient in the past, to model possible impact of new treatment schemes, etc.
A digital twin is a virtual model of a real physical object which mirrors all characteristics and functionalities of this object. Unlike of other virtual models, a digital twin is a dynamically updated model which is working in parallel to its physical twin, i.e. the real object. It means that we need to collect data being registered as a result of long-term monitoring of this real object. This meets the case of life-long monitoring of patient's health status, especially for patients with chronical diseases. Thus, the idea of digital twin application in healthcare consists in creation and dynamic update of a patient's individual data set. This individual data set contains synchronized and aggregated multimodal data obtained from various medical investigations and stored with respect to a personalized semantic model of a human body, its organs and systems of organs. This personalized semantic model can also reflect chronic diseases and individual body conditions of the patient.
A human body on the whole as well as its organs and systems of organs can be characterized by many parameters measured by using a wide range of medical equipment. Data we obtain from different medical investigations and tests are multimodal as they describe a certain specific aspect of human organism functioning. To create the digital twin of a physical twin, these multimodal data must be synchronized and aggregated. However, since multimodal data for a digital twin can be obtained by using wide range of devices, tools and instruments which generate multimodal data in specific (sometimes unique) formats, aggregation of multimodal data in the data set of a digital twin appears a non-trivial task and it requires new solutions.
Analysis of Recent Researches and Publications
Since digital twin technology has wide range of applications in engineering, manufacturing, research, medicine, education, there is a growing number of researches on this topic.
Thus, Talkhestani et al. [2] propose an engineering approach to integrate multi-domain engineering models on the Digital Twin based on anchor points of mechatronic components and Product Lifecycle Management platform during the engineering phase. For a successful synchronization, first the anchor points of a mechatronic component must be identified from a data source of a manufacturing cell, where its actual status is reflected. Then, any occurring changes must be adjusted to the shell of a mechatronic component in the production structure on the platform.
Madni at al. [3] present an overall vision and rationale for incorporating digital twin technology into model-based system engineering (MBSE). The authors give a recommendation to make digital twin technology an integral part of MBSE methodology and experimentation testbeds.
Hu et al. [4] propose a method for reduction of computing resources in the information processing center for efficient interactions between human users and physical machines.
Iglesias et al. [5] consider digital twin applications for JET divertor. These applications enable increasing the JET reliability, operational limits, and accuracy of experimental results.
Uhlemann et al. [6] present a concept for the realization of the digital twin contribution to the development of cyber-physical production systems.
Ayani et al. [7] discuss digital twin modelling by using Simumatik3D® library which allows to reflect the geometry, kinematics, logic and interfaces of the digital twin's real analogue.
Modoni et al. [8] present the digital twin conceptual model and discuss the digital twin synchronization challenges and technological solutions. One of challenges relates to managing the real-time and historical data. The authors claim that the technological system supporting the digital twin must be endowed with scalable capabilities that enable to harvest real-time data which can be captured, processed and transformed into significant insights in an efficient manner.
Cai et al. [9] present sensor data integration and information fusion to build digital twins virtual machine tools for cyber-physical manufacturing. The authors discuss the techniques for deploying sensors to capture machine-specific features, and analytical techniques of data and information fusion are presented for modelling and developing digital twins virtual machine tools. The authors claim that the presented technique can be used as a building block for cyber-physic manufacturing development.
Lohtander et al. [10] discuss the main characteristics and the restriction needed to describe Micro Manufacturing Unit model, which is used later on as an element of overall a digital twin. The digital twin consists of a machine, material, method, measurement and modelling fields like a physical world describes it.
As a result of learning mentioned above and other published researches, we can conclude that in the majority of researches there is a lack of attention to data synchronization and aggregation.
Formulation of Research Objective
The primary objective of this research is to develop a concept of digital twin information technology for medical decision support systems. The secondary objective is to analyse various medical data formats and to develop an approach to synchronization of multimodal medical data; this approach will enable aggregation of multimodal data sequences obtained from a wide range of medical diagnostic equipment with the purpose of a patient's digital twin creation.
Presentation of the Main Research Material
To develop a digital twin information technology, we need to analyse types of data sequences to be obtained, presented, processed, stored, transmitted within this technology. Since these data sequences are obtained from a wide range of medical devices and tools used in medical practice, the first task is to analyse the medical equipment in terms of data obtained from these medical devices and tools.
All medical devices and tools can be divided into several groups which include diagnostic equipment, medical laboratory equipment, medical monitors, therapeutic equipment, life support equipment, etc. In this paper, we consider only devices and tools used for diagnostics.
Diagnostic devices and tools can be classified as measuring medical tools (thermometers, sphygmomanometers, glucometers, etc.), laboratory equipment (biochemistry analysers, haematology analysers, etc.), medical imaging equipment (ultrasound, MRI machines, PET, CT scanners, x-ray machines, etc.), functional diagnostic equipment (electrocardiographs, electroencephalographs, electromiographs, rheography devices, spirometers, etc.). These devices and tools can be either digital or non-digital. Measurement obtained by using non-digital tools (e.g. mercury thermometer, manual sphygmomanometer) are usually recorded manually either as paper notes or as electronic notes. Measurements obtained from digital equipment are presented as files of certain formats depending on the type of medical equipment used. The most popular medical data file formats are presented in Table 1.
Table 1
Medical Data File Formats
Data File Type File Extension Main Data Type Investigation Type Availability of Time Stamps
File Extension Format .FEF Biological signal Electrocardiography There is a time sample array measured data section; this section can be absent [11].
BioSemi Data Format .BDF Multichannel biological signals Electroencephalography There are four time-related fields: "Start date of recording", "Start time of recording", "Number of data records" (it equals to -1 if this number is unknown), "Duration of a data record, in seconds" [12].
LabPas HL7 .HL7 Blood test results Haematology analysis There is field "Date/Time of the Observation" [13].
Ultrasound File Format .UFF Ultrasound research data Ultrasound Investigation There is "local_time" field (string value according to ISO 8601) [14].
NIFTI .NII MRI data (imaging data, statistical values, and other data such as any vector, matrix, label set or mesh) Magnetic Resonance Imaging (MRI) The first three dimensions are reserved to define the three spatial dimensions, while the fourth dimension is reserved to define the time points [15, 16].
DICOM .DC3, .DCM, .DIC Medical images and related data Computer Tomography (CT), Positron Emission Tomography (PET), X-Ray investigation, etc. There are fields "Study date" and "Study time" [17].
Comma- Separated Values .CSV Integer values, real values (in ASCII) Spirometry, Thermometry, Sphygmomanometry, etc. There is no predefined time field [18].
Metadata section
"TIME" (4 bytes| Wrapped file type (4 bytes) Length of time tuple (4 bytes)
Time stamps section
Time values tuple (N bytes)
Main section (wrapped data file)
Original data file (N*m)
Fig. 1. A file-wrapper structure
In certain diagnostics procedures, such as endoscopy in general and colonoscopy in particular, where the result of investigation is video stream, general purpose file formats are used.
As we can conclude on medical data formats, most medical data formats predefine metadata field for storing time stamps, however, the structure of the files differs significantly. Thus, there is a need in some upper level file format which can be used as a wrapper for any other formats. The main requirement to such file-wrapper is to present time stamps, which are extracted from the wrapped file content, in evident form easy for further data synchronisation and aggregation.
Let us assume that data, which describe an object of study based on relatively long term of monitoring, are to be received from N sources. These sources can be of any type: a digital device (sensor), an analog tool if data is being received in real-time, or a local storage, a cloud storage, paper records if data has been collected in past. From each data source, we obtain a data sequence of a certain modality. It is evident that the timeline of data generation (recording) differs for each modality. For example, if we deal with data of life-long monitoring of a patient's health status, then we can face with a case when there are the following results of medical investigations obtained in different days, months, or even years: MRI, ultrasonic, cardiogram, blood tests, temperature records, blood pressure records. Some of these investigation results can be paper records, e.g. temperature and blood pressure records. It is important that each record (both digital and non-digital) is accompanied with time record defining the date and sometimes the time when this record is taken. Besides, in a digital twin application, we can deal with data records are being taken in real-time, simultaneously with their processing in the application.
In any case, from each source, we obtain a multi-image [19, 20, 21] as follows:
h = [TWWLM)ïL1i
(1)
where tj is a time value ( tj 6 T); d[ is a data value of a certain modality (d[ 6 Mj);
T is time values set; Mj is data set of./-modality;
j=[1...N].
Thus, we can establish a direct relation between and d . However, in most practical cases, each data sequence (dj1 is arranged as a file of a certain format which depends on a particular source of data. Such a file is characterized by time of its creation or modification, but in general case either a file does not keep any time stamps of data values or these values can be extracted by using a certain format-specific procedure. To simplify the multi-image processing, we can wrap a file of any type in a file-wrapper (Fig. 1), which includes three sections: a metadata section, a time values section, and a main section where the wrapped file is stored. The metadata section takes 12 bytes and contains the keyword "TIME", an identifier of the file type (e.g. "nii" for NIFTI file format, "dcm" for DICOM file format, etc.), and a size of the time values tuple. The time stamps section contains time values placed in ascending order. The main section contains the wrapped file as a whole; this file is ready for using, e.g. for data reproduction.
Such wrapping enables unifying synchronization procedure of data sequences obtained from different sources (devices, tools, records, etc.).
A general scheme of multimodal data synchronization and aggregation is shown on Fig. 2.
Fig. 2. Multimodal data aggregation as a multi-image
The approach to multimodal data synchronization is based on assumption that we can temporarily store Ni values of i-modality in the Data Collecting module. Let us consider the case when we have data being registered by two sensors. These sensors are independent, and they generate data with frequency f1 and f correspondingly.
Let the first sensor generate the date sequence (tuple) d1 = within the timeline
t1 = (tg, i* tl, £5, tg} as it is shown on Fig. 3a. Let also the second sensor generate the date sequence (tuple) d1 = (df, df} within the timeline t2 = (tf, tf, tf,} as it is shown on Fig. 3b. Then the task is to synchronize these data sequences according to the Digital Twin timeline (Fig. 3c).
It is important that the local time of an investigation can differ from a global time of the digital twin. Besides, if investigation events within the patient's health status life-long monitoring can be carried out in different time zones. Another problem in data synchronization can be related with time skew caused with data transfer delays which can appear in e-health and m-health applications. These reasons require development of mathematical models [22] for different cases of multimodal data synchronization in digital twins.
t t
tj
"Î
• • w (a)
t22 t23
«22
• • (b)
I I t3 t4 l5 4 1 I 6
"Í
w - -•- w
Timeline of sensor 1
Data registration events of sensor 1
Timeline of sensor 2
Data registration events of sensor 2
Timeline of Digital Twin
Digital Twin's data of modality 1 Digital Twin's data of modality 2
(c)
Fig. 3. Data synchronization principle: (a) Data Sequence from Sensor 1; (b) Data Sequence from Sensor 2; (c) Multi-Image Data of Digital Twin
The data synchronization and aggregation procedures are essential part of the digital twin information technology which enable further processing of compound multimodal data structures.
The general concept of the information technology for a patient's digital twin creation is presented in Fig. 4. The first stage of the technology is to obtain data about the biomedical object. This data can be obtained from various diagnostic medical equipment, such as medical devices, medical tools, laboratory equipment, medical records. The data can be of both digital and non-digital nature. Digital data are presented in various file formats depending on a medical device (sensor) used for measurements.
Fig. 4. Digital twin creation scheme
Since data files obtained on the first stage differ in their structure, the next stage is to convert every file into the universal format which contain time stamps in a predefined format proposed in the previous section. This conversion is carried out as wrapping an initial file by high level metadata which includes time stamps section. As a result, timed file is obtained for each initial file of a certain type. Time stamps can be defined in several ways: they can be extracted from the initial file metadata or they can be added manually.
The next stage is data synchronization and aggregation. The synchronization procedure uses time stamps available in metadata of timed files. Synchronized data sequences are presented as aggregated data structure called multi-image [19, 20, 21]. The obtained multi-image is a complex representation of the corresponding digital twin data.
To create the digital twin, we need to know a semantic model of a corresponding physical twin, i.e. the patient. This semantic model is supposed to be based on medical standards such as [23]. The personalized semantic model of the patient is initial data for this information technology, along with data to be obtained from diagnostic medical equipment. The synchronized and aggregated data is stored in conformity with the semantic model. The data representation model can include several levels [24].
Both data processing and reproduction of the digital twin are based on external requests of information retrieval, analysis, modelling, prediction, etc. depending on specific tasks of the patient's health status monitoring and investigation.
These main stages of the proposed information technology can be implemented as corresponding components of an advanced medical decision support system.
Conclusions
Implementation of the digital twin concept into healthcare field creates an opportunity for development of the next generation medical decision support systems. These new systems are to be based on personalized semantic models and synchronized and aggregated multimodal medical data sets. These new key elements to be added due to application of the proposed digital twin information technology will enable more accurate, flexible and time-wise manipulation with patient's health status data being obtained during long-term (life-long) medical monitoring. In its turn, it creates favourite conditions for easier, deeper and more accurate medical diagnostics.
The proposed approach enables multimodal data collection, synchronization and aggregation. The data sequences are collected from digital sensors of different types. The purpose of aggregation is obtaining a multiimage which as a data model for the biomedical object's digital twin.
The range of potential applications for digital twin technology in healthcare is quite wide and includes e-health and m-health. In particular, it can be useful for Hospital-at-Home applications for chronical patients.
References
1. Michael Grieves, John Vickers. Digital Twin: Mitigating Unpredictable, Undesirable Emergent
Behavior in Complex Systems. Transdisciplinary Perspectives on Complex Systems, 2018, pp. 85-113.
doi: 10.1007/978-3-319-38756-7 4
2. Behrang Ashtari Talkhestani, Nasser Jazdi, Wolfgang Schlögl, Michael Weyrich. Consistency check to synchronize the Digital Twin of manufacturing automation based on anchor points. Proceedings of the 51st CIRP Conference on Manufacturing Systems, 2018, pp. 159-164. doi: 10.1016/j.procir.2018.03.166
3. Azad M. Madni, Carla C. Madni, Scott D. Lucero. Leveraging Digital Twin Technology in ModelBased Systems Engineering. Systems, vol. 7, no. 1, p. 7, Jan. 2019. doi: 10.3390/systems7010007
4. Liwen Hu, Ngoc-Tu Nguyen, Wenjin Tao, Ming C.Leu, Xiaoqing Frank Liu, Md Rakib Shahriar, S M Nahian AI Sunny. Modeling of Cloud-Based Digital Twins for Smart Manufacturing with MT Connect. Procedia Manufacturing, Vol. 26, 2018, pp. 1193-1203. doi: 10.1016/j.promfg.2018.07.155
5. D. Iglesias, P. Bunting, S. Esquembri, J. Hollocombe, S. Silburn, L. Vitton-Mea, I. Balboa, A. Huber, G.F. Matthews, V. Riccardo, F. Rimini, D. Valcarcel. Digital twin applications for the JET divertor. Fusion Engineering and Design Journal, Vol. 125, 2017, pp. 71-76. doi: 10.1016/j.fusengdes.2017.10.012
6. Thomas H.-J. Uhlemann, Christian Lehmann, Rolf Steinhilper. The Digital Twin: Realizing the Cyber-Physical Production System for Industry 4.0. Proceedings of the 24th CIRP Conference on Life Cycle Engineering, 2017, pp. 335-340. doi: 10.1016/j.procir.2016.11.152
7. M. Ayani, M. Ganebäck, Amos H. C. Ng. Digital Twin: Applying emulation for machine reconditioning. Proceedings of the 51st CIRP Conference on Manufacturing Systems, 2018, pp. 243248. doi: 10.1016/j.procir.2018.03.139
8. Gianfranco E. Modoni, Enrico G. Caldarola, Marco Sacco, Walter Terkaj. Synchronizing physical and digital factory: benefits and technical challenges. Proceedings of the 12th CIRP Conference on Intelligent Computation in Manufacturing Engineering, 2018, pp. 472-477. doi: 10.1016/j.procir.2019.02.125
9. Yi Cai, Binil Starly, Paul Cohen, Yuan-Shin Lee. Sensor data and information fusion to construct digital-twins virtual machine tools for cyber-physical manufacturing. Proceedings of the 45th ME North American Manufacturing Research Conference, 2017, pp. 1031-1042. doi: 10.1016/j.promfg.2017.07.094
10. Mika Lohtander et al. Micro Manufacturing Unit and Corresponding 3D Model for the Digital Twin. Proceedings of the 8th Swedish Production Symposium, 2018, pp. 55-61. doi: 10.1016/j.promfg.2018.06.057
11. Värri, A, ENV 14271, File Exchange Format for Vital Signs and its use in digital ECG archiving. Proceedings of 2nd Open ECG Workshop "Integration of the ECG into the EHR & Interoperability of ECG Device Systems", Berlin, Germany, 2004, 2 p.
12. Which file format does BioSemi use? Available at: https://www.biosemi.com/faq/file_format.htm (accessed 21 September 2019).
13. Specification for the HL7 Lab Data Interface, Oracle® Health Sciences LabPas Release 3.1, Part Number: E48677-01, 2013, 39 p.
14. Olivier Bernard et al. The Ultrasound File Format (UFF) - First draft. Proceedings of 2018 IEEE International Ultrasonics Symposium, 2018. doi: 10.1109/ULTSYM.2018.8579642
15. The NIFTI file format. Available at: https://brainder.org/2012/09/23/the-nifti-file-format (accessed 21 September 2019).
16. Data Format Working Group (DFWG), The NIFTI-1 DATA FORMAT, 2004, 30 p. Available at: https://www.nitrc.org/docman/view.php/26/204/TheNIITI1Format2004.pdf (accessed 21 September 2019).
17. Digital Imaging and Communications in Medicine. Available at: https://www.dicomstandard.org/ (accessed 21 September 2019).
18. CSV Files. Available at: https://people.sc.fsu.edu/~jburkardt/data/csv/csv.html (accessed 21 September 2019).
19. I. Dychka, Ye. Sulema, Logical Operations in Algebraic System of Aggregates for Multimodal Data Representation and Processing. Research Bulletin of the National Technical University of Ukraine "Kyiv Polytechnic Institute", Vol. 6, 2018, pp. 44-52. doi: 10.20535/1810-0546.2018.6.151546
20. I. Dychka, Ye. Sulema. Ordering Operations in Algebraic System of Aggregates for Multi-Image Data Processing. KPI Science News, Vol. 1, 2019. doi: 10.20535/kpi-sn.2019.1.157245
21. Sulema, Ye. ASAMPL: Programming Language for Mulsemedia Data Processing Based on Algebraic System of Aggregates. Advances in Intelligent Systems and Computing, Springer, Vol. 725, 2018, pp. 431-442. doi: 10.1007/978-3-319-75175-7_43
22. Yevgeniya Sulema, Etienne Kerre. Multimodal Data Representation and Processing Based on Algebraic System of Aggregates, preprint, 2019, 37 p.
23. International Classification of Functioning, Disability and Health. Available at: https://www.who.int/classifications/icf/en (accessed 21 September 2019).
24. Yevgeniya Sulema, Ivan Dychka, Olga Sulema. Multimodal Data Representation Models for Virtual, Remote, and Mixed Laboratories Development. Lecture Notes in Networks and Systems, Springer, Vol. 47, 2018, pp. 559-569. doi: 10.1007/978-3-319-95678-7_62
