CC BY
28
Research article
DOI: https://doi.org/10.18721/JCSTCS.16202 UDC 62-503.55
CLOUD DISTRIBUTED CONTROL SYSTEM BASED ON OPEN PROCESS AUTOMATION PLATFORM
Potekhin' Q , A.P. Alekseev2, E.V. Kuklin3, Ya.D. Khitrova4, Yu.N. Kozhubaev5
1,2,3,4,5 Peter the Great St. Petersburg Polytechnic University,
St. Petersburg, Russian Federation
Abstract. The article shows the relevance of using cloud technologies in the field of industrial automation of technological processes. Typical architectures of modern automated control systems, as well as new standards and approaches to the design of industrial control systems developed by international communities are analyzed. A prototype of an open cloud distributed control system based on IEC 61131 has been demonstrated. The dependences of the computing power of virtual controllers on the number of processed objects are given. The Open Process Automation initiative aims to enhance the full lifecycle benefits of industrial control systems through the use of a standards-based, open, secure, interoperable architecture and open business model. The OPAS standard based on this initiative uses a "standard of standards" approach.
Keywords: Industry 4.0, OPAS, Cloud DCS, Cloud computing, Internet of Things
Citation: Potekhin V.V., Alekseev A.P., Kuklin E.V., et al. Cloud distributed control system based on open process automation platform. Computing, Telecommunications and Control, 2023, Vol. 16, No. 2, Pp. 17-28. DOI: 10.18721/JCSTCS.16202
© Potekhin V.V., Alekseev A.P., Kuklin E.V., et al. 2023. Published by Peter the Great St. Petersburg Polytechnic University
Научная статья
DOI: https://doi.org/10.18721/JCSTCS.16202 УДК 62-503.55
ОБЛАЧНАЯ РАСПРЕДЕЛЕННАЯ СИСТЕМА УПРАВЛЕНИЯ НА БАЗЕ ОТКРЫТОЙ ПРОМЫШЛЕННОЙ ПЛАТФОРМЫ АВТОМАТИЗАЦИИ
В.В. Потехин' Q , А.П. Алексеев2, Е.В. Куклин3, Я.Д. Хитрова4, Ю.Н. Кожубаев5
1,2,3,4,5 Санкт-Петербургский политехнический университет Петра Великого,
Санкт-Петербург, Российская Федерация
Аннотация. Сегодня массовое сотрудничество в сфере разработки программного обеспечения и открытых архитектур меняет фундаментальную структуру бизнеса и перестраивает методы работы организаций в условиях жесткой конкуренции. В статье показана актуальность использования облачных технологий в сфере промышленной автоматизации технологических процессов. Разобраны типовые архитектуры современных АСУ ТП, а также разрабатываемые международными сообществами новые стандарты и подходы к проектированию промышленных систем управления. Продемонстрирован прототип открытой облачной распределенной системы управления на базе МЭК 61131. Приведены зависимости вычислительной мощности виртуальных контроллеров от количество обрабатываемых объектов. Инициатива Open Process Automation направлена на улучшение всех преимуществ жизненного цикла промышленных систем управления благодаря использованию основанной на стандартах, открытой, безопасной, совместимой архитектуры и открытой бизнес-модели. Стандарт OPAS на базе этой инициативы использует подход «стандарт стандартов». В рамках тестирования архитектуры в соответствии с OPAS был сделан ряд выводов об использовании технологии.
Ключевые слова: Индустрия 4.0, OPAS, облачная РСУ, облачные вычисления, АСУ ТП, интернет вещей, архитектура автоматизации
Для цитирования: Potekhin V.V., Alekseev A.P., Kuklin E.V., et al. Cloud distributed control system based on open process automation platform // Computing, Telecommunications and Cont trol. 2023. Т. 16, № 2. С. 17-28. DOI: 10.18721/JCSTCS.16202
Introduction
Today, massive collaboration in software development and open architectures is changing the fundamental structure of business and reshaping the way organizations operate in a highly competitive environment. Collaboration fueled by open methodologies and peer-to-peer production, is forcing management to rethink their strategies. Organizations that previously built proprietary systems are beginning to develop open-source products and creating public foundations where everyone can develop and contribute to push the boundaries of their business as well as the boundaries of the industries in which they operate.
It is important to note that at the moment, the development under the Open Source concept is far advanced in the field of information technology (IT), but in the field of industrial automation is still dominated by the proprietary segment. Companies are reluctant to disclose their code base, mainly the flagships of programmable logic controllers (PLC, PLC) and SCADA have closed interfaces, which do not easily create a synergy of equipment from different manufacturers, forcing companies to use the products of one company [1].
The lag of operation technologies (OT) from IT is caused by the fact that the development of open solutions is not fully supported by large companies and spheres. Open Source developers are not only small
© Потехин В.В., Алексеев А.П., Куклин Е.В. и др. 2023. Издатель: Санкт-Петербургский политехнический университет Петра Великого
startups looking for new revolutionary solutions. These are companies with billions of dollars: Google, Apple, Facebook, Amazon, releasing their products with open source code, available for modification to each individual developer.
To develop OT at the same speed as IT, it is necessary to move to the concept of an open industrial platform, where development is carried out jointly with developers interested in improving products, free competition, and rapid implementation of new technologies in existing facilities. Previously, the IT sphere borrowed the concept of lean manufacturing and developed the DevOps methodology for continuous product improvement. In the OT-sphere it is necessary to borrow the Open Source concept as one of the options for the effective implementation of new tools and technologies in the field of industrial automation. The Open Process Automation Forum (OPAF) is addressing this challenge.
The article discusses new methods for solving process control problems, as well as the results of their application on assembled laboratory benches.
Open industrial automation platform standards
Open Process Automation is an industry initiative aimed at improving the full lifecycle benefits of industrial control systems through the use of a standards-based, open, secure, interoperable architecture and open business model. The Open Process Automation Forum of The Open Group is the primary beneficiary. As of July 2022, OPAF consists of 800 member organizations, most major distributed control system (DCS) vendors, many hardware and software vendors, and system integrators [2].
The OPAF standards and architecture implement the new Industry 4.0 concepts of Cloud, Edge, and Field computing, while allowing legacy management systems to be connected. The standards are generic guidelines for the design of process control systems in the new paradigm.
Quality attributes have been defined as goals for the Open Process Automation Standard (OPAS):
♦ Interoperability,
♦ Modularity,
♦ Scalability,
♦ Securability,
♦ Reliability,
♦ Portability,
♦ Affordability,
♦ Availability,
♦ Discoverability,
♦ Evolvability,
♦ Manageability,
♦ Compatibility,
♦ Configurability,
♦ Discoverability,
♦ Usability,
♦ Flexibility,
♦ Testability,
♦ Reusability,
♦ Traceability.
The attributes of interoperability and portability are basic compared to currently available commercial RSCs and PLCs.
A "standard of standards" approach is used to define the standard. OPAF has interoperability agreements that allow information to be exchanged before publication with many organizations, including NA-MUR, ZVEI, and PL Copen.
1. Architecture
Fig. 1. OPAS architecture
Figure 1 [10] shows the target system architecture according to OPAS requirements, consisting of:
— OT Data Center (on premise) combines DCNs directly involved in managing equipment through a physical connection;
— OT Data Center (external) includes remote device management (DCNs) components;
— Enterprise IT Data Center is responsible for organizing enterprise business logic and communicating with enterprise-level objects;
— OPAS Connectivity Framework (shared data bus) and control structures, including the Global Discovery Server, is a set of services enabling the interaction of network components in accordance with the OCF standard;
— DCN (Distributed Control Node) is a distributed control node, which is a service or physical device capable of connecting to a shared OCF bus;
— ACP (Advanced Computing Platform) is a process control platform.
1.1 Distributed Control Node
The main task of the DCN executor is the direct control of the device involved in the production of the "controlled device". The application that controls the device has a 4-level structure [3], shown in Figure 2 [11].
The DCN application uses the FILO format, a four-layer structure where:
— Layer F, the application configuration layer, is the program itself, written in the Java programming language or IEC 61131/ IEC 61499;
— Layer I is the runtime layer which reproduces the configuration;
— Layer L is the layer of interaction with external systems, the interface;
— Layer O is the operating system layer.
1.2 Shared Data bus
OPAS Connectivity Framework is responsible for network integrity, reliable data transfer, authorization, authentication, and monitoring of devices on the network. Includes Global Discovery Server [4]. It uses a system to maintain service discovery, for example, Consul, a specification that is used to manage the server hardware, RedFish.
1.3 Advanced computing platform
ACP is responsible for high-level process control related to coordination of different DCN performers. It is also involved in short-term forecasting of the production process, for which it is equipped with a database and additional computing power.
1.4 Enterprise IT Data Centers
Function Block
Diagramming tool
Function Block Diagram (compiled)
Function Block
Execution Engine
Connectivity,
Storage,...
OS;Tasks, Schedule,
Sockets...
Configuration Editina s
Java editor
Configuration Application system ~ Management Management Management Services Services Services
Layer-F Oownload/UploadJ
Java Byte
Layer-F Applications I Coi Code
..— Layer-J Application:
—i»,. Java Runtime
— Layer-L Applications/se Environment
S fig 5 5Li t-
OC F Networking Infrastructure
s - O-PAS Version 1 Specifications Red Arrows - Future Interface Specifications
Fig. 2. Application structure of DCN
Enterprise IT Data Centers has the largest computing power and is responsible for the primary accounting of equipment, collecting heterogeneous technological information, aggregation, forecasting and reporting. [5] From this component, information about the composition of the installed equipment, the assignment of IP-addresses to network devices, the locations of devices is taken. This is also where the log collector is located.
2. Prototype of Cloud-based DCS
2.1 Prototype of ExxonMobil
ExxonMobil has developed a prototype that meets the quality attributes described earlier in this article. The essence of the experiments they performed on the prototype are described in the figure below.
Interoperability is the ability of two or more systems/components to exchange and use information. This was demonstrated by combining components from 10 different vendors into a single system. For example, three components were used to perform a simple PID control cycle: one component provided input, another component performed PID calculation, and the third component sent output to the field.
Interchangeability is the ability to replace one component with another while the system continues to operate. Basic I/O and regulatory control were originally performed in an IEC 61499 environment on a Raspberry Pi device. This device was replaced with a DCN provided by Intel.
Configuration portability is the ease with which configuration information can be exported from one application, imported, and then deployed to another application. The control function was originally run in an IEC 61499 runtime environment provided by 4DIAC, which is an open source implementation of the IEC 61499 standard. The configuration of this application was exported and imported into another IEC 61499 environment. Finally, the information was deployed and successfully implemented in another execution engine provided by NXTControl, which is a commercial implementation of IEC 61499.
Application portability is the ease with which an application running in one environment can be reassigned and deployed to another environment. A simple PID control algorithm was run on an edge in a DCN provided by Intel. The controller was reassigned and deployed in an NXTControl environment running in ACP.
2.2 Structured description of prototype
We managed not only to repeat the experiments described above, but also to carry out high-load tests of a physical DCN and a virtual DCN.
A schematic of the prototype and scenarios are shown in Figure 4.
Fig. 3. Checking quality attributes
Fig. 4. Architecture of platform prototype
The stand was based on an open IaaS platform used as an ACP. Soft-PLC, virtual controllers running on Linux OS, were used as DCNs.
In our case the evaluation of quality attributes was as follows:
— Interoperability was demonstrated by combining components from several different vendors into a single system. For example, a virtual DCN took over the function of PID control, while the other two physical DCNs from different vendors were responsible for analog and discrete signals.
— Interchangeability: a physical DCN was replaced by a virtual DCN with the same program. It also shows the attribute of application portability.
We were able to apply a gateway to connect non-OPAS devices to the shared bus. It was also possible to test the possibility of virtual NDP-code debugging on the virtual controller and model with further loading into the physical DCN.
2.3 Test results
The results of the prototype testing are presented in this paragraph. Testing involved determining and comparing groups of characteristics by performing the checks listed below:
Fig. 5. Common diagram of test
— High-load testing.
— Testing of cloud NMC for signal throughput rate.
— Cloud NMC backup in the cloud.
Test 1 — High-load testing
This test verifies compliance with virtual DCN target function requirements by establishing and comparing functionality and characteristics.
The user sets setpoints/process variables (SV) of PID controllers from the web SCADA (web-SCADA). The cloud distributed control system (CDCS) controller implements PID control loops, which supply control actions — control of valve opening percentage. The calculated percentage of valve opening in the controller is transmitted to the objects in the model via IloT-gateway over the OPC UA protocol. The number of PID controllers is about 100. In addition, the current values of analog sensors and the forecast value are displayed.
User can also change the mode of PID-controllers (AUT-MAN-CASCAD), set values to the valve output in the manual PID-controller mode, and change PID-controller coefficients from web SCADA (web-SCAD).
This test allows you to set the maximum loads on the cloud-based DCS.
Tests were conducted on a virtual machine with the following computing characteristics:
— CPU: Intel Xeon Processor (Cascadelake) 2.7 GHz 1 Core;
— RAM: 2 GB;
— ROM: 20 GB;
— Network adapter: 10GbE Intel C622, 1000 Mb/s.
Test Order:
1. The telemetry values are read without load.
2. Next, the projects were loaded into the cloud-based DCS controller for 50, 100, 150, and 200 sensors in turn, and then telemetry readings were taken for each option.
3. We added node power to 2 CPUs and up to 4 GB of RAM, and then performed steps 1—2 again.
Examples of test results are shown in Figures 6—8.
CPU load was unloaded as a percentage.
RAM load was loaded in absolute ratio.
Average cycle time was calculated in absolute value:
Cycle time = K * x,
where K is adjustment factor, x is the number of sensors/objects
As a result of the test, the readings of telemetry without load, as well as with load at 50, 100, 150 and 200 sensors were measured. As the number of sensors increases, the telemetry readings increase linearly.
Fig. 6. Load CPU
Fig. 7. Load RAM
Fig. 8. Average cycle time
When processing 200 sensors with predictive analytics, there is a 120 ms deviation in controller cycle time, which is 120%. With this virtual machine configuration, no more than 150 sensors can be effectively controlled. After increasing the computing power of the virtual machine on which the runtime environment is deployed, the behavior of the node did not change: when the number of sensors is increased, the telemetry readings grow linearly.
Test 2. OPC-testing for signal throughput rate
In this test we check the speed of signals passing through the full route (SCADA — DCN — IIoT-gate-way — Output Device — Input Device — IIoT-gateway — DCN — SCADA).
This test verifies compliance with the virtual DCN's target function requirements by establishing and comparing functionality and features: Signal Delay Times. Signal Delay Times are fixed as follows:
— During the test, the input and output signals are recorded in the database and plotted in Web-SCADA.
— After the test passes, the csv file with the time stamps and the corresponding values on the input and output signal plots should be uploaded.
— The signal delay is calculated by the difference of the input and output signal timestamps.
Fig. 9. Common diagram of test 2
Testing of SCADA system for performance and quality of processing of input analog and discrete signals received via physical communication channels:
1. Open SCADA system in the browser, "Speed" tab.
2. Start generation of signals in the form of a sine wave with a period of 10, 60 and 100 seconds.
As a result of the test, the performance and processing quality of the input analog and discrete signals received via physical communication channels were tested. The average delay for all signals was 108 ms. Generation of signals from the signal setter is displayed in Web-SCADA without data loss. The delay was calculated using the following formula:
Delay = T - T ,
J in out1
where T is input signal time, Tout is output signal time.
Test 3. OPC redundancy in the cloud
Web-SCADA and soft-PLC were created in the MasterSCADA development environment. Master-SCADA allows you to organize the redundancy of the runtime system. The program is loaded to the controller copy, executed in parallel without I/O access, while the main (master) controller is running. As soon as the failure occurs in the main controller, the system switches to the second (standby) controller without significant delays and suspension of the programs execution with duplication of the current states in the previous controller. After control transition, the standby controller is not initialized with initial values; it contains the same values as the primary controller.
This test demonstrates the results of redundancy, both the hardware part (disconnecting the compute node from power) and the software part (disconnecting the virtual machine). As the virtual machine is deployed on one of the compute nodes, its shutdown will be the same as that of the compute node.
Testing of the system for redundancy is done as follows: on the redundancy tab, there is a counter, which is incremented every half a second. After starting the counter you need to shut down virtual machine/computing node, where virtual controller is deployed.
When incrementing the counter, all values are written to the database on the local controller (each to its own).
After the main controller shutdown, control went to the backup controller's virtual machine. The standby controller started writing values from 66, which suggests that about 6 values were lost during the switchover. This amounts to about 1.5 seconds (2 values once per second).
Results of the test:
— MasterSCADA allows implementing software redundancy. Computing nodes implement hardware redundancy (redundant virtual machines are located on different computing nodes).
— Switching delay was about 1.5 seconds.
— MasterSCADA does not record the data in the backup archive, until the moment of switching controllers.
Software, developed on the basis of the IEC 61131 standard [6], as MasterSCADA in the prototype of OCSADA, in recent years is criticized because of non-compliance with modern methods of software de-
Fig. 10. Signal generation
velopment. Current IEC 61131-3 [7] compliant software architectures for industrial process measurement and control systems do not conceptually support reconfiguration and distribution. On the other hand, according to the articles "Design and Execution Issues in IEC 61499 Distributed Automation and Control Systems" [8] and "Is IEC 61499 in Harmony with IEC 61131-3" [9] identified as requirements for future automation systems:
Portability. Software tools can open and correctly interpret program components and system configuration that were developed in other software tools.
Configurability. Any device and its software components can be configured by software tools from multiple vendors.
Interoperability. Distributed devices can communicate with each other to perform the functions of a distributed control system.
Reconfigurability. Software tools can change software and hardware configuration while the device is running.
Distributability. Software components can be distributed to different devices in the system.
To address the limitations as well as new challenges in the development of industrial automation systems, the technical committee of the International Electrotechnical Commission (IEC) was tasked with developing a new standard. The standard was named IEC 61499.
Conclusion
The Open Process Automation initiative aims to improve the full lifecycle benefits of industrial control systems through the use of a standards-based, open, secure, interoperable architecture and an open business model. The OPAS standard, based on this initiative, takes a "standards standard" approach. As part of the OPAS architecture testing, the following conclusions were drawn.
For test 1 on the high-load test, telemetry readings were measured without load as well as with load when 25, 50, 75, and 100 PID controllers were operating. As the number of PIDs increases, the telemetry readings increase linearly. At the same time, it was proven possible to dynamically change the maximum computing power on the virtual machine, thereby allowing the virtual controller to load more objects. The 25 controls increase the RAM fill by about 20 MB, the CPU load by 10%. CPU load increases quite quickly when you increase the number of PID controllers, it may be worth increasing the number of cores in the virtual machine with a multiple increase in objects. With this configuration the virtual controller will run 200 PID controllers.
For test 2 on the speed of signals, we tested the performance and processing quality of the input analog and discrete signals received through the physical communication channels. The average delay
for all signals was 108 ms. Signal generation from the signal setter was displayed in Web-SCADA without data loss.
For test 3 on redundancy of the OPC in the cloud, the processes occurring when one of the nodes is powered down were investigated. The switching of the control servers occurs seamlessly for the system components. When power fails on the control node, control shifts to the backup node; delays are observed only in the operation of the web interface of the IaaS platform. If the control server is restarted, the web-interface crashes with an error until the server reboots. MasterSCADA allows you to implement software redundancy. Compute nodes implement hardware redundancy (redundant virtual machines are on different compute nodes).
OPAS includes standards that result in portability and interoperability. One such standard is IEC 61499. The IEC 61499 standard is part of the OPAS architecture. The requirements of this standard for the control system are the same as those for OPAS. This standard mainly differs from the IEC 61131 standard in the way functional blocks are processed. In IEC 61499 the processing of functional blocks is event-driven: event inputs and outputs are added to the interface of the functional block.
As a result of the work, new methods and technologies for solving process control problems on the basis of the assembled laboratory bench were considered. Tests were conducted to show how the process can change with the application of new control approaches and what the benefits of their use are.
REFERENCES
1. Church P., Mueller H., Ryan C., Gogouvitis S.V., Goscinski A., Tari Z. Migration of a SCADA system to IaaS clouds — a case study. Journal of Cloud Computing: Advances, Systems and Applications, 2017, pp. 6—12.
2. Bartusiak R.D., Bitar S., L.DeBari D., Houk B.G., Heaton M., Strebel R., Stevens D., Fitzpatrick B., Sloan P. Open Process Automation: A standards-based, open, secure, interoperable process control architecture, 2020, pp. 2—3.
3. Lizhi Wang, Yanuyan Qin, Zefan Tang, Peng Zhang P. Software-Defined Microgrid Control: The Genesis of Decoupled Cyber-Physical Microgrids. IEEE Open Access J. Power Energy, 2020, pp. 173—182.
4. Wenbin D., Laurynas R., Peng W., Valeriy V., Xinping G. A Cloud-Based Decision Support System for Self-Healing in Distributed Automation Systems Using Fault Tree Analysis. IEEE Transactions on Industrial Informatics. 2018, pp. 989-1000.
5. Vrba P., Tichy P., Marfk V., Hall K.H., Staron R.J., Maturana F.P., Kadera P. Rockwell Automation's Holonic and Multiagent Control Systems Compendium. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews. 2011, Vol. 41, pp. 14-30.
6. International Electrotechnical Commission. IEC International Standard IEC 61131-3: Programmable Controllers, Part 3: Programming Languages, IEC. 2003.
7. Otto, Hellmans K. IEC 61131: A general overview and Emerging trends. IEEEInd. Electron. Mag. 2009, Vol. 3, pp. 27-31.
8. Strasser T., Zoitl A., Christensen J.H., Sunder C. Design and Execution Issues in IEC 61499 Distributed Automation and Control Systems. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews. 2011, Vol. 41, no. 1, pp. 1-2.
9. Zoitl T., Strasser C., Sunder Baier T. Is IEC 61499 in harmony with IEC 61131-3? IEEE Ind. Electron. Mag. 2009, Vol. 3, pp. 49-55.
10. Khokhlovskiy V., Oleynikov V., Kostenko D., Onufriev V., Potekhin V. Modernisation of a Production Process Using Multicriteria Optimisation Logic and Augmented Reality. In: Proceedings of the 30th DAAAM International Symposium, DAAAM International, Vienna, Austria, 2019, pp. 0500-0507.
11. Woo E.H.C., White, P., Lai C.W.K. Ergonomics standards and guidelines for computer workstation design and the impact on users' health - a review. Ergonomics, 2015, Vol. 59 (3), pp. 464-475.
12. Bucur P.A., Armbrust P., Hungerlander P. On the propagation of quality requirements for mechanical assemblies in industrial manufacturing. Expert Systems with Applications. Klagenfurt, Austria, 2019.
13. Ximing L. Configuration, Programming, Implementation, and Evaluation of Distributed Control System for a Process Simulator. Electronic Thesis and Dissertation Repository of The University of Western Ontario. Ontario, Canada, 2015.
14. Katalinic B., Eliseev A., Breido I., Bobryakov A., Kabanov A., Khomchenko V., Potekhin V., Stazhkov S., Filaretov V. Experience of application of network technologies in engineering education. EAI Endorsed Transactions on Energy Web, 2018, Vol. 5(16).
15. Fedorov A.V., Zobnin S.S., Potekhin V.V. Prescriptive analytics in distributed automation systems, in Proceedings of Symposium on Automated Systems and Technologies, Hannover, PZH Verlag, 2014, pp. 43-49.
INFORMATION ABOUT AUTHORS / СВЕДЕНИЯ ОБ АВТОРАХ
Vyacheslav V. Potekhin Потехин Вячеслав Витальевич
E-mail: [email protected] https://orcid.org/0000-0001-9850-9558
Anton P. Alekseev Алексеев Антон Павлович
E-mail: [email protected]
Egor V. Kuklin Куклин Егор Вадимович
E-mail: [email protected]
Yana D. Khitrova Хитрова Яна Дмитриевна
E-mail: [email protected]
Yury N. Kozhubaev Кожубаев Юрий Нургалиевич
E-mail: [email protected]
Submitted: 20.04.2023; Approved: 25.06.2023; Accepted: 06.07.2023. Поступила: 20.04.2023; Одобрена: 25.06.2023; Принята: 06.07.2023.
