26Система сообщений для брокера в маршрутизации сервера RabbitMQ для использования его в служебной шине предприятия
Гладун Анастасия Михайловна,
магистр прикладной математики и информатики, ведущий программист ООО «АТОН», [email protected]
В качестве системы сообщений брокера в статье будет рассмотрен сервер RabbitMQ. Рассматриваемый сервер принимает сообщения от отправителей и отправляет обратно подтверждение получения сообщения, после чего сообщение перенаправляется подписчикам системы. Подписчики подтверждают получение сообщения или уведомляют о недоставленном сообщении. В качестве альтернативы сообщение может находиться в фазе очереди на передачу до тех пор, пока это сообщение не будет доставлено адресату. После того, как сообщение будет доставлено адресату, оно будет автоматически удалено из системы. Основным преимуществом сервера RabbitMQ является его гибкость в маршрутизации сообщений между разными подписчиками и получателями. В качестве наглядного примера будут рассмотрены способы отправки одного и того же сообщения трем получателям. Это сообщение поступает на узел, который распределяет его на 3 одинаковых сообщения и ставит в три очереди для получателей, которым оно будет доставлено.
В статье автор исследовал возможности брокера сообщений RabbitMQ для использования в служебной шине предприятия.
Ключевые слова: системы, брокер сообщений, RabbitMQ, разработка программного обеспечения.
Introduction
The RabbitMQ server refers to a distributed and horizontally scalable message broker system that ensures communication between different subprograms by using the AMQP protocol, and when connecting additional modules, subprograms can communicate by using MQTT, HTTP protocols, etc.
The RabbitMQ message broker system is more simplified and can be described by the following points:
• It has a message sender, recipient and a sent event;
• It has a message store that determines message priority can be referred to as a mailbox»;
• It has subscribers, which are sent message receivers;
When sending messages, any number of sent messages from an unlimited number of senders can be stored in the mailbox, and an unlimited number of subscribers can receive messages. The main reason for the use of the RabbitMQ message broker system is that the application is implemented on the Erlang/OTP platform, thereby ensuring highly stable and scalable message transfer priority, being the main node in the system. The second reason for the use is the entire openness of the application, which is distributed under Mozilla Public License and implements open AMQP protocol. Configuration or programming libraries are available in all platforms and programming languages, in Node.js also. RabbitMQ key feature and benefit analysis The key features and benefits RabbitMQ has are:
• Open source code
The system advantage was gained as a result of joint cooperation between LShift, LTD and Cosmetic FT. An open source positive feature is its flexibility and customization. After all, improvements and additions to the system can be made.
• Independence from the system and its developer
Independence from the system and its developer is implemented under the AMQP (Advanced Message Queuing Protocol) specification, thereby making it available to clients in almost all languages and programming platforms.
• Minimum system requirements
Such a system requires less than 40 MB of RAM to run the central RabbitMQ combined with a plug-in like a user interface. However, it should be considered that if a large number of messages are sent to the mailbox push-up list, RAM memory consumption can be increased.
• Client library for most modern programming languages
When choosing a way to develop programs that communicate with RabbitMQ, there are no bindings to a developer or a programming language. In fact, RabbitMQ is often used as a central element between applications made in different programming languages. Such a system provides a bridge for languages like Java, Ruby, Python, PHP, JavaScript and C # to exchange data between different operating systems and environments.
• Flexibility to control message exchange
RabbitMQ provides flexibility to control the trade-offs required between message exchange reliability, capacity and productivity. Since no universal-type applications exist, messages can indicate whether they are to be saved to disk before being sent, and whether they are not to be lost in case of a server failure.
• Plug-ins used for high latency environment
Since not all network topologies and architectures have the same consumption, the RabbitMQ message brokerage system is implemented to exchange messages in the environment with a significantly low latency. Along with that, its main modules (World Wide Web) work in a high latency environment. In this regard, RabbitMQ can be clustered in a single local network environment and messages can be shared across multiple data centers.
• External plug-ins used
RabbitMQ offers a flexible plug-in system as a central point for application integration. For example, external plug-ins for storing messages directly in databases in records are available.
• Enhanced system security
Client connections can be protected via SSL communication only and by verifying the client certificate. User access can be controlled at the virtual host level, thus providing message and high-level resource isolation. In addition, access to configuration functions, reading from queues, and writing to an exchange are controlled by mapping to a regex template. Finally, plug-ins can be used to integrate with external authentication systems, e.g. LDAP.
Review of the running RabbitMQ service bus
RabbitMQ Event Bus
Message Publisher
Website or
Mobile application
Microservice having its
own database.
Putli^l/
Queue 1 containing
some messages
RabbitMQ Server
Event bus API
Queue 1 containing
some messages
Message Subscriber
Subs
Microservice having its own database
cribe
Event bus API
Microservice having its own database
Subsc'ibe
Event bus API
Figure 1 - RabbitMQ service bus architecture
The RabbitMQ service bus is a link between multiple microservices through which microservices can publish messages in various queues available within the RabbitMQ service bus. Other microservices may later subscribe to the messages available in RabbitMQ service bus queues. A microservice can update entities after an event is received, which can cause more events to be published.
An easy way to develop a corporate messenger will be further considered.
Clients
Android
IOS
RabbitMq Backend application
Figure 2 - RabbitMQ-based corporate messenger
As shown in the diagram, the system consists of the following parts:
• RabbitMQ;
• Backend application;
• Client applications: Android, iOS.
For the architecture to function, the exchanges and queues created when the backend application was first started are to be configured.
Major exchanges and queues are listed below:
• conversation.outgoing is an exchange, a division type, which is required to receive incoming messages sent by clients and route them in a queue;
• conversation.incoming is an exchange, a topic type, designed for sending processed messages to the user exchange. In this case, a topic type enables sending only those messages to users that relate to the dialogs which they participate in;
• chat-application-messages is a queue for the server application to process incoming messages.
After all used exchanges and queues are initialized, the conversation.outgoing exchange is connected to the chat-application-messages queue and the server message processor is created from the chat-application-messages queue.
The user registration functionality is the first step in all system part integration.
When a user registers with a server using one of the clients (iOS, Android), the server application creates a fanout-type exchange in RabbitMQ with a unique name generated and returns this exchange name to the client application in which the user is registered.
Since that moment, all authorized customers will also receive the name of the existing exchange.
Having received the user's exchange name, clients create a temporary queue in RabbitMQ, which exists only when the client is connected to the RabbitMQ message broker, and connect the queue to the exchange.
Creating a unique exchange for each user enables receiving messages from all clients (iOS, Android) in parallel.
When the Internet on the client is disconnected, or when the connection to the RabbitMQ message broker is lost, the temporary queue is automatically deleted on the RabbitMQ server, thereby preventing redundant messages and optimizing message delivery. When the connection is restored, a new queue is created.
Having registered, the user is ready to send and receive messages.
Messages are sent in dialogs. The system is to create conversations with both one and several contacts.
When creating a conversation, each user exchange is connected to the conversation.incoming backend application exchange by using the routeId conversation indicator.
The overall message delivery sequence is shown in Figure 3.
Further, the messages delivery sequence will be clarified. Incoming messages are transferred to the conversation.existent exchange, which sends them to the chat-application-messages queue. The server application processes all messages sent to the chat-application-messages queue.
When checking the message, it checks the validity of the user's token, conversation ID and the ability of the user to send messages to the conversation, processes the attachment that should be sent with the message, and the message is stored in the backend application database, and then is sent to the conversation.incoming exchange with the routeId conversation indicator. Conversation.incoming exchange sends a message to all exchanges having the same route key as the message sent.
Filtering sent messages in RabbitMQ allows to conveniently scale the number of contacts receiving messages without increasing the message delivery time.
Figure 3 - The general sequence of message delivery
A specific user exchange sends a message to all queues subscribed to this exchange, i.e. for all client applications.
RabbitMQ message broker can be applied in the enterprise service bus with regards to the listed system capabilities.
Several scenarios to use the above mentioned features will be considered:
• Edge set up
Setting up Edge to send messages to RabbitMQ. However, the messages are to be forwarded to the Microsoft Azure service bus for their further processing, so many Azure big data processing capabilities can be used.
• Hybrid cloud
It refers to the cases when a company partners with a third-party producer that uses RabbitMQ for messaging. The producer uses a different cloud. While migration to Azure is in process, data can be shared by bridging to RabbitMQ using the Microsoft Azure service bus.
• Third-party integration
It is used provided that a third-party producer uses RabbitMQ as a broker and wants to send the data, but the data is outside another organization. SAS key can be provided to give access to a third-party producer to a limited set of Microsoft Azure service bus queues where they can forward their messages to.
The list can be extended, but most uses mentioned above enable bridging between RabbitMQ and Azure.
Having logged into an account, a user must go to the Azure portal and create a new Microsoft Azure Service Bus namespace. Namespaces are the scope containers to locate messaging components, such as queues and partitions.
After the Microsoft Azure Service Bus namespace is created, a user is to click the Queues button on the left in the Entities section so that a new queue could be added.
The queue will be called from-rabbitmq. This reminds of where the messages come from. All other settings can be left as default, or they can be adjusted if needed.
To send messages from RabbitMQ to the Microsoft Azure service bus, the Shovel plugin supplied with RabbitMQ is recommended to be used. The module and its visual interface can also be enabled by using the rabbitmq-plugins enable rabbitmq_shovel_management command.
The command may run as Administrator. Then credentials required to connect RabbitMQ to Azure are to be received.
In addition, a Shared Access Signature (SAS) policy for the queue being generated is to be created so that RabbitMQ could publish messages to it. The SAS policy specifies what actions the external party can perform with the resource created. The idea is that RabbitMQ can send messages but cannot listen to or manage the queue.
To add the SAS policy, the Send check box must be selected and Create must be clicked. After a policy is created, it must be clicked to view the main connection string. the main connection string is recommended to be used so that RabbitMQ could communicate with the Microsoft Azure service bus.
Before the connection string could be used, it must be converted to the RabbitMQ AMQP connection format. Therefore, connection string must be converted. To do it, connection string must be inserted into the form, and then Convert is to be clicked. As a result, a connection string that supports RabbitMQ will be created. (This Web site runs locally in the browser, so no data is transferred over the network.) Its source code can be accessed on the GitHub website.
Then the RabbitMQ control plug-in should be open in the browser http ://localhost: 15672/#/dynamic-shovels and Admin is to be gone to - > Shovel Management, where a new Shovel module can be added to send messages from the RabbitMQ queue to the Microsoft Azure service bus queue. In addition, the generated Shovel azure module is to be called and AMQP 0.9.1 is to be selected as the source protocol.
Azure can be used as the queue name. If this queue does not exist, RabbitMQ creates it automatically. The name of the existing queue can be also selected. Defaults can be set up for other parameters.
Then, AMQP 1.0 is to be selected as the protocol in Destination. In the URI field, the connection string created from converting the Azure connection string to the RabbitMQ format is to be entered. It should look like this: amqps://rabbitmq-hovel:[email protected]. net:5671/?sasl=plain.
In the Address field, the name of the Microsoft Azure service bus queue should be entered, in this case it is called from-rabbitmq. Add Shovel must be clicked and the installation must be ready to receive messages.
The process of publishing messages from RabbitMQ to the Microsoft Azure service bus will be also considered. In the RabbitMQ control interface, Queues can be navigated to, the azure queue can be selected, and for the Publish message panel can be searched. A form to post directly to the queue appears. In the example under consideration, first message is to be simply inserted in Payload and Publish Message is to be clicked.
In order to make sure that the message was sent from RabbitMQ, the Peek tab should be selected and the Peek button should be clicked to retrieve the latest messages from the generated queue. To check the message contents, it should be clicked.
Thus, having performed all the actions mentioned above, the Microsoft Azure service bus provides messages from RabbitMQ. All the steps to be performed are listed below:
- creating a Microsoft Azure service bus namespace;
- adding a queue to the namespace;
- adding a SAS policy to the queue;
- creating the queue connection string;
- enabling the RabbitMQ Shovel plug-in and control interface;
- converting the Microsoft Azure service bus connection string to AMQP format for RabbitMQ;
- adding a new Shovel module to RabbitMQ and connecting it to the Microsoft Azure service bus;
- publishing messages.
Following the instructions given above, the organization running outside Azure was integrated. The Shovel plug-in delivered messages from RabbitMQ to the Microsoft Azure service bus. This provides vasr benefits, as reliable third-party producers get the possibility to connect to applications using the Azure deployment. After all, the message service aims to create connections.
The diagram has several tracks: a human resources department employee, a company employee. The initial task for a company employee is to fill in n application form. The data flow then moves to the human resources employee track. The Accept Employee Application task is performed by an HR employee. After some time delay, the task "Creating an order in the 1C: SP system" starts. It is followed by the user task "Uploading order to electronic media." Further, the HR employee creates a task card in Directum 5.7 . After that, the HR department employee uploads the order into the generated task in Directum 5.7 . Next, the employee checks the order relevance, and the task goes to the gateway, which directs the process according to the response. If an incorrect order is uploaded, the HR employee replaces the order with another one. If the correct order is uploaded, the process goes to the task "Start approval for the order in Directum 5.7 EDI" and the process is completed. To reduce time and optimize the process, the two specified information systems re to be integrated using the RabbitMQ message broker.
Figure 6. Optimized process for launching HR orders. BPMN diagram
The integrated system operation principle will be as follows (see Figure 1). The application, specifically 1C: SP, sends messages to the RabbitMQ node (exchange point), at this point RabbitMQ sends an acknowledgement of message receipt back to the 1C: SP system. The receiver, specifically se Directum 5.7 EDI, is permanently connected to RabbitMQ and waits for the message broker to return the message. The recipient
acknowledges the receipt of the message or reports an error. If the delivery fails, RabbitMQ resends the message until it is delivered. Once successfully sent, the message is removed from the queue.
The process can be compared to sending a parcel by mail. First, the parcel arrives at the post office, then they are sorted between post offices. Then the postman delivers them to the specified addresses and makes sure that the parcel has been handed over to the recipient.
The BPMN diagram illustrating the optimized HR order loading process is shown in Figure 6.
After the information system for personnel order approval service was implemented, the process of launching personnel orders changes as follows. After a company employee filed an application, it is handed over to an HR employee. An HR employee accepts the application and creates an order in the 1C:SP system. Next, the script starts running. The script is responsible for creating a task card for approval in Directum 5.7 EDI and uploading the order into the generated task card. After that, the employee initiates the order approval in the Directum 5.7 EDI and the process is completed.
Comparing Figure 2 and Figure 3, the process of launching personnel orders is noticeably simplified due to the optimized and automated tasks like "Uploading an order to an electronic medium" and "Loading an order into a generated task card". These units were automated by using the personnel order approval service.
Table 1 compares the time spent on personnel orders launched by the HR employee before and after the information system integration.
Table 1
Span time comparison
± Launching HR orders before information system integration Launching HR orders after information system integration
Application admission 42 sec. 42 sec.
Creating an order in 1C: SP 1 min. 5 sec 1 min. 5 sec
Saving the order to electronic media 37 sec. N/A
Uploading the order to EDI Directum 5.7 57 sec. N/A
Uploading the order from 1C: SP to EDI Directum 5.7 N/A 36 sec.
Launching the order in EDI Directum 5.7 27 sec. 27 sec.
Overall time spent 3 min. 48 sec. 2 min. 50 sec.
Table 1 shows that the time spent running personnel orders before integrating information systems exceeds the time spent running personnel orders after information systems were integrated by 58 seconds. Suppose an HR employee runs 5 HR orders a day, then the time saved per week is 1450 seconds or 24 minutes 10 sec. The saved time per month is 5800 seconds or 1 hour 36 minutes 40 sec. The time saved per year is 69600 seconds or 19 hours 20 minutes.
Having compared the integration result, it was clear that the chosen integration method can reduce the timing budget, and thereby speed up the process of launching personnel orders. This method is not exclusive and implies additional innovations and improvements.
The proposed integration method has automated the process of launching personnel orders. The development of information system integration speeds up the process of launching personnel orders and makes it possible to perform tasks asynchronously. The main advantage of the chosen approach is that software tools developed on its basis are autonomous and cross-platform. The RabbitMQ-based integrating information system method can be used to solve intersystem integration problems occurring in any area.
Conclusion
In the given article, the RabbitMQ service bus architecture was discussed. Each component belonging to the RabbitMQ service bus API was also considered. In addition, a simple app that publishes messages to the queue was created. Then, there is a subscriber or recipient of the message who picks up the message from the queue. By using a RabbitMQ message broker, effective mobile messengers can be developed for both iOS and Android, thus solving the issue of application communication. As a result, developers pay more attention and spend more time on implementing business logic of the solution.
RabbitMQ message broker used in enterprise service bus Gladun A.M.
ATON LLC
The RabbitMQ server will be considered as a broker message system in the article. The server under consideration receives messages from senders and sends back the acknowledgement of the message receipt, after which the message is redirected to system subscribers. Subscribers acknowledge the message receipt, or notify about a message undelivered. Alternatively, the message may be in the transmission queue phase until this message is delivered to the addressee. After the message is delivered to the addressee, it will be deleted automatically from the system. The main advantage of the RabbitMQ server is its flexibility in routing messages between different subscribers and recipients. For an illustrative example, the ways the same message should be sent to 3 recipients will be consider. This message arrives at the node, which distributes it into 3 identical messages and puts it in three queues for the recipients to whom it will be delivered. In the article, the author investigated the features of the rabbitmq message broker to be used in the enterprise service bus. Keywords: systems, message broker, rabbitmq, software development. Reference
1. Luchkin A.S. Library for simplifying work with the rabbitmq message broker. // Certificate of registration of the computer program 2021682095, 12/29/2021. Application No. 2021681929 dated 12/29/2021.
2. Ignatov E. Communication of the assembler with third-party services through the message broker rabbitmq. // In the book: Joint Conference "PDF: from training to development". Collection of conference abstracts. Moscow, 2022. pp. 178-179.
