Process Connection Diagram
Overview
Enactor uses a configurable diagram to define queues and message flows across Enactor applications as well as between Enactor and external systems. This is called the Process Connection Diagram. This can be used to define queues and how they are connected to each other and how messages should flow.
This document intends to introduce this to developers to be able to have a deeper understanding of this diagram.
What You Will Learn
This document describes each element in the connection diagram and their usages as well as how to configure a new queue.
Connection Diagram
A queue can be defined in the connection diagram by simply defining a 'Queue' object in the connection diagram.
A Connection Diagram can be visualised using the Enactor Tools Connected Process Editor.
The connection diagram is technically represented as an XML and a developer can view this as a text file if required as well.
This diagram would contain various elements.
Connected Applications
A connected application will be a logical application definitions that defines incoming and outgoing connections to that application.
The connected application will define few properties.
The XML representation of a connected application would be similar to the following.
<core:connectedApplication>
<core:name>TransformedRetailTransaction</core:name>
<core:processId>TransformedRetailTransaction</core:processId>
<core:hostId>\#\{getConfigValue('DeviceType.EstateManager','ESTATE_MANAGER')\}</core:hostId>
<core:connectedApplicationProcessMetaData>
<core:inputs>
<core:input>Transactions</core:input>
<core:input>RetryTransactions</core:input>
</core:inputs>
<core:outputs>
<core:output>TransformedRetailTransaction_Failed</core:output>
<core:output>TransformedRetailTransaction_Retry</core:output>
<core:output>TransformedRetailTransaction</core:output>
</core:outputs>
<core:inboundQueues>
<core:inboundQueues>Updates</core:inboundQueues>
</core:inboundQueues>
</core:connectedApplicationProcessMetaData>
<core:uiSettings xsi:type="tools:ConnectedProcessEditorSettings" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<core:constraint xsi:type="core:UIPositionConstraint">
<core:x>4900</core:x>
<core:y>492</core:y>
</core:constraint>
<tools:displayedSections>
<item>INPUTS</item>
<item>OUTPUTS</item>
</tools:displayedSections>
</core:uiSettings>
</core:connectedApplication>
Following are some important properties of this connected application.
Property | Description |
|---|---|
| Process ID | This is the unique identifier for the process. This is used in development to uniquely identify a connection. |
| Name | This is a user friendly name and is used to display this process in the Connected Process Editor |
| Host ID | This defines which device type this connected process is valid for when executing. The value #{getConfigValue('DeviceType.EstateManager','ESTATE_MANAGER')} specifies that this connected application is valid for the device type equal to the value defined against DeviceType.EstateManager in enactor.xml and in case that value is not defined, use 'ESTATE_MANAGER' as the default value to compare. |
| Inputs | This will be a list of input connection definitions which is represented as a list of strings. |
| Outputs | This will be a list of outputs connection definitions which is represented as a list of strings. |
| Inbound Queues | This defines the queue that are considered as inputs to this process |
Queues
A queue is s simple definition of a queue. The deinition of a queue itself is only a logical definition and depending on the other elements around the queue, specially the queue connectors and queue connections, the queue can take various shapes such as a file system queue, database queue or even an external topic such as a Google Pub/Sub or a Kafka Topic.
Following is the XML representation of this.
<core:queue>
<core:queueDescription>Pos local database transaction queue</core:queueDescription>
<core:queueId>PDPPosTransProc</core:queueId>
<core:queueName>PDPPosTransactions</core:queueName>
<core:metaData>
<core:inputConnectionProviders>
<core:provider>LocalJDBC</core:provider>
<core:provider>HttpBackOffice</core:provider>
</core:inputConnectionProviders>
<core:outputConnectionProviders>
<core:provider>LocalJDBC</core:provider>
<core:provider>HttpBackOffice</core:provider>
<core:provider>HttpEMPDP</core:provider>
</core:outputConnectionProviders>
</core:metaData>
<core:uiSettings xsi:type="tools:QueueEditorSettings" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<core:constraint xsi:type="core:UIPositionConstraint">
<core:x>729</core:x>
<core:y>219</core:y>
</core:constraint>
</core:uiSettings>
</core:queue>
The queue properties would be visualized in two different variations.
Standard Properties will be displayed in the standard properties view.
Inbound and outbound connection properties are represented in the outline.
Following are some important properties of a defined queue.
Property | Description |
|---|---|
| Queue ID | This is the identifier of the queue that is used to uniquely identify this queue programatically. |
| Queue Name | This is the actual queue name that would be materialized in the messaging system used as the queue/topic |
| Variables | List of variables related to this Queue. This has two use cases. 1. Define system variables such as 'enactor.coreUI.ThreadSuffix' or 'enactor.coreUI.MessageSystemThreadedQueue' which provides system instructions specially for multi threaded processing. 2. Define variables names that should be resolved to get the final queue name. For example, if a queue name is defined as 'Device(#{deviceId})', a variable named 'deviceId' of type String should be defined in order to properly resolve this variabled. Also note that this is only the definition of the variable name. The variable value should be available in the ELContext at the time of the queue name resolution in order for the variable value to be resolved. |
Process Connection
A process connection is a representation of a message flow or a connection between two elements in this diagram. This can be
- Application Output -> Queue : This suggests that if you write a message to this application output, the message will go to the connected queue.
- Queue -> Application Input : This suggests that if you write a message to this queue, you can consume it using the connected application input.
- Application Output -> Web Service : This suggest that if you write a message to this application output, the message will be sent to this connected web service.
- Web Service -> Application Input : This suggests that if you send a message to this web service, this can be consumed by the connected process. Note though that these are Enactor specific specialized web services only.
- Queue -> Queue Connector : This suggests that the message that you write to this queue will be read by this queue connector as the source queue with the purpose of forwarding it to a different queue.
- Queue Connector -> Queue : This suggests that this is the queue that the message will be ultimately sent to by the queue connector after reading from a source queue.
A process connection takes the shape of a line in the visual representation.
The standard property editor provides a way to edit the properties of this. However, it's always easier to use the designer to draw the connections using the Link tool in the pallete.
The XML representation would take the following form.
<core:processConnection>
<core:connectionId>1</core:connectionId>
<core:fromId>RetailCustomerOrders</core:fromId>
<core:fromOutputId>cancelOrder</core:fromOutputId>
<core:fromType>PROCESS</core:fromType>
<core:toId>CustomerOrderService</core:toId>
<core:toType>SERVICE</core:toType>
<core:providerId>RetailCustomerOrdersService</core:providerId>
</core:processConnection>
Following are some important properties of a process connection.
Property | Description |
|---|---|
| From ID | The component the connection starts from |
| To ID | The component the connection goes to |
| From Host ID | To which device types the from connection is valid for. For example if this is set to ESTATE_MANAGER, the from connection is only valid for ESTATE_MANAGER device types. Therefore, if you were to use this connection diagram in POS, POS would not consider this connection as a valid connection for itself. |
| To Host ID | To which device types the to connection is valid for. |
| Provider ID | This connection itself defines a flow of a message. The provider Id defines which mechanism should we use to deliver the message from source to the target. This can be a Queue Connection Provider or a Web Service Provider |
Queue Connection Provider
A Queue Connection Provider defines a mechanism a message can be delivered from A to B. This includes both reading a message from a queue and writing a message to a queue. Enactor platform defines a set of generic interfaces that together represents a messaging mechanism.
All of these together makes a new transport mechanism for a queue.
com.enactor.core.queues.connection.IQueueConnection
com.enactor.core.queues.IQueueReader
com.enactor.core.queues.IQueueWriter
com.enactor.core.queues.connection.IQueueSession
com.enactor.core.queues.connection.IQueueConnectionFactory
A Queue Connection Provider component in the Process Connection Diagram defines such a mechnism that can be used to read and write messages in queues with some additional configurations to support it.
Queue Connection Providers are listed in the document outline. There is not other visual representation of this in the diagram.
The standard properties section will describe the common properties used in this.
The XML representation of a Queue Connection Provider will look lke below.
<core:queueConnectionProvider>
<core:endpointReference xsi:type="core:HttpQueueProviderEndpointReference" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<core:accountCredentialsId>MESSAGE_SERVICES</core:accountCredentialsId>
<core:url>\$\{Server.BOProc.URL.Base\}/WebCore/HTTPMessageServiceServlet</core:url>
</core:endpointReference>
<core:connectionProviderId>HttpBackOffice</core:connectionProviderId>
<core:name>Http Back Office</core:name>
<core:connectionFactoryClassName>com.enactor.messageService.HTTP.HTTPMessageServiceConnectionFactory</core:connectionFactoryClassName>
</core:queueConnectionProvider>
Following are some important properties of a Queue Connection Provider.
Property | Description |
|---|---|
| Connection Provider ID | This is the unique identifier and this provider will be referred to by other componenets in the connection diagram using this ID |
| Connection Factory Classname | This is the class name of the implementation that should be used when this provider is in use. This is a pointer to say which messaging mechanism we should use for this provider. |
| Endpoint Reference | This is a complex object with more of it's own properties that will be described later in this document. The purpose of defining an endpoint reference is to provider connection configurations to this messaging mechanism. The purpose of defining an endpoint reference w ithin a provider is so that we can have more than one provider with the same implementation but pointing to different endpoints/environments |
Enactor platform supports following messaging mechanism vis Queue Connection Providers out of the box.
- HTTP Message Service - This can be used for reading and writing messages to and from an HTTP endpoint that support Enactor message objects.
- File Message Service - This can be used to write and read messages to and from a file that is saved to the disk.
- Database Message Service - This can be used to utilize the database table MessageQueue to read and write messages.
- JMS Message Service - This can be used to communicate with queues messaging systems that supports standard JMS message protocol.
- Kafka Message Service - This can be used to communicate with Kafka topics
- Google Message Service - This can be used to communicate with Google Pub/Sub topics.
- Kinesis Message Service - This can be used to communicate with AWS Kinesis streams
- SFTP Message Service - This can be used to utilize SFTP protocol to read and write messages.
The above list is a high level list and there are variations of these available out of the box that provide different functionality on top of these like batch processing and listener implementations.
Queue Connector
A Queue Connector as the name suggests is a component that connects two queues. It can be used to read messages from one queue and write that message to a different queue. This is one of the most used components in the connection diagram.
The most common usecase of this is to build a reliable message transfering mechanism with external systems.
As an example, if POS wants to send a transaction to the Estate Manager, but if at the time, Estate Manager is offline, POS will not be able to send this transaction to Estate Manager. Therefore POS will either have to keep retrying without moving on or discard the message. As a solution to this,
- First we write the transaction to a database queue in POS.
- Next we add a Queue Connector to read messages from this database queue and send it to Estate Manager.
- The 'PosTransactions' queue in the above image is a database queue that resides in the POS database.
- The 'ServerTransactions' queue in the above image is a database queue that resides in the Estate Manager database.
- The 'Pos Queue Up Connector' acts as the intermediate agent to read messages from POS and push it to Estate Manager.
- The Queue Connector will only commit the consumed message from 'PosTransactions' queue if the message was successfully written to 'ServerTransactions' queue. Otherwise it will rollback the Queue Session retry until the target system is available. The retry will happen at the next poll and the poll interval can be configured. Please see the properties section.
The standard property editor will allow you to modify the available properties.
Following is the XML representation of this.
<core:queueConnector>
<core:name>Pos Queue Up Connector</core:name>
<core:processId>PDPUpQC</core:processId>
<core:hostId>PDP_SERVER</core:hostId>
<core:queueConnectorMetaData/>
<core:uiSettings xsi:type="tools:QueueConnectorEditorSettings" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<core:constraint xsi:type="core:UIPositionConstraint">
<core:x>2396</core:x>
<core:y>367</core:y>
</core:constraint>
</core:uiSettings>
</core:queueConnector>
Looking at the properties and the XML, we can see that this does not define the source queue or the target queue. This is because the source and the target will be defined by a process connection, which is a connecting line between the relevant queues and the queue connector.
Following are some important properties of the Queue Connector.
Property | Description |
|---|---|
| Process ID | This is the unique identifier of the component which will be used to reference this in other components |
| Host ID | This specifies which device type this applies to |
| Poll Interval | This is the poll interval between two polls from the source queue in milliseconds. If set to '0' this will default to 30,000. |
| Batch Size | This is the batch size to poll. If not defined, 10 will be used as the default batch size. |
How Does a Queue Connector Work Underr the Hood
A background service is configured to run the queue connector process which evaluates all the Queue Connectors available, then initiates worker threads to poll and process messages per Queue Connector.
For example, on the POS, usually, we would have a Service Type named PosQueueConnector and a Service Definition that is using this Service Type. This service will invoke the platform process MessageService/RetailQueueConnector process which would invoke the MessageService/QueueConnector platform process that is responsible for managing the Queue Connectors.
These processes iterates thorugh the Queue Connectors and creates a work list and iterates through them on configured polling schedules. There can be more than one such thread running and the number of threads can be configured using the enactor.xml property QueueConnector.NumberOfThreads. If not configured, the number of threads defaults to 5.
Endpoint Reference
An endpoint reference is a definition of an external endpoint. This can take many forms such as
- A REST endpoint
- A SOAP endpoint
- A messaging provider endpoint
Endpoint References can be embedded within Queue Connection Providers, Web Service Connection Providers as well or web service connections to provide connection details for the perticular component.
The standard properties of an Enadpoint Reference can be viewed by selecting the relevant parent component.
Following are some important properties of an Endpoint Reference.
Property | Description |
|---|---|
| URL | This is the endpoint URL this endpoint refers to. This can include variables as well. |
| Account Credentials Id | An account credential entity can be tied to an endpoint reference and that account credential can enrich the details of the endpoint. For example, if the referred account credential defines a property named 'url', this will override the URL specified in the connection diagram making the URL configurable without modifying the account credential. |
Following is a Rest service with embedded Endpoint Reference represented in the XML.
<core:restServiceConnection>
<core:endpointReference xsi:type="core:restEndpoint" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<core:isDynamicEndpoint>true</core:isDynamicEndpoint>
<core:url>\#\{baseUrl\}payments/authoriseCustomerCard</core:url>
<core:httpMethod>POST</core:httpMethod>
<core:queryParams/>
<core:headers/>
<core:payloadFormat>JSON_CLASS</core:payloadFormat>
<core:requestTransformParams/>
<core:responseTransformParams/>
</core:endpointReference>
<core:connectionProviderId>AuthoriseCustomerCard</core:connectionProviderId>
</core:restServiceConnection>
Following is a Queue Connection Provider with an embedded Endpoint Reference.
<core:queueConnectionProvider>
<core:endpointReference xsi:type="core:HttpQueueProviderEndpointReference" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<core:accountCredentialsId>MESSAGE_SERVICES</core:accountCredentialsId>
<core:url>\$\{Server.BP.URL.Base\}/WebCore/HTTPMessageServiceServlet</core:url>
<core:headers/>
</core:endpointReference>
<core:connectionProviderId>HttpBP</core:connectionProviderId>
<core:name>Http Business Process Server</core:name>
<core:connectionFactoryClassName>com.enactor.messageService.HTTP.HTTPMessageServiceConnectionFactory</core:connectionFactoryClassName>
</core:queueConnectionProvider>
Web Services
A web service is a definition of an indipendant endpoint that the application needs to reach. Web services are defined within a Web Service group and is usually acompanies by an endpoint reference.
Usually, an Process Connection ends with with a Web Service to describe that a perticular connection points to this specific web service.
Web Service Group
A web service group is simply a group of web services defined a one single logical unit.
