Skip to content

Using Integration Patterns

All integration scenarios are about making the right information available at the right time in the appropriate format in order for a business decision to be made. Although each programme or project requirement may be unique, the solution required to facilitate it can be designed using common integration patterns and assembled from common building blocks. Before the different scenarios and patterns are described, it is important to have a common understanding of some of the terms used and their meanings. This section describes the common terms.

As previously stated, each requirement can be grouped into one of three types of integration requirement; A business process step, an event trigger, or accessing data.

Business Process Step

This group of integration requirements focuses on how to get information from the underlying applications to support a specific step in a business process. The resulting data from the step in the process is then sent on to the next step in a process or sent on to another application within the business process flow. The data being exchanged may contain some or all of the following:

  • control data – that is data that tells the next step in the process what to do next
  • metadata – data that describes the data that will be processed – as an example the schema or style-sheet that describes the actual data format to be processed
  • data - the actual that is to be processed.

The different types of data may be sent in separate messages or may be combined into one message.

A transaction is a set of data that needs to be processed through the enterprise. It may be that the transaction is processed within one system or application or typically through many systems in order to complete. Transactions can be either long running or short lived. A long running transaction may be triggered by an event – for example, the receipt of an order and it is necessary to maintain the state of the transaction all the way through the fulfilment cycle, which may be over several days. By contrast, short lived transactions are completed in a very short period of time and typically involve the transfer of data from one system to another. Once it is known that the data has reached its’ destination, the transaction is complete. Throughout the processing, the transaction integrity needs to be maintained – this means that all components that are involved in the processing of the transaction must ensure that the transaction data is not corrupted and care must be taken to code the processes to handle errors gracefully.

Event Trigger

A business event or a change in data acts as the trigger that causes data to be distributed from the application to other systems. The data may be published by the system over a messaging mechanism and all interested systems would subscribe to receive this data if they are interested. In this case, the data would typically be published in real time, using a common format – usually based on JSON using a Harmonised Data Model format. The publishing system is not aware of which applications subscribe to this data, this is handled by the configuration within the middleware layer. Receivers would need to either process the standardised data as is, or transform it to the format which they can utilise. In some cases, the receiving system may store the updates on a receiving queue and process the updates in batches.

Accessing Data

The requirement is to make either small or large sets of data available to systems in order to continue with their function. Data may be pushed i.e. the sending application or system initiates the processing of information exchange or pulled – where the receiving system initiates the activity and obtains the data it needs. The data may be in a format which is parameterised - a name value pair – specific tags define what data is present, and the actual value for that tag is then contained in the data separated by an agreed delimiting character, or be formatted using a harmonised data format – agreed and defined by the Integration Team in conjunction with the Data team. The exchange of information may be an initial load – i.e. a set of data is sent from one system to another in order to replicate the data for the first time; a set of data may contain only that data that has changed since the last set of data was published– i.e. a delta, or a data refresh – where data has been requested by a system in order to re-synchronise from a specific point in time. The refresh may consist of a set of data that may include a set of ‘deltas’ and in some cases an initial load as well.

Decisions on which integration pattern is the most appropriate one to use needs to take into consideration several parameters. Below are the high level parameters that will help make that decision:

  • Transport protocol – this refers to the method that is used to move the data between systems; this is not the network level protocols. This can be message based using a queuing mechanism, file based for larger data sets. The other key decision point as to which protocol to use depends on the type of data being processed – where data is of a confidential nature (example employee details) a secure protocol such as https, SFTP or encrypted files or messages must be used.

  • Payload size – this is the size of the type of data that is to be transferred. For small payloads, (typically less than a megabyte), message based flows are more suited. Some message transports can handle payload sizes of up to 4Mb with their default configuration. Beyond that size, special configurations have to be implemented to cater for the larger payloads. It is also good practice to separate large payloads into separate channels for processing through the integration layer so that they do not have an impact on the processing of smaller payloads.

  • Frequency – the correct pattern to use will also depend on the frequency of how often the data needs to be processed. Some data must be made available immediately to other systems. In this case, data must be processed in ‘real time’ or in ‘near real time’. For this type of frequency, a message based or service based paradigm is better suited. Other data is sent periodically, perhaps at the end of day, or once a week. Typically, the volume of data here would be large – in which case it could be processed in batches.

  • Reliability – this refers to whether the data must be persisted or not within the integration layer. In some cases, where frequent updates are being made to the same data, it may more efficient in processing terms to ignore updates sent over a period of time and only process the most recent update. It is recommended, however to persist data and guarantee delivery of it in most cases.

  • Idempotent – the integration layer must be able to perform the same function to the data and produce the same result each time it is utilised with the same parameters.

  • Synchronous / Asynchronous – consider whether responses to a particular request have to be processed and returned immediately (synchronous) or can be given at a later time. For messaging / batch processing scenarios, asynchronous calls are advisable as this allows systems to process the data in their own time and does not hold up any processing within the separate applications. For service oriented integration, synchronous responses are usually required.