Skip to content

Small package data replication

Scenario ID

  • SC-SDR

Scenario Name

  • Small Data-Replication

Characteristics

  • Scheduled, small in size (<6Mb), file or message based, one to one (1-1) or one to many systems(1-n)

Description

  • Batch scenario where a scheduled process runs at frequent intervals to keep other systems up to date. The requirement is not immediate replication, but close to real-time, from hourly down to every minute.
  • Messages are small and delivery is usually required to avoid systems from getting de-synchronized.
  • Even if guaranteed delivery is built into the interface, there is usually a full refresh weekly or monthly which is done via (SC-BDR), or as a high volume burst of individual messages or small batches, (SC-EDR or SC-SDR), in which case, this creates peaks of high-volume flows of data. These should be scheduled preferably out of business hours.
  • Both messages and files can be used, with messages being preferred. The only reason to use files is to re-use full-refresh extract/load scripts.
  • Transport of files is currently often FTP or FTPS, and for messages, there is local Message Queue (JMS, AMQP) servers.
  • As with scenario SC-EDR, there are often several subscribers, so the same choices should be made as for SC-EDR.
  • The use of ESB will drive re-use of the same data messages, and the publisher should only have to know what and when to publish something, while the ESB manages the further distribution of data.

Recommendations

  • Use a common message model for data objects being transferred within the integration layer. This allows for further de-coupling of systems.

Example

  • None

Pattern Reference