Sync Strategies Part 3: Real-Time Syncs

Document created by Adam Arrowsmith Employee on Jul 21, 2016Last modified by Adam Arrowsmith Employee on Jul 21, 2016
Version 2Show Document
  • View in full screen mode

This four-part series discusses various methods and considerations for keeping multiple applications in sync, including one-way, bidirectional, real-time/event-based, and hub-and-spoke.

 

 

 

Overview

It’s time to get real, with real-time sync strategies. In Part 3, we talk about how real-time or event-based integration patterns allow you to sync incremental data changes between applications immediately and efficiently.

 

In the previous two installments, we talked about batch-style approaches to data synchronization, where the integration syncs data incrementally by querying each source application looking for new or modified records every so often. In a real-time approach, the integration just sits back and waits for the source application to proactively send changes. What better way to sync only changes than if someone tells you exactly what changed!

 

One of the strongest advantages of real-time integration is just that: changes to records are synced as soon as they occur vs. waiting until a scheduled job comes looking for them. For time-sensitive integration scenarios, a real-time approach is usually the preferred option.

 

You can use real-time integrations to sync data in one or both directions between applications. However if you sync data bidirectionally between the same two applications, make sure each application can somehow detect changes made by the integration itself so as not to endlessly fire messages back and forth for the same change! See Sync Strategies Part 2 for ideas on how to prevent this situation using application triggers.

 

In addition to data synchronization use cases, real-time integration patterns can be leveraged for a number of other integration scenarios including user-interface “mashups”, data entry validation and enrichment, and business process invocation. In this article we will focus on data integration however the AtomSphere platform and features discussed below can support all of these scenarios.

 

Real-Time Integration Considerations

Before diving right into the how-to, let’s look at some of the technologies and considerations involved with real-time integration.

 

Technologies

When it comes to real-time communication technologies, two of the most commonly used today are web services and Java messaging services (JMS). Generally speaking, JMS-style messaging such as Apache ActiveMQ and IBM WebSphere MQ is used to send messages between applications within the same local network whereas web services are used to send messages between applications on different networks or clouds across the internet. For this discussion we’re going to focus on web services for cloud application integration.

Web services can be implemented in variety of ways from formally defined SOAP and REST specifications, to proprietary XML formats or really any data format communicated over HTTP(S). The specification (or lack thereof) will usually be determined by the capabilities of the applications being integrated. For example, if the source application generates outbound messages using a self-defined SOAP format, then the integration will have to accommodate that SOAP format. If the source application offers options or if you’re developing from scratch, you have more freedom to decide the format to use. (We’ll save the SOAP vs. REST vs. custom conversation for another day.)

 

Comparison

When compared to batch-style integration, real-time integration is a little more sophisticated, involving more moving parts and therefore more considerations when implementing.

 

Advantages

  • Syncs changes immediately instead of waiting for next scheduled poll.
  • Avoids needless queries to source application. This is advantageous if system resources or API governance limits are concerns.

 

Disadvantages

  • Requires configuration and/or development on the source application to detect changes and send messages to the integration endpoint.
  • Is not supported by all applications, although is becoming more common with cloud-based applications.
  • For cloud to on-premise scenarios, requires firewall/network configuration to expose the local integration endpoint to the public internet. (See Bonus Topic below for a potential solution.)

 

Additional Considerations

  • The messaging mechanism in the source application should implement some level of fault tolerance capability to acknowledge and resend messages.
  • For very high volumes, infrastructure load balancing techniques will typically be needed.

 

What Does Real-Time Integration Look Like, Really?

Time to get into the good stuff: how it actually works. Remember, conceptually the “big idea” is:

  1. The integration server is configured to “listen” for changes from the source application.
  2. When the source application detects a change, it sends message to the integration.
  3. The integration receives the message and processes it immediately (doing whatever the integration needs to do, presumably sending it to some destination application).
  4. Optionally, at some point the integration sends a message back to the source application to acknowledge the receipt or completion of the original message.

 

With that in mind let’s take a closer look at the two main patterns, synchronous and asynchronous, and how to configure the Web Services Server connector to implement these patterns as part of AtomSphere’s Services Enablement feature.

 

Synchronous Pattern - Request/Reply

The synchronous or “request/reply” pattern is used when the source application sends the message but then waits for a response from the integration because it needs to perform further processing upon completion. Keep in mind this is the source application waiting, not the end user.

 

Sync 1 Sequence.png

 

  1. End user creates or modifies a record.
  2. Source application detects the change, sends a message containing the record data to the integration, and suspends further processing while it waits for a response.
  3. The integration receives the message, performs any transformations, and sends to the destination application.
  4. The integration handles the response and returns a confirmation message to the source application.
  5. Once the source application receives the response, it continues with any post-processing.

 

The integration process and operation configuration might look something like this:

 

Sync 1 Process.PNG

 

Sync Operation.PNG

 

The two configurations that make this process execute synchronously are 1) the specification of an Output Type in the operation and 2) the use of a Return Documents step at the end of the process. If you specify an Output Type, be sure that documents will always reach a Return Documents step no matter what conditional path they take, otherwise an empty message will be returned to the source application.

 

Asynchronous Pattern - Fire and Forget

The asynchronous or “fire and forget” pattern is used when the source application sends the message but does not need to know the final disposition of the integration (at least not immediately). It will expect a technical acknowledgement (e.g. an HTTP 200 response) to confirm the integration received the message, but after that the application will continue about its business.

 

Async 1 Sequence.png

 

  1. End user creates or modifies a record.
  2. Source application detects the change, sends a message containing the record data to the integration.
  3. The integration receives the message and immediately returns an acknowledgment that it received the message.
  4. The integration then processes the message, performing any transformations and sending to the destination application.

 

The integration process and operation configuration might look something like this:

 

Async 1 Process.PNG

 

Async Operation.PNG

 

The subtle but critical configuration difference that makes this process execute asynchronously is the omission of an Output Type in the operation (and consequently no need for a Return Documents step). When the operation is configured without an Output Type the Atom’s web server will automatically return an HTTP 200 response to the source application before processing the documents.

 

Variation 1: Lightweight Message

One common variation is for the source application to send a lightweight message containing only the minimal reference information (such as a system ID) to the integration instead of the full record details. The integration then queries the source application to extract the full record before processing and sending to the destination. This approach is advantageous when the source record is complex and/or the fields required for the integration are subject to change, making configuration or coding changes difficult in the source application. It is often easier to accommodate these requirements in a configuration-based integration tool (like AtomSphere!).The example below shows the variation in the context of a synchronous pattern but it is applicable to both patterns.

 

Sync 2 Sequence.png

 

Sync 2 Process.PNG

 

In the process, note the connector call immediately after the Start step to query the source application, presumably passing in a record ID dynamically from the lightweight message received.

 

Variation 2: Source Writeback

Another common variation is for the integration to update some information in the source application upon completion by initiating a new call via the source application’s API. As discussed in Sync Strategies Part 1 this is often done to exchange internal IDs or update a status field after a successful sync.

 

The example below shows the variation in the context of an asynchronous pattern but again it is applicable to both patterns.

 

Async 2 Sequence.png

 

Async 2 Process.PNG

 

At the end of process, note the connector call to the source application to update the record being processed.

 

Final Thoughts

Real time integration is a great way to sync records incrementally but there are some design and infrastructure considerations to address. However if supported by the endpoint applications and record update timeliness is critical to your integration scenario, it’s your best option.

 

Stay tuned for our final installment...Sync Strategies Part 4: MDM!

 

Bonus Topic: Getting Around Firewalls for Cloud-to-On-Premise Integrations

So you want an event-based integration from your brand new cloud application to your existing on-premise application but you can’t open firewall ports to allow the inbound traffic into your network...what are you to do? Well if your scenario can be asynchronous you can leverage the Execute Process API and use the Atom Cloud as a proxy. If you are not familiar with the Execute Process API, it is a simple XML-based API that lets you programmatically execute a process running on any of your Atoms with the ability to provide runtime parameters in the form of process properties (read more).

 

Here’s how:

  1. Build a process that will receive the event-based message from the cloud application, construct the Execute Process request, and call the AtomSphere Execute Process API using the HTTP connector. Include the record ID from the message as a process property (name is arbitrary) in the Execute Process request.
  2. Deploy that process to the Atom Cloud as a web service so it is publicly accessible.
  3. Build a second process that begins with a query to the cloud application with a filter on the record ID. In the connector step parameters, configure the filter to retrieve its value from the process property (same name of course).
  4. Deploy that process to the local Atom.
  5. Configure the source application to send event-based messages to the process endpoint running in the Atom Cloud. Don’t forget the cloud application will need to provide AtomSphere web services authentication.

 

This works because nothing is “pushed” to the local Atom. Remember the local Atom is constantly pinging the AtomSphere platform every few seconds to signal it is alive as well as to download any pending messages such as new process deployments, extension changes, and in this case process execution requests. Because this communication is always outbound--originating from the local Atom--there is no need to open inbound ports in the firewall. The net result is an event-based invocation with a few seconds latency.

6 people found this helpful

Attachments

    Outcomes