rene.klomp

OData Client Connector Adds Concurrency Support [V4 Only]

Blog Post created by rene.klomp Employee on Apr 4, 2018

The Open Data Protocol (OData) is a data access protocol built on core protocols like HTTP and commonly accepted methodologies like REST. The protocol enables the creation and consumption of REST APIs, which allow clients to publish and edit resources, identified using URLs and defined in a data model, through the mapping of CRUD (Create, Read, Update, Delete) operations to HTTP verbs.

 

 

Resource Operation

OData leverages the following HTTP verbs to indicate the operations on the resources.

  • GET: Get the resource (a collection of entities, a single entity, a structural property, a navigation property, a stream, etc.).
  • POST: Create a new resource.
  • PUT: Update an existing resource by replacing it with a complete instance.
  • PATCH: Update an existing resource by replacing part of its properties with a partial instance.
  • DELETE: Remove the resource.

 

Basic operations like Create (POST) and Read (GET) obviously do not pose any challenges around concurrency. It is perfectly alright to have multiple concurrent transactions creating or reading resources or individual entities. However, it is different for updating and deleting.

 

Concurrent updating of a resource by two transactions that try to update the same resource at the same time can lead to problems like 'the lost update problem' - the second transaction will overwrite the update made by the first transaction and so the first value is lost to other concurrently running transactions that need to read the first value. These transactions will read the wrong value and will end with incorrect results.

 

OData V4 and Concurrency

OData V4 supports ETag for Data Modification Request and Action RequestThe ETag or Entity Tag is one of several mechanisms that HTTP provides for e.g. web cache validation, which allows a client to make conditional requests. This allows caches to be more efficient, and saves bandwidth, as a web server does not need to send a full response if the content has not changed. ETags can also be used for optimistic concurrency control as in the case of OData, as a way to help prevent simultaneous updates of a resource from overwriting each other.

 

An ETag is an opaque identifier assigned by a web server to a specific version of a resource found at a URL. If the resource representation at that URL ever changes, a new and different ETag is assigned. Used in this manner ETags are similar to fingerprints, and they can be quickly compared to determine whether two representations of a resource are the same.

 

From the OData V4 Protocol Specifications (Ref. OData Version 4.0. Part 1: Protocol Plus Errata 03 ):

 

11.4.1.1 Use of ETags for Avoiding Update Conflicts
If an ETag value is specified in an If-Match or If-None-Match header of a Data Modification Request or Action Request, the operation MUST only be invoked if the if-match or if-none-match condition is satisfied.

The ETag value specified in the if-match or if-none-match request header may be obtained from an ETag header of a response for an individual entity, or may be included for an individual entity in a format-specific manner.

 

So for every update we must realize that the ETag value may be out-of-date, so when you try an update request, always first use the GET request to get the ETag of the specified entity.

 

Boomi OData V4 Connector

In the latest release of the OData Client Connector concurrency is now supported [V4 Only] through the mechanism as described above. Both headers if-match and if-none-match have been added as a configurable option 'Concurrency Mode' in the OData Client Operation. The Concurrency Mode determines how any supplied ETags should be evaluated. Furthermore, ETag support has been added for the OData Client Connector as a Document Property.

 

 

Example Use Case - Microsoft Navision 2017

As part of a full Boomi platform POC that involved our Master Data Hub (or 'Hub') with sources Salesforce, SAP and Navision (2017) we implemented the typical Channel Updates synchronization process by taking the Process Library synchronization templates and specifying these for the various sources. For the Channel Updates (for the 'Contact' Domain) that need to be updated in Navision we end up with the following typical (POC) 'CUD' process:

 

 

When we zoom in on the Map shape doing the transformation from the Hub (MDM) XML profile - fetched as part of the channel updates - to the JSON profile that we need to construct as part of the OData Update Request we see the following:

 

 

Naturally we recognize all the typical field mappings for the Contact domain as defined by us in the Hub on the left-hand side to Navision on the right-hand side. More important in the light of this article we further zoom in on the 'editLink and ETag' function that will build the editLink and will get the ETag for the Entity at hand and set the ETag Document Property accordingly.

 

 

We use a Connector Call as part of the second branch in this function to retrieve the ETag for the Entity we want to update and set the ETag Document Property accordingly:

 

 

Now when a Data Steward makes a change in one of the 'Contact' Golden Records - perhaps to solve a Data Quality Error - a channel update for Navision will be pending (as in this case Navision is setup as a source that will accept channel updates):

 

 

Now, the Data Steward changes the Phone number for the 'Contact' Golden Record in the Master Data Hub from +31243880000 to +31243882222...

 

 

As said, this generates Channel Updates including one for Navision:

 

 

When we now execute the Fetch Channel Updates process for the Contact Domain in AtomSphere we can see the following...

First of all, we see the Channel Update going down the expected path or branch - the one that takes care of updates:

 

 

Second of all we can inspect the response document to see the update was actually applied!

 

 

Also, inspecting the Atom logs, we can see how indeed the ETag nicely got applied in the If-Match header as configured as part of our OData Client Connector Operation!

 

sendRequestHeader] >> PATCH /xxx_xx9999nl_a/ODataV4/Company('Acceptatie%20Xxxxxxxx%20Xxxxxx')/Contact('C00004') HTTP/1.1
sendRequestHeader] >> Accept: application/json;odata.metadata=full
sendRequestHeader] >> Content-Type: application/json;odata.metadata=none
sendRequestHeader] >> If-Match: W/"JzMyO3VoTUFBQUo3LzBNQU1BQXdBREFBTUFBMEFBQUFBQUE9MTA7MTI5NDk0NjU4OTA7Jw=="
sendRequestHeader] >> OData-MaxVersion: 4.0
sendRequestHeader] >> OData-Version: 4.0
sendRequestHeader] >> Content-Length: 206
sendRequestHeader] >> Host: nlxxxx99xxxxxx.xxxxxxxxx.com:8048
sendRequestHeader] >> Connection: Keep-Alive
sendRequestHeader] >> User-Agent: Apache-Olingo/4.3.0.boomi2
sendRequestHeader] >> Authorization: NTLM XXXXXXXXX==

 

 

Ref. Some of the basic statements around OData as a standard were taken from wikipedia.org and odata.org.

Outcomes