Performing Complex Data Transformations using Boomi

Boomi has been an IPaaS leader for many years. A lot of this achievement is attributed to the fact that the AtomSphere platform is a low-code development tool which has an intuitive graphical user interface. Some customers think that due to the ease of use of Boomi, it isn’t possible to do complex data transformations. Here is an example of how Boomi is able to manage complex transformations without writing a single line of code. There were 2 use cases provided to me.

 

The First Use Case:

The first use case was one where they had given a XML data structure with mock data which had teachers and the subjects which they are teaching, and students who had specific hobbies. What the client wanted me to do was to systematically match the student who had the same hobbies as the subject the teacher was teaching.

 Here is the source XML:

Figure 1 - Original (“Source”) Payload

This XML has two nested structures one for “teachers” and one for “students”. The teachers teach a certain “subject”. The students have a “hobby” which correlates with the subject. This XML needs to be transformed to the following structure:

 

Figure 2 - Desired Payload Output

 

Solution:

 

Here is the process which performs the orchestration from the source payload to the desired output:

Figure 3 - Transformation Integration Process.

1. The first branch uses a map shape (“MAP Source to Students XML”) to transform data into an intermediary profile which will be stored in a document cache. 

           
 

The purpose of this is to use the subject as a primary key and to group the students accordingly.

              Figure 4 - Mapping the Student Hobby to Student Key.
 

In this map shape, the student hobby is being mapped to the student key, which is used as the primary identifier when stored in the doc cache, and all the students who have the same hobby will be grouped together. The output looks like this:

Figure 5 - Desired Data Structure going into the  Doc Cache.

 

The Document Cache allows you to temporarily hold data during runtime, and within the scope of the integration. For more information please see: https://help.boomi.com/en-US/bundle/integration/page/c-atm-Document_Cache_components.html. 

             

Figure 6 - Doc Cache with Student Data. 

 

2. In the first map shape of the second branch: 

       
 

The same mapping is used to group teachers as per the subjects, with the output looking like this:

Figure 7 - Grouping of Teachers via the Hobby.

 

3. In the second map shape on branch 2:  

     
 

There is transformation of this transient data structure to my final desired output:

Figure 8 - Transform the Temporary Data Structure into the Desired Output. 

This is where you can see how powerful Boomi can be. You are able to perform data aggregation from various sources, the doc cache being an example of one. On the left, you’ll notice that there is data coming from the doc cache within the source profile itself, and now it just becomes a matter of looking up the students using the course/hobby as a key.

 

The Second Use Case:

The second use case provided by the customer was where they had a data set which consists of different books they had in an inventory. They wanted to split this data set and separate those which have been released after 2004. This use case includes complex data transformation such as mathematical functions, carrying property values and decision logic prior to mapping. 

Figure 9 - Process for Calculating the Cost of Books

 

Here is the source JSON:                                             

Here is the target JSON:





 

First Branch:

Starting off the process, we can set Dynamic Process Properties (“Set publishedAfter property”) in Boomi which hold a particular value during runtime execution.
   

 

In this scenario we’re holding the value “2004” since this is the parameter we’re using to validate whether the books were published before or after.

 

Second Branch:

 

1. Next step in the branch is to preemptively set the values of the currency ratios in a JSON profile which looks like this:

Figure 10 - Data Structure with the Currency Ratios. 

 

2. The Data Process shape splits it into its respective currency type, and then sends the payload off to a sub-process. This sub-process sets all of the currency ratios as Dynamic Process Properties, which will be used later in data transformation when I perform mathematical functions.

 

Third Branch:

 

1. In the third branch I have a decision shape, validating whether the books are published after or before 2004. If they are after 2004, I want to then transform that JSON data structure into an XML profile and perform mathematical functions. 

 

2. Next I have a data process shape to combine the JSON after the decision shape. What this does is combine all the books published after 2004.

 

3. This is how this map looks:

Figure 11 - Map to Calculate the Price of the Book             


 

As you can see, in the middle there is a user-defined function (“FUNC: Calculate Price”) where users can create their own custom function to assist in data transformation. This is a really powerful tool, especially when data transformation requires multiple, complex steps.  Inside the function, you’ll find many different functions which are in conjunction with each other:

Figure 12 - User Defined Function to Calculate the Price for each Currency Type. 




 

Both of these use cases are highly complex situations where normally you would require a lot of coding knowledge and development-specific skill-sets. However, using Boomi, you are able to circumvent this and go straight to building out your integration process with minimal to no knowledge of coding. What’s important at the end of the day is to understand the underlying logic of the problem or transformation regardless of what development tool you use. But with Boomi, development is a lot easier and faster, making your complex integrations robust.