How to Create a Do-While Loop in the Process using a Custom Counter

Document created by sheng_liao462475 Employee on Aug 26, 2016Last modified by Adam Arrowsmith on Oct 23, 2017
Version 5Show Document
  • View in full screen mode

Use Case

This design pattern is used when you need to execute a series of process shapes repeatedly until some condition is met before continuing with the rest of the process logic. This is similar to a "Do While" loop in traditional programming. To control the number of iterations of the loop, you can use a custom process property as a "counter" variable.

 

Some common scenarios for this type of looping include:

  • Asynchronous job processing: When you need to submit an external job request (for example, requested a large data report/extract), repeatedly query the status of that job until the job is "complete" before retrieving and processing the result.
  • Design Pattern: Handling API Pagination: When using a technology connector such as the HTTP Client to extract records from an API, repeatedly retrieve pages of results until all records are collected before processing those records.

 

This pattern should NOT be used to simply loop through documents individually. Remember process shapes inherently loop through and execute for each document within a group. For situations in which documents should be executed individually (e.g. one at a time), use the Flow Control shape.

This is an advanced pattern and can result in run-away process executions if not designed and configured carefully. You should be familiar with process execution, design, and troubleshooting before implementing.

 

Approach

  1. Set an initial value for the counter property.
  2. Check if the counter value has exceeded some maximum value.
    1. If no...
      1. Perform the logic to repeat.
      2. Increment the counter value.
      3. Connect the line back to the previous shape that checks the counter value.
    2. If yes...
      1. Continue with the rest of the process logic as required.

 

Implementation

The two options below focus on how to increment the counter using scripting or native map functions. In practice, these techniques would be embedded within a process specific to your scenario.

 

Option 1: Use custom scripting to increment the counter within the loop

The following screen shot illustrates the process flow and is not fully configured.  You will need to configure the process shapes accordingly.

 

  1. Initiate the process flow.
  2. Use a Set Properties shape to initialize the value of a dynamic process property to use as the counter.
    1. Note: In this situation a dynamic process property is preferred over a Process Property component because it is easier to reference the property via scripting.
  3. On branch 1, use a Decision shape to determine if you need to continue the loop.
    1. On the True path, use a Data Process shape with a Custom Scripting step (see script below) that retrieves the current property value, increments by 1, and updates the property with the new value. Note the script increments the property once for the group of documents, not once per document.
    2. On the False path, you could either connect to a Stop shape or the shapes you need for the process.

  4. On branch 2, you can add the shapes you need for the process.

 

Increment Dynamic Process Property Counter Groovy Script

import java.util.Properties;
import java.io.InputStream;
import com.boomi.execution.ExecutionUtil;

// Retrieve current Dynamic Process Property value
counter = ExecutionUtil.getDynamicProcessProperty("counter");

// Convert string value to int to do math
int intValue = Integer.parseInt(counter);

// Increment value by 1
intValue = intValue + 1;

// Convert int value back to String
counter = Integer.toString(intValue);

// Set the dynamic process property
ExecutionUtil.setDynamicProcessProperty("counter", counter, false);


// Pass documents through...
for ( int i = 0; i < dataContext.getDataCount(); i++ ) {
  InputStream is = dataContext.getStream(i);
  Properties props = dataContext.getProperties(i);

  dataContext.storeStream(is, props);
}

 

Option 2: Use map to increment the counter within the loop

The following screen shots illustrates the process flow and are not fully configured. You will need to configure the process shapes accordingly.

 

 

 

 

  1. Initiate the process flow.
  2. Use a Set Properties shape to initialize the value of a dynamic process property to use as the counter.
  3. On branch 1, use a Decision shape to determine if you need to continue the loop.
    1. On the True path, use a Message shape to set the current value of the counter property as the current document data. In this example the property value is placed within a simple XML structure:
      <Base><BaseElement>{1}</BaseElement></Base>
    2. Then use a Map shape configured to map that simple XML Profile (create the profile manually) to itself. Use a Math Add map function to increment the value by 1 before outputting.
    3. Use another Set Properties shape to store the updated values from the document's Current Data.
    4.  On the False path, you could either connect to a Stop shape or the shapes you need for the process.
  4. On branch 2, you can add the shapes you need for the process.

 

Usage Considerations

  • This is a recursive pattern. As with recursion in any type of programming you must carefully design and configure the boundary conditions to avoid getting stuck in an infinite loop. Here are a few tips to help mitigate during development:
    • When first developing, test on local atom so you can manually terminate the Atom service if you accidentally fall into an infinite loop.

    • For debugging, output the property value to process log using the Notify shape.

  • IMPORTANT Keep the number of shapes executed recursively to an absolute minimum (this is another reason for using a subprocess). After several hundred shape executions (the exact number will vary based on Atom memory configuration) the process will fail with a StackOverflowException.

10 people found this helpful

Attachments

    Outcomes