How to Retrieve Query/queryMore Results with the HTTP Client Connector

Document created by josh_hutchins778263 Employee on Aug 3, 2017Last modified by Adam Arrowsmith on Aug 11, 2017
Version 9Show Document
  • View in full screen mode

This article describes an example process that retrieves multiple pages of results from an API (such as the AtomSphere platform API) that uses a "queryMore" token approach using the HTTP Client connector.

 

 

Use Case

This article provides a REST implementation that utilizes the query and queryMore functionality for endpoints that only return a limited amount of records and further incorporating the use of a queryToken to return remaining results. This article is a specific example of the general pagination pattern described here: Design Pattern: Handling API Pagination. This particular design pattern will soon be available via the Process Library under the name: API REST Pagination w/ queryMore and HTTP Client.

 

In this example we will be using the AtomSphere platform API, but this approach is applicable to any API with a similar query/queryMore interaction. The specific API URL is: https://api.boomi.com/api/rest/v1/YourAccountID/AccountUserRole/query to return the object records for AccountUserRole. However, because there are a theoretical 165 records to query and because this particular API returns a maximum of 100 records per page, we are going to have to make a second call.

 

Approach

We will be making use of the queryMore call and utilizing a queryToken passed from the first page of 100 records to pull the next page of records. I have created a sample process that will utilize the following shapes to accomplish our goal:

 

  • HTTP Client Connectors: For query and queryMore calls
  • Set Properties: For tagging our documents with our Dynamic Document Property key value for our cache shapes.
  • Add to Cache: For loading each "Page" worth of information top to bottom
  • Decision shape: Determines if we have more records to query based on whether there is a queryToken in our current document's data.
  • Message Shape: Used to separate our queryToken from the main data to pass into our second HTTP Client call.
  • Data Process shape to Combine files
  • Load from Cache: Loading our cached documents top to bottom

 

Implementation

 

 

Here's what we're doing with this process, step by step: 

 

  1. Start: No Data.
  2. HTTP Client: Get: Using the initial connection URL https://api.boomi.com/api/rest/v1/YourAccountID/AccountUserRole/query with Basic Authentication, we will be retrieving the first page of 100 records as per the limit of the API. Because there are more than 1 page worth of records in total, a queryToken value is generated as part of the response data. This queryToken will allow us to request the second page containing our remaining 65 records when passed into our next HTTP Client with the queryMore call later on. 
  3. Branch: 2 branch shapes
  4. Set Properties: Setting Dynamic Document Property RecordSet with a value of 1 to document that passes through
  5. Add to Cache: Store document into Index under Index QueryData, Profile Element Key Combine and Dynamic Document Property of RecordSet.
  6. Decision Shape: Using our queryMoreResult XML Profile and element queryToken as First Value. We are then setting a comparison of "Not Equal To" with a second value of '' . This will pass as true as our current document will contain the queryToken
  7. Message Shape: Passing {1} using our queryMoreResult XML Profile and element queryToken to seperate our token from the rest of the data.
  8. HTTP Client: Get: Using the initial connection URL https://api.boomi.com/api/rest/v1/YourAccountID/AccountUserRole/queryMore and passing the queryToken into connector to pull the second page of 65 records . 
  9. Branch: 2 branches
  10. Set Properties: Setting Dynamic Document Property RecordSet with a value of 1 to next document that passes through
  11. Add to Cache: Store second document into Index under Index QueryData, Profile Element Key Combine and Dynamic Document Property of RecordSet.
  12. Decision Shape: Using our queryMoreResult XML Profile and element queryToken as First Value. We are then setting a comparison of "Not Equal To" with a second value of '' . This will pass as false as our new second document will not contain a "queryToken" as it is the last page of records. 
  13. Load from Cache: Index QueryData, Profile Element Key Combine and Dynamic Document Property of RecordSet with a static value of 1 (The value we attached to our documents via set properties shape)
  14. Data Process: Combine Documents:Using our queryMoreResult XML Profile, we'll be combing data based on the Result element. This combine is very handy because it will total our @NumberOfResults attribute (100 from first document, 65 from second document) and remove our queryToken element as it is neither the root element, nor an element contained in our combine hierarchy. 
  15. Your choice! Do you want to send this into a connector? Do you want to return this document to a master process?

 

 Below is the request and response profile that will be used for the two HTTP Client Connectors, as well as the Add and Load cache shapes:

 

1 person found this helpful

Attachments

    Outcomes