AnsweredAssumed Answered

Sending an XML listing as multiple XML objects for web service server

Question asked by chris.c on May 31, 2017
Latest reply on Jun 1, 2017 by jason_r_walsh667087

I am downloading a large number of big files from an AWS S3 bucket. To download the files efficiently I am using the S3 connector to perform a Bucket listing (retrieved as XML), splitting the listing into in batches of Objects to be downloaded from S3 and then sending them over an http client connector to a second process to perform the downloads.

 

I am using document batching and flow control to begin multiple controlled downloads of S3 files across our molecule. The end result is several running processes across the molecule, each with a queue of large files to download and process.

 

I am trying to get the initial S3 listing and batching process to take advantage of the "Multiple XML Objects" Input Type on the Web Services Server operation for the download process. This should allow us to manually re-trigger any file download individually within a batch from within Process Reporting.

 

Currently I am unable to get the XML that is passed between the 2 processes to display as more than one document in Process Reporting.

 

Is this just a case of needing to get the XML profile right?

 

Example S3 request submitted for download:

 

<?xml version='1.0' encoding='UTF-8'?><s3ObjectListing>
<s3ObjectSummary>
<key>segment_map/part-r-00010.gz</key>
<fileName>part-r-00010.gz</fileName>
<isDirectory>false</isDirectory>
<lastModified>2017-05-30T15:10:20-04:00</lastModified>
<size>125829120</size>
<bucketName>b2b-sandbox</bucketName>
</s3ObjectSummary>
<s3ObjectSummary>
<key>segment_map/part-r-00011.gz</key>
<fileName>part-r-00011.gz</fileName>
<isDirectory>false</isDirectory>
<lastModified>2017-05-30T15:10:28-04:00</lastModified>
<size>125829120</size>
<bucketName>b2b-sandbox</bucketName>
</s3ObjectSummary>
<s3ObjectSummary>
<key>segment_map/part-r-00012.gz</key>
<fileName>part-r-00012.gz</fileName>
<isDirectory>false</isDirectory>
<lastModified>2017-05-30T15:10:36-04:00</lastModified>
<size>125829120</size>
<bucketName>b2b-sandbox</bucketName>
</s3ObjectSummary>
</s3ObjectListing>

 

 

Request profile (one of a few variations I've been trying):

 

 

Display each document to be downloaded within in a batch:

Outcomes