AnsweredAssumed Answered

Is S3 Get Object limited to 1000 Documents?

Question asked by ryan.hurdsman280071 on Nov 13, 2017
Latest reply on Nov 14, 2017 by ryan.hurdsman280071

I am pulling an Object Listing from S3, then processing through the listing performing a S3 Get Object.  It appears that the S3 Get Object is only processing the first 1000 documents when 9018 documents are inbound Shape Source Data.  Any Ideas on what I am doing wrong?

 

2017-11-13T21:33:04Z INFO Start Jet Capital Amazon S3 Connection: amazons3 Connector; New Amazon S3 Connector Operation Executing Start Shape
2017-11-13T21:33:04Z INFO Start Jet Capital Amazon S3 Connection: amazons3 Connector; New Amazon S3 Connector Operation Parameter 1 value = Production/DefiServicing/Transactions/
2017-11-13T21:33:11Z INFO Start Jet Capital Amazon S3 Connection: amazons3 Connector; New Amazon S3 Connector Operation 1 document(s) found for processing.
2017-11-13T21:33:11Z INFO Start Jet Capital Amazon S3 Connection: amazons3 Connector; New Amazon S3 Connector Operation Shape executed successfully in 6505 ms.
2017-11-13T21:33:11Z INFO Data Process Executing Data Process Shape with 1 document(s).
2017-11-13T21:33:11Z INFO Data Process [1] Data Split: XML Executing with 1 documents in
2017-11-13T21:33:13Z INFO Data Process [1] Data Split: XML Split resulted in 9018 document(s).
2017-11-13T21:33:13Z INFO Data Process [1] Data Split: XML Completed with 9018 documents out
2017-11-13T21:33:13Z INFO Data Process Data Process Execution (DATA_SPLIT) produced 9018 documents.
2017-11-13T21:33:13Z INFO Data Process Shape executed successfully in 1571 ms.
2017-11-13T21:33:14Z INFO Connector Jet Capital Amazon S3 Connection: amazons3 Connector; Get Object Executing Connector Shape with 9018 document(s).
2017-11-13T22:38:42Z INFO Connector Jet Capital Amazon S3 Connection: amazons3 Connector; Get Object Shape executed successfully in 3929567 ms.
2017-11-13T22:38:42Z INFO Business Rules Executing Business Rules Shape with 1000 document(s).
2017-11-13T22:38:47Z INFO Business Rules Shape executed successfully in 4553 ms.

Outcomes