AnsweredAssumed Answered

Performance Batch Count with Data Process

Question asked by ankur.bora249354 on Sep 25, 2018
Latest reply on Oct 2, 2018 by ankur.bora249354

While executing a process to query a database , map the record set  to flat file and  then create a .csv file to a FTP location , the execution time was found to be high when  record count is larger i.e. more than 100 K.

When the data base operation is Configured for grouping options , with a batch count set , the response time found to be faster  (  close to 10 times as shown execution speed below) . Execution Speed Batch vs non Batch


The output files generated ( as shown here )  in both the cases are same - implying that there is no data loss.


Output files Batch vs non Batch



The process flow as shown below where a data process shape is connected after the data operation is set with batching


Using Batch Count With Data Process


The question is - is it a recommended practice to apply data process  when batch count is used ? It appears that batch count when set , is splitting the query into multiple  queries and the data process shape is combining them back together into a single file.  This feature is found to be very useful , can others share similar experience specially on retrieving records from complex DB query.  Additionally , can anyone share the underlying flow specially the role of data process shape ?