AnsweredAssumed Answered

Processing Multiple Source Data - Boomi vs Database

Question asked by milowski@ohsu.edu on Jan 17, 2018
Latest reply on Jan 18, 2018 by milowski@ohsu.edu

Our organizations is heavy Oracle Database, SQL, and PLSQL driven.

I'm fairly fresh to Boomi based development and currently working on the project which involves consuming 3 large (about 100,000 records each) data sources, performing some data manipulation and records matching at the end producing one consolidated database record.

I'm considering various approaches and wondered if some of the Boomi experts out here have any opinions:

  • Direct Database reads
    • get all data from source A as a batch
    • save it to the destination table
    • get data from source B and check if the record already exists in the destination; if so perform an update if not do the insert
    • get data from source C and check if the record already exists in the destination; if so perform an update if not do the insert
  • Cache the data
    • read source A and cache it
    • read source B and cache it
    • read source C and cache it
    • create a process that reads the cached records and does correct transformations and records matching
    • Insert the record into destination table
  • Use PLSQL processing
    • read source A and save it to destination table
    • read source B
    • call PLSQL procedure which will do all heavy lifting of data transformation and matching rules and eventually save the destination table
    • read source c
    • call PLSQL procedure which will do all heavy lifting of data transformation and matching rules and eventually save the destination table

 

So obviously I'm most comfortable to jump into PLSQL procedure to do all processing, however, this really limits Boomi to just pass-through mechanism. I believe it can do more and I would like to make it happen in Boomi but I'm bit concerned about best approach vs performance. Thoughts...?

Outcomes