Looking for best practices for loading large XML based files (up to 1 GB or even more) to be processed by Boomi
can you give us a little more details about your process like is the atom installed on your hardware or in cloud. will there be any logic or processing of the file, if not and its just file transfer then i dont think there will be an issue.
Boomi is best when a document is broken into batches
Having said that for a simple passthrough you can just process any file size. If you plan to do some manipulation on data, I would advice to break them into batches and send to Boomi. I say this because when you send a huge junk of data, higher chances of failure and the whole file at times gets suspended due to probable network issue or any other internet cliche. If you send batches, you can count on higher success rate of those batches. Unlike huge file size you do not have to wait for further analysis on batches of data. And as per my observation, even throughput increases when you process them in batches.
This is what my best practice is
First, thank you for the quick answer (i was thinking about breaking the files but was wondering if Boomi has internal solution for that - specifically for XML based files). Out project is based on Atoms and Molecules that load data and move data between organizations, some of the data comes in big XML files and i was wondering if i need to add external modules (like an XML file breaker) in order to process such files efficiently.
I understand. Before you take any call on breaking a file, you may do a POC to get some metrics on time and performance. I do have some processes that run for few hours while processing a high volume file (without batching) but you have to keep in mind that you apply exception handling in all possible points if time is not constraint.
Retrieving data ...