AnsweredAssumed Answered

Data mix/loss when splitting large XML Documents

Question asked by dvanmierlo on Apr 13, 2018
Latest reply on Apr 29, 2018 by Sjaak Overgaauw

During a test where large XML documents are split, I noticed that data gets mixed and results in data loss.

I have created a simple test scenario to replicate this issue.

 

An XML document is read from disk, split by its Destination element. All split documents are written back to disk. Each test file contains the exact same Destination elements. The number of StockItem elements vary based on the file size.

With small files, like 100KB, there is no issue. With large file, 15-20MB, initial it seems to work fine. But after repeated re-runs, the last split document contains the Destination data of the first split document. The other split documents are fine.

 

The result is that the first split document is occurs twice and the Destination data of the last split document is lost.

 

The test is perfomed by executing the deployed test process on an Atom on my local laptop. The specs of my laptop are:
   Dell Inspiron 5567
   Windows 10 Enterpise
   CPU Intel i5-7200U 2.50GHz
   RAM 8GB
   SSD Samsung 850 EVO 250GB (C: 142GB, D: 100GB)

During the test the RAM usage was 75% and remains the same during the test. The CPU usage was about 30% during the test. Before and after the test it was between 0% and 3%.

 

The Atom VMOptions are:
   -Djava.endorsed.dirs=D:\Boomi AtomSphere\Atom - Local Atom Dennis/endorsed
   -Xmx512m
   -Dsun.net.client.defaultConnectTimeout=120000
   -Dsun.net.client.defaultReadTimeout=120000
   -Dsun.net.http.retryPost=false
   -Dcom.boomi.container.securityCompatibility=2017.06
   -Dfile.encoding=utf-8

 

How can this issue be solved?

 

 

Outcomes