RE: Exception Out Of Memory (BTS04 Uses .NET1.1 DOM Mapper)
From: Mike Watson (MikeWatson_at_discussions.microsoft.com)
Date: Fri, 18 Feb 2022 09:49:05 -0800
I was going to reply to your original post, but have not had time until now...
I have been in the same situation with you on this matter. I have been
talking with the MS Support and they have confirmed that this is an issue.
I do not know if you can adapt the workaround that I found works great for me.
I had an orchestration that called, via the Sql Adapter, a stored proc. The
size of the Xml file was approx 12Mb. This then needed to be mapped into a
single flat file (approx size 5mb). The mapping was simple (apart from a
As an example:
Source Schema :
RootNodeA (Sql Response Schema)
Destination (flat file schema)
The map linked like this
As said above, the System.OutOfMemory issue was happening in a 12mb Xml file.
After doing some tests I found that removing the complexity of the source
schema gave better results.
Firstly, I changed the Sql Query, instead of returning data via select *
from where … inner join, outer join etc.. ( this returning, via Sql Xml the
Xml hierarchy as seen in the example Source schema, above). I populated a Sql
Server #temptable and returned this.
So BizTalk built a Xsd based on : Select * from #temptable (notice no joins
Doing this return the following XSD format…
New Source Schema
Now, replacing the old source schema with the new one enabled me to build
and deploy a map that could process up to 100MB Xml, producing 40Mb flat file
(on a 1mb memory, p4 box in 8mins). The map had formatting, logical
functoids, custom xslt and looping functiods added without any problems
whatsoever. I only tested to 100Mb as we have no requirements to use data
over 60Mb (Xml file), but I am sure that 100mb+ is possible.
So, to conclude : If possible try to remove the hierarchy in your source and
destination schemas as much as possible this will help you processes the
As you can see, this is a workaround, not a fix. And, like I said earlier,
you may be able to adapt this idea to your own situation.
> I have run into OutOfMemory Exception in BTS04 when mapping. My files are
> max'd at 90,000 rows, (roughly 10mb) any higher and it “WILL” throw
> OutOfMemory exception in map.
> I have been back & forth w/ MS on this matter and we all have concluded that
> it is the .NET framework mapping object. It uses DOM to map everything in
> First, any good application now & days doesn’t use DOM. We learned this back
> in 01-02. I am amazed that MS still haven't fixed this, even after newly
> released .NET1.1 Service Pack. What were they thinking?
> Second, I asked MS if this is going to be fixed in BTS04, and their answer
> was they were looking into some customizations in the 2006 release. Uh, 2006!
> You’re kidding. They have barely released BTS04 and now they say that there
> is a memory cap and won't be explored until the next release!
> For a long time, I hated MS for these exact reasons. After the whole .NET
> world was released they started to redeem themselves. They actually looked
> like they were getting their act together. I guess no matter how they
> advance, their true colors always show.
> Folks, correct me if I am wrong but if you are running a business using
> conversions, you are bound to go over 20mb. Now, MS tells us that the
> software has limitations in size. I have a written/certified email that
> specifically states if you have a file size of over 10mb in BTS04 that you
> are trying to map; it may/will throw OutOfMemory and eventually suspend the
> MS has been saying from day one that BTS04 is completely redesigned to
> handle large messages. (Stated directly in the BTS04 unleashed book) What
> they fail to say is that this is the case only if you are not planning on
> doing any mapping. If you are, then the files will have to be chunked quite
> small and you loose the whole purpose of why you were implementing a
> conversion solution to begin with. Mapping is KEY, and one of MS solutions is
> to not map if you don't have to. (Declared in writing by MS)
> It just goes to show the level MS is at. What they really should state in
> their disclaimer is that we write software for consumers, and not businesses.
> If you want business level processing, then seek other products such as
> BridgetGate TEI, BEA, or an IBM product.
> I am not writing this just to criticize MS for a cheap/bad design. I am
> writing this to help spare the next business architect from going through the
> same amount of wasted time & energy I have spent. Roughly 4mo before finally
> finding out this limitation! What a waste!
> P.S. The custom pre-processing that they explain as one work-around doesn’t
> always work. If you have a large message that chunks, it will still throw an
> error. Not, to mention what happens if you use a flat file with sub-records.
> Jason W