RE: Exception Out Of Memory (BTS04 Uses .NET1.1 DOM Mapper)

Tech-Archive recommends: Repair Windows Errors & Optimize Windows Performance

From: Mike Watson (MikeWatson_at_discussions.microsoft.com)
Date: 02/18/05


Date: Fri, 18 Feb 2022 09:49:05 -0800

Hi,

I was going to reply to your original post, but have not had time until now...

I have been in the same situation with you on this matter. I have been
talking with the MS Support and they have confirmed that this is an issue.
I do not know if you can adapt the workaround that I found works great for me.
I had an orchestration that called, via the Sql Adapter, a stored proc. The
size of the Xml file was approx 12Mb. This then needed to be mapped into a
single flat file (approx size 5mb). The mapping was simple (apart from a
looping functoid).

As an example:

Source Schema :
RootNodeA (Sql Response Schema)
        ElementAA
        NodeB
                ElementBA
                ElementBB
                ElementBC
                NodeC
                        ElementCA
                        ElementCB

Destination (flat file schema)
        RootNode
                NodeA
                        ElementAA
                        ElementAB
                        ElementAC
                        ElementAD
                        ElementAE
                        ElementAF

The map linked like this

Source Destination
ElementAA ElementAA
ElementBA ElementAB
ElementBB ElementAC
ElementBC ElementAD
ElementCA ElementAE
ElementCB ElementAF

As said above, the System.OutOfMemory issue was happening in a 12mb Xml file.
After doing some tests I found that removing the complexity of the source
schema gave better results.

Firstly, I changed the Sql Query, instead of returning data via select *
from where … inner join, outer join etc.. ( this returning, via Sql Xml the
Xml hierarchy as seen in the example Source schema, above). I populated a Sql
Server #temptable and returned this.
So BizTalk built a Xsd based on : Select * from #temptable (notice no joins
etc..).
Doing this return the following XSD format…
New Source Schema
RootNode
        ElementA
        ElementB
        ElementC
        ElementD
        ElementE
        ElementF

Now, replacing the old source schema with the new one enabled me to build
and deploy a map that could process up to 100MB Xml, producing 40Mb flat file
(on a 1mb memory, p4 box in 8mins). The map had formatting, logical
functoids, custom xslt and looping functiods added without any problems
whatsoever. I only tested to 100Mb as we have no requirements to use data
over 60Mb (Xml file), but I am sure that 100mb+ is possible.

So, to conclude : If possible try to remove the hierarchy in your source and
destination schemas as much as possible this will help you processes the
large files..….

As you can see, this is a workaround, not a fix. And, like I said earlier,
you may be able to adapt this idea to your own situation.
Cheers,
Mike Watson

"jason" wrote:

> I have run into OutOfMemory Exception in BTS04 when mapping. My files are
> max'd at 90,000 rows, (roughly 10mb) any higher and it “WILL” throw
> OutOfMemory exception in map.
>
> I have been back & forth w/ MS on this matter and we all have concluded that
> it is the .NET framework mapping object. It uses DOM to map everything in
> memory!
>
> First, any good application now & days doesn’t use DOM. We learned this back
> in 01-02. I am amazed that MS still haven't fixed this, even after newly
> released .NET1.1 Service Pack. What were they thinking?
>
> Second, I asked MS if this is going to be fixed in BTS04, and their answer
> was they were looking into some customizations in the 2006 release. Uh, 2006!
> You’re kidding. They have barely released BTS04 and now they say that there
> is a memory cap and won't be explored until the next release!
>
> For a long time, I hated MS for these exact reasons. After the whole .NET
> world was released they started to redeem themselves. They actually looked
> like they were getting their act together. I guess no matter how they
> advance, their true colors always show.
>
> Folks, correct me if I am wrong but if you are running a business using
> conversions, you are bound to go over 20mb. Now, MS tells us that the
> software has limitations in size. I have a written/certified email that
> specifically states if you have a file size of over 10mb in BTS04 that you
> are trying to map; it may/will throw OutOfMemory and eventually suspend the
> message.
>
> MS has been saying from day one that BTS04 is completely redesigned to
> handle large messages. (Stated directly in the BTS04 unleashed book) What
> they fail to say is that this is the case only if you are not planning on
> doing any mapping. If you are, then the files will have to be chunked quite
> small and you loose the whole purpose of why you were implementing a
> conversion solution to begin with. Mapping is KEY, and one of MS solutions is
> to not map if you don't have to. (Declared in writing by MS)
>
> It just goes to show the level MS is at. What they really should state in
> their disclaimer is that we write software for consumers, and not businesses.
> If you want business level processing, then seek other products such as
> BridgetGate TEI, BEA, or an IBM product.
>
> I am not writing this just to criticize MS for a cheap/bad design. I am
> writing this to help spare the next business architect from going through the
> same amount of wasted time & energy I have spent. Roughly 4mo before finally
> finding out this limitation! What a waste!
>
> P.S. The custom pre-processing that they explain as one work-around doesn’t
> always work. If you have a large message that chunks, it will still throw an
> error. Not, to mention what happens if you use a flat file with sub-records.
>
> Sincerely,
> Jason W
> jworkman1@comcast.net
>



Relevant Pages

  • Need help with Flat File assembler
    ... I need to convert an incoming XML file into a flat file. ... I have defined the map. ...
    (microsoft.public.biztalk.general)
  • RE: Convert xml to pipe delimit file in BziTalk
    ... You must be having the sample flat file with pipe delimeter. ... 2-Create the schema using Add the Flat file schema wizard and the sample ... 4-Add the map file to map the source schema to destination flat file schema. ...
    (microsoft.public.biztalk.server)
  • Re: XML To Flat File Error
    ... Now I can get the flat file via your method,but how can I use the schema ... now,my aim is use the biztalk to translate a xml file into a flat file,and ... > You don't need a map to generate the flat file. ...
    (microsoft.public.biztalk.general)
  • Re: Move csv file from one folder to another when output should be xml file
    ... and stored in in the Out folder as a XML file with all the records from the ... The difference is that the output XML file should be according to an example ... Most of the differences between the flat file schema and the output schema ...
    (microsoft.public.biztalk.general)
  • XML to Flat File Error
    ... I am working to convert an XML file to a flat file (because the EDI ... If I set up my schema to be of type Delimited, and have the Child Delimiter ...
    (microsoft.public.biztalk.general)