Importing from multiple XML files

Imar Spaanjaars

Posted on 03/04/2023 10:43:17

Good day,

I have a customer that wants to import ecom data from a nightly generated XML file that they send to us over FTP. The data is quite large and they are splitting up the XML file in multiple files. Is there a way using standard components to treat these files as one from an import perspective? I tried the XML provider and pointed to a folder that contains multiple files but it treats each file as a separate table mapping. Is there some kind of subscriber or other hook I can tap into to combine the files before the import job runs?

(I know that on our end we can handle very large XML files so I could do it with a single file but it seems the sending party cannot deliver this).

Thanks!

Imar

Replies

Dmitriy Benyuk

Posted on 03/04/2023 11:58:44

This post has been marked as an answer

Hi Imar,

there is no such functionality in the xml provider. But if you have the "correct/suitable for DW format" table/tables in the splitted xml files you can try the "CreateMappingsAtRuntime" option, so it would look on the files from the source folder and try to build the mapping on the fly each time the job runs: https://doc.dynamicweb.com/forum/integration/using-xml-provider-to-read-multiple-files-from-a-folder-only-import-first-file-partly?PID=1605.
If that is not possible solution then you can use the applicatin start observer and create the .net folder file watcher class that can be triggered once the folder files are changed - so the subscriber can concatenate the files into one and then the Data Integration job will use the concatenated file at its run.
BR, Dmitrij

Votes for this answer: 1

Imar Spaanjaars

Posted on 06/04/2023 17:59:58

Thanks, that sounds like a possible solution.

Imar

You must be logged in to post in the forum

Developer forum

Importing from multiple XML files

Replies