Data integration from multiple files

Arnór Halldórsson

Posted on 23/04/2018 19:00:59

Hi,

We have a client that´s using a 3rd party PIM system (InRiver) and when the client makes changes to the product catalogue in the PIM they send us information about the product / group that they changed in the form of XML files. However, this data is usually scattered over multiple files. F.e. let´s say that the client just changed 5 products, and moved them to a group. Then we might get 12 files, where the EcomProducts items might be scattered over 5 different files, the EcomProductCategoryFields items might be scattered over 2 files etc etc...

The import activity is set to import XML files from a source folder, and it works when I manually merge the tables from the xml files into one file where all the items are merged under their respective parent table. But when I run the activity with the files unmerged, it only imports a small fraction of the data and leaves some information out, seems rather random how it chooses wich data get´s imported and wich get´s left behind.

So my question is, does dynamicweb not support importing data from multiple files unless all information for each table is united in a single file and all items are merged under their respective tables?

Replies

Dmitriy Benyuk

Posted on 24/04/2018 10:19:00

Hi Arnór,
in general there is a restriction "only one table must be used per one xml file" when it works fine. However it could be possible to use the
XML->Ecom provider job where you have several EcomProducts xml files in one folder but in that case it works when:
- you have a "manual"/not automatic mapping (fx EcomProducts1.xml ->EcomProducts, EcomProducts2.xml -> EcomProducts)
- your files are not changed from time to time(I mean their quantity/names, so the manual mapping is valid) and shouold have same columns per entity xml file, eg. EcomProducts files should have same columns set (ProductId, Number, Name etc), EcomProductCategoryFieldValue files should have theirs same repeating set of columns, etc
Maybe you can try to write some general xslt to merge your several xml files into appropriate one ?
Regards, Dmitrij

Arnór Halldórsson

Posted on 24/04/2018 13:11:59

Thanks for your response Dimitrij :)

That would work, I think, if I always had a specific set of xml files. But the number of files, and the contents of each may vary from day to day. Depending on how many products the client publishes from the PIM, sometimes I might receive 10 files, sometimes 30, with the naming convention PIMyyyyddMHHmmssFFF.xml. The files arrive over the span of a few seconds and I don´t know what file contains what tables before I open the files.

Do you think that the XSLT script will work in my case? Doesn´t it depend on a fixed set of files, with a fixed model for the contents of each file? Maybe the best solution would be to change the logic of the web service that receives the XML. Instead of just forwarding the XML string from each request to a new XML file, cache the request contents, and after the last request (however I detect that) sort and merge the xml tables and write them to a single file that DynamicWeb integration can read.

Dmitriy Benyuk

Posted on 24/04/2018 14:53:08

Then you need to use the second option with parsing responses to one xml suitable for Dynamicweb format. Maybe you can determine the requests for another product/session by checking the interval between the last file creation(if its a few seconds - its same request, if more - another request, etc)
Regards, Dmitrij

Arnór Halldórsson

Posted on 24/04/2018 14:56:55

That sounds like something that would work. Then merge the contents of the files after all the data has been transmitted and remove the old files.

Thank you very much for your feedback :)

Best regards,

Arnór

You must be logged in to post in the forum

Developer forum

Data integration from multiple files

Replies