Developer forum

Forum » Integration » Importing address takes a very long time

Importing address takes a very long time

E-mail notifications
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply
Posted on 23/04/2024 16:31:58

Hi there,

I have an XML file with user and addresses. When I trim the file to 2K users and 3K addresses, the import completes in under 20 seconds. When I expand the file to 45K addresses, the job never seems to complete (or at least, it's been running for the past 25 minutes). Is there something in the address importer that would explain this exponential increase in run time? Is 45K and accepted amount of addresses to import?

Thanks!


Replies

 
Dmitriy Benyuk Dynamicweb Employee
Dmitriy Benyuk
Posted on 23/04/2024 18:13:07
Reply

Hi Imar,
if you are using the UserProvider then it tries to find the existing users for the importing addresses and to fill the correct user ids (that is needed when the existing user is searched by name, customer number, email, external id) so this probably takes most of the time.
But if it takes so long than we need to look on how to improve that
BR, Dmitrij

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 23/04/2024 21:35:01
Reply

It eventualy completed but it took over 25 minutes. Looks like an N+1 issue, so agreed that it would be good to see how it could be improved.And what's worse, I think there was a table lock or other perormance issues while the import was running I couldn't even get a SELECT COUNT(*) FROM AcessUserAddress to work in SQL Server.

Are you interested in seeing the import file? I could send it to you.

Imar

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 23/04/2024 21:36:45
Reply

Also, until then, would I be better off if I used the DynamicWeb provider with a table script to map from the customer number in the XML to a DynamicWeb AccessUser ID?

 
Dmitriy Benyuk Dynamicweb Employee
Dmitriy Benyuk
Posted on 24/04/2024 08:46:37
Reply

Hi Imar,
yes, it would be nice to have that file to reproduce the issue, so you can send me a link to it.
Try to use DynamicwebProvider meanwhile.
BR, Dmitrij

 
Dmitriy Benyuk Dynamicweb Employee
Dmitriy Benyuk
Posted on 02/05/2024 11:46:45
Reply
This post has been marked as an answer

Hi Imar,
there is a new version of the UserProvider where this issue is fixed:
https://www.nuget.org/packages/Dynamicweb.DataIntegration.Providers.UserProvider/4.0.2
BR, Dmitrij

Votes for this answer: 1
 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 02/05/2024 16:57:44
Reply

Thanks Dmitriy, the import went down from 25+ minutes to a litle over 1 minute. Much better!

Imar

 

You must be logged in to post in the forum