Developer forum

Forum » Integration » UserProvider - limits?

UserProvider - limits?

Aki Ruuskanen
Reply

Hi,

I have several thoussand users to import for a temporary campaign. I have imported files with 500 or so users without problems. Now I tried with 3000 users and I get a timeout error after 30 seconds saying "Job Failed with the following message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding."

Are there som limits to how many users can be imported?

Regarsd / Aik


Replies

 
Morten Bengtson
Reply

Hi,

Try changing the executionTimeout (seconds) in web.config

<httpRuntime executionTimeout="600" maxRequestLength="512000" useFullyQualifiedRedirectUrl="false" minFreeThreads="8" minLocalRequestFreeThreads="4" appRequestQueueLimit="100" enableVersionHeader="true" requestValidationMode="2.0" requestPathInvalidCharacters="&lt;,>,*,%,:,\,?" />

Does it take about 10 minutes (executionTimeout default) before the timeout error occurs or does it happen sooner?

Note: If you set this number to something higher than the sessionTimeout value you can get some weird errors.

 
Aki Ruuskanen
Reply

I get the error after about 30 seconds.

Log:

 

04-02-2013 17:04:23 Job Failed with the following message: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.  
04-02-2013 17:04:23 Finished job - AceImport.  
04-02-2013 17:04:23 Batch failed.  
04-02-2013 17:03:49 Starting job - AceImport.  
04-02-2013 17:03:49 Importing data to table: AccessUser  
04-02-2013 17:03:48 Starting batch

 

 
Morten Bengtson
Reply

The default sql command timeout is 30 seconds, so my guess is that the provider fails to dispose some resources like sqlconnection, sqlcommand or sqldataadapter, which will lead to timeouts.

Someone at DW should probably take a look at it.

 
Aki Ruuskanen
Reply

The module cannot even import a file with 1500 users without getting a timeout.

Any suggestions how to import users? 

 
Jonas Dam
Reply

Hey Aki,

Sounds like we didn't increase the timeout on the connection on the userProvider.

I'll get it fixed for the next service release.

In the mean time, I'm afraid the best way to handle it is to split up the file in to smaller portions, and import them one at a time.

If this solution cannot work for you, and if you are in a hurry, please contact our support, and we'll try to work out a solution.

/Jonas

 
Jonas Dam
Reply

Hi Aki,

I have just tested an import on 10000 simple rows on my development machine, and they fly through in a few seconds.

Is it possible for you to send me a copy of the CSV file containing your users, and the activity you've set up, so that I can try and recreate the problem?

The activity file can be found in the folder files\Integration\jobs.

My email is jkd@dynamicweb.dk

 

thanks, Jonas

 
Morten Bengtson
Reply

I just created a new import using the UserProvider but get an exception when I try to edit the import job (NullReferenceException).

The log contains no useful information.

Then I tried again - same result. Is there a checkbox for "make userprovider work" ;-)

I followed the example used in the documentation of UserProvider.

Other providers work without any issues (DW 8.2.1.0).

/Morten

 

 
Aki Ruuskanen
Reply

Hi Jonas,

Thanks for the reply. I'll send you the files.

Regards / Aki

 
Jonas Dam
Reply

Morten, that sounds strange - I have just tried creating, running and editing import activities using CSV source and User destination, both on my development machine and on one of our test servers runing 8.2.1.0.

Can you give me some more information, maybe a url if it's running on a publicly available server, or maybe a copy of the activity you're trying to edit?

 
Aki Ruuskanen
Reply

I have noticed other bugs also in the Data Integration (8.2.0 with swedish admin language). For example you cannot edit a CSV source after the activity is created. You get a message saying that the folder does not exist.

/Aki

 
Morten Bengtson
Reply

Hi Jonas,

It is a fucked up test-site that has been upgraded lately. I will try setting it up on a clean install later and see if that works :)

If the error still occurs I can send you a copy of the files.

 
Aki Ruuskanen
Reply

I upgraded to 8.2.1 and seems to work better. At least on files with 1500 rows. I'll give it shot with larger files.

/Aki

 
Jonas Dam
Reply

Heh - anything can happen on one of  those :-)

If you continue to see problems, you should definitely submit a bug-report (or send the info to me, and I'll add a bug report).

 
Aki Ruuskanen
Reply

Are there any plans to fix this. Tried to do a userimport again with the latest version of DW with the same result. A timeout after ten minutes or so...

Regards / Aki

 
Jonas Dam
Reply

Hi Aki,

What version are you using? 

The timeout should be increased to 30 minutes for user imports in version 8.2.1.3 and all versions after that.

Regards, Jonas

 
Aki Ruuskanen
Reply

I'm currently using 8.2.1.6.

 

 
Jonas Dam
Reply
This post has been marked as an answer

Argh! I see that when I last looked at this, the test import finished in just under 10 minutes on my development machine, so I missed a second timeout setting which set the timout to 10 minutes instead of 30 seconds.

This is absolutely my mistake, and I apologize.

I have now increased the timeout to 60 minutes in the UserDestination, and made sure that there are no more timeout settings in the code that overwrite this setting.

We will be sending out a release either today or tomorrow, and the fix will be included in this release.

 

Votes for this answer: 1
 
Aki Ruuskanen
Reply

OK, great. The I'll wait a bit so I don't have to split my file to a number of smaller files. :)

Regards / Aki

 
Aki Ruuskanen
Reply

Using 8.2.1.8. Still the same error.....

 
Jonas Dam
Reply

Hi Aki,

I'm sorry to hear that you're still having problems.

I have just tested the import on several differnet machines, including Test machines that are set up clean, and my own developer machine, using version 8.2.1.8.

The import of all 40000 users took about 20 minutes on the slowest machine, but I didn't see any timeout issues, so I am fairly confident that the timouts set in code has been increased to 60 minutes.

I have sent you an email with a link to a website running 8.2.1.8, which has completed the import in 15 minutes without problems.

Maybe you can look at this test site, and see if you can recreate your problem there, so that I can see it?

/Jonas

 

 
Aki Ruuskanen
Reply

OK, I'll try that.

/Aki

 

 

You must be logged in to post in the forum