Lucene is killing my import

Martin Nielsen

Posted on 26/05/2011 09:11:01

Hi DW,

I've created an activity for the import/export module that imports some products from a CSV file.
In my activity i create a Product object for each of my products and call the .Save() method.
I have around 5500 products and when i run my import it takes around 10-12 minuts to complete.

I've done some profiling to figure out whats taking time in my rutine, and i noticed that this function ProductIndexer.HandleDataItemModifications
which is called in Products.SaveAndConfirm() is eating 8½ minuts of my total import time.
From what i can see it's the method ProductIndexEntry.CreateQueueEntry inside HandleDataItemModifications
that's taking it's time.

Can i disable Lucene during import, and then just run complete Index update once i'm done?

Regards
Martin

Replies

Pavel Volgarev

Posted on 26/05/2011 09:29:36

Hi Martin,

Thank you for the information. This bug will be fixed in the nearest release (bug number #6574).

-- Pavel

Pavel Volgarev

Posted on 26/05/2011 09:35:03

Btw, are you sure that there is a ProductIndexEntry.CreateQueueEntry which is the bottle neck? Could you send me the report for the profiler?

Thanks in advance.

-- Pavel

Pavel Volgarev

Posted on 26/05/2011 09:52:41

Also, please make sure that automatic updates are turned off (Management Center -> eCommerce -> Advanced configuration -> Searching -> Scheduled updates -> uncheck "Automatic updates").

-- Pavel

Martin Nielsen

Posted on 26/05/2011 11:46:20

Hi Pavel,

"Automatic updates" is unchecked.
Btw. i sent you the profiler report to your e-mail.

// Martin

Pavel Volgarev

Posted on 26/05/2011 12:42:00

Hi Martin,

Since it's more like a feature request I had to obsolete the bug that I mentioned about earlier in this thread. Instead the new product backlog item has been created:

#6575 "Add an ability to disable sending out of notifications to the indexer when product is saved"

-- Pavel

Martin Nielsen

Posted on 27/05/2011 08:47:18

Hi Pavel,

The solution might be a new feature, but the problem is a bug.

The problem i'm seeing is this:
I have an activity that imports products from a CSV file (approx. 5800 products).
I import the products one language at a time, which means my activity runs 4 times, each time saving products with a different languageID (one for each of my 4 ecom languages)

What i'm seeing in that when i chain up my imports, to run one after another the runtime for each activity just goes haywire.

First run (DK): 10 min
Second run (DE): 15 min
Third run (FR): 45 min
fourth run (GB): 1h45min

The same amount of products is imported, but they just take longer and longer to finish.

I'll do some profiling for the entire run today, but something seems to be wrong.

Regards
Martin

Martin Nielsen

Posted on 27/05/2011 09:00:39

I can see that i can empty the queue from the administration interface, can i do that via the API?
Maybe that could be a workaround for me.

Pavel Volgarev

Posted on 27/05/2011 09:04:51

Hi Martin,

var queue = Dynamicweb.Searching.IndexManager.Current.GetIndexerQueue("Products");
queue.Clear();

-- Pavel

Martin Nielsen

Posted on 27/05/2011 11:46:11

I've run import with at the end of each import.

var queue = Dynamicweb.Searching.IndexManager.Current.GetIndexerQueue("Products");
queue.Clear();

And i now have stable times across each run.

// Martin

You must be logged in to post in the forum

Developer forum

Lucene is killing my import

Replies