Developer forum

Forum » Development » How fast should DW be able to index 45000 product?

How fast should DW be able to index 45000 product?

E-mail notifications
Martin Nielsen
Reply
Posted on 12/03/2015 08:50:06

Hi guys,

We're building a new version of rather larger eCom solution, and on the old version we build our own Solr index for our facets and searches because DW didn't perform to our liking at the time.

Now here 2 years later i'm testing out the built in Lucene index again, but i'm seing really slow indexing times, and also frontend is slow. (Compared to our solr implementation)

Our data:

45000 products (Split over 5 languages)

56 custom product fields

10 multi select product categori fields with about 15-25 fields options each.

Indexing time:

On Lucene i'm currently at 70 minutes and it's still not done.

On Solr we can index our products in about 10-15 minutes.

 

My machine is an I7 with SSD, so i'd assume these measures would be even worse on the production server.

Can i do something to optimize our database somehow?

 

The above test are done on 8.6.0.4.

 

// Martin


Replies

 
Nicolai Høeg Pedersen
Posted on 12/03/2015 13:32:06
Reply

Hi Martin

Are you running on the new index or the 'old' one?

BR Nicolai

 
Martin Nielsen
Posted on 12/03/2015 13:45:12
Reply

I'm on the old one i guess. I'm not using the new repository module, it's still a little to beta for me :)

 
Nicolai Høeg Pedersen
Posted on 12/03/2015 13:48:57
Reply

Hi Martin

The new module would index this in a matter of few minutes.

We would very much like you to use that version and are willing to through some hours at you to help with the implementation.

BR Nicolai

 
Martin Nielsen
Posted on 12/03/2015 15:32:38
Reply

Hi Nicolai,

Our primary reason to go back to the standard index in Dynamicweb was to get acces to SmartSearch because the new website needs a lot of specific productslists that we cannot handle with simple groups.

Our deadline for the project is May 1. so is it reasonable to think that repositories will be able to work with smartsearches?

Some of our other concerns regarding repositories are that a lot of the configuration is moved away from the Product catalog module and into the management center, and this means, that we loose a lot of flexibility.

An example:

- We have a product list with 5 filters. With "Repositories" we'd need an index and a facetgroup with 5 fields/filters.
- If we need to use the same productlist on a page but only show 3 filters, we'd have to created new index with a new facetgroup with 3 filters.

Deciding what facetgroup to show on a given productlist should be on the paragraph module

 

Also, will repositories perform better than the current implementation when rendering the frontend?
AFAIK the current bottleneck with facets is calculation the facetcount for each option, has anything changed here with the new implementation?

 

We're could be interested in using Repositories, but it really depends on how fast the new repositores can come to par with the current Lucene index in regards to features.

 

I've only looked into Repositories at TechConf, so my assumptions are based on that, a lot could have happened since. Feel free to call me on 99 28 72 75 and we can have a chat.

// Martin

 

You must be logged in to post in the forum