Developer forum

Hi Mikkel

We are running a sprint these weeks to have DW handle 1.000.000+ products. 50.000 of them in one Group and a total of 10.000 Groups.

You have some things to consider:

• Do you put all products in one group?
• Try disabling eCommerce URLs. We have a performance issue that we are fixing that could be related
• Enable "Enable optimized product retrieval" on the catalog module
• In Management center, ecom, advanced, General, Product - Group display cache. Enable end set cache to whatever. Will give you better performance on load.

In the test and load tests we have done so far this is doable. With a lot of variants we see some potential issues that we are fixing.

Take a look at the points above and let me know the results. We are VERY interested in getting this to work, and we will be glad to assist and make improvements to handle it.

BR Nicolai

Hi Nicolai.

Thanks for the input and good to know that you are working on this. I would put them all in one group, not for a specific reason just because it is easy. Could this also be a problem?

I will work some more on this tomorrow and get back!

Yes - 500.000 products in one Group could be an issue. Mostly in backend I guess. And also in the paging - just today on our Development branch we made an improvement on this specific thing.

Lets keep in touch.

BR Nicolai

Hi Mikkel,

I just tried to replicate your scenario with 500.000 very simple products in one group. Your imported products had no variants I guess?

In my test running on latest code this scenario works well in backend but requires the "Enable optimized product retrieval" that Nicolai mentions above for it to work in front end.

Kind Regards

Christian

Hi guys.
First of all, I wrote I used DW 8.3.0.2 this is not correct I’m actually using DW 8.2.3.8
Today I imported 2.000.000 products. Simple products, no variants and no group relation. Pretty much the same result as with 500.000 products.
1. Application startup takes forever
2. All products in DW backend = timeout
Besides this, everything seems to work.
I have done everything suggested by you Nicolai. The problem regarding 1. is the eCommerce URL’s in the backend. If I disable this the application start is back to normal.
I need to give my boss some feedback, do you have any idea when you would have a fix ready for this

Hi Mikkel,

A lot of optimizations that are in 8.3 are not implemented in 8.2.3.x. So it would be really cool if it was possible for you to try the same procedures with latest 8.3...

Regards /Snedker

Hi again.

Im sorry but i can't invest more time in this. I will put the project on standby and wait untill you get a version out that can handle 2 million products :)

Please let us/me know when you have something ready :)

The only real problem i see is the application startup time caused by the SEO friendly URL stuff so AFAIK we are close ?

Hi Mikkel,

We did the first optimization of the URL problem in 8.2.3.9. So if you just roll your demo site up to this hotfix, you should see a huge improvement in startup time. But no further optimization will happen in the 8.2.3 branch so testing on 8.3 is much preferred.

Kind Regards

Christian

Hi Christian.
Thanks for the feedback.
I noticed this yesterday, because I had to upgrade the site to 8.2.3.11 (due to another bug/case).
Christian or Nicolai could you give me some insight in the other optimizations you are doing ATM and where you are doing these optimizations, pitfalls, focus areas ETC. I am looking for some kind of roadmap / version where DW will be mature and ready to handle this. Our client is on hold until we have something solid and when you do, I can invest more time testing.
Thanks for all help so far.