Developer forum

How many fields do you have...?

Hi Nicolai,

3320 ProductCategoryFields from which ~ 300 are Reference fields.

Before you tell me they are "too many fields", you have to understand that they have a marketplace integration where they need to supply specific fields to that marketplace whilest documenting their products according to their needs. The Marketplace has about 1300 fields defined.

They have a lot of categories and subcategories with very different product types. Hence the need for different fields.

Will they end up using all of them? I don't know yet and probably they don't know it either.

You can have a look here: http://atomico.dotfusion.ro/

The solution holds 2 webshops, each webshop with it's own catalog and fields. There are about 20.000 products in the shop right now.

The solution is now hosted on a shared machine with a shared SQL Server Standard database.

I am looking forward to hear your opinion.

Thank you,

Adrian

Of course you have 3300 fields... You know my opinion!

We will have no quick solution to this one. We will have to completely rewrite the UI of queries to support something like that...

When creating indexes, instead of just adding all fields, add only those you need. That will make the list of fields much shorter in query editor. Also you can limit the complexity of the queries so you do not have that many fields in it. At least use a summary for the free text part.

On my machine it is acutally not too bad - the Total-Products.query which seems to be the largest, it takes maybe 5 secs to load.

Since the list of fields is populated to each criteria and all criteria can be edited in one screen, it will make a large DOM. We would have to editing of criterias one by one in seperate screens to solve this issue.

BR Nicolai

Hi Nicolai,

I am well aware of your opinion, and that's why I hesitated to post this :)

However, PIM is supposed to help them document the products. The old interface worked well as performance but it lacked the search functionality on the fields.

In the query I have only defined the fields that are category specific and will be used for filtering.

I have even simplified it since they initialy have defined fields for each Level 3 category and we had a lot of duplicates. I have consolidated them at Level1 and got rid of about 2500 fields.

The summary field is kept to a minimum. Just name, number and Description.

You should try Atomico-Products.query. That is the largest one. In my tests, it does not matter the number of Parameters as the number of Query expressions and the number of fields.

I guess that populating the dropdown options for the left-hand rules is the one that's killing the mojo. And that load is increasing with the number of query expressions.

Maybe limit the initial number of options in the dropdown could do the trick while the Search in the field would get access to the whole list. Just a thought.

Thank you,
Adrian

Hi Adrian

It is more the number of fields in the index that is the problem. If you add all 3340 fields to the index, they will show up in the selectboxes in the query editor. But you only query on maybe 200-ish fields, so if you just include only the fields in the index that you actually need to search, the problem would probably go away.

Your query Atomico-Products.query contains 207 parameters, 239 group expressions or type "or" with a total of 336 expressions. I cannot imagine that query would even work or perform.

You might want to consider another design. I cannot imagine what the idea is and anyways our search is not exactly designed for that kind of scenario. You should consider something built for extreme scenarios.

BR Nicolai

You might want to explore 9.8 dynamic facets feature. See dump.

That will let you support facets and hence filtering on an index on fields that has not been specified in either query or facet definitions.

BR Nicolai

Hi Nicolai,

I have only added in the queries the fields they will use for filtering.

THis is a usual scenario when a customer has a large variety of products with multiple "relevant" fields. 

I know that "Relevant" is very relativ as they might think something is relevant for their customers when in fact it just clutters the interface and ruins the experience.

But you probably also know that customers are stubborn and very hard to reason with and they always "know better" no matter how many years you have under your belt.

One option would probably be to exclude the Marketplace fields from the index. That would decrease the number of fields included in the Index, at least.

The Dynamic Facets feature sounds very good. I will look into it.

Thank you,
Adrian

Well, it will not perform.

Best of luck!

Hi Nicolai,

In this case, you should define the metrics for "performing".

If this would come as an offical recommendation from the vendor, I can take it to the customer and say they cannot use 3000 fields for their product definitions and they should look somewhere else for a solution.

While this explanation could make sense for a regular ecommerce solution, I don;lt think it will work for PIM solutions.

I have a prospect that is looking to implement in PIM the entire ETIM standard. Compared with ETIM standard, my situation here is a mere joke.

Based on your feedback, I should stop any further talks with them and just dismiss the project because we won;t support that number of fields.

And I am totally fine with that as long as there is an official information about limitations. Something like : "We can't support more than xxxx fields and xxxx product categories and xxxxxx products". Or we can support it but you need XXX Azure machines with x configuration.

I know I am often seen as a wild goose chaser but in this particular situation, this guys have a justified need to use those fields. I agree I might have used the wrong approach or the wrong tweaks but this is no reason to dismiss this situation as it will not happen again.

I will try the approaches you have suggested hoping that it might improve the performance.

Thank you,

Adrian

Hi Nicolai,

I can't find anything about the dynamic facets.

The tutorial for defining facets is based on the regular approach.

Is there somewhere documented? Maybe in a video?

Thank you,

Adrian

You can setup group based facets on the group - see: https://doc.dynamicweb.com/documentation-9/ecommerce/product-catalog/product-groups#2689

Dynamic facets here: https://doc.dynamicweb.com/documentation-9/indexing/indexing-search/facets#8689

Let me know if you need more guidance.

BR Nicolai

Hi Nicolai,

I was about to submit a similar post, but was just waiting to find the time to provide better repro steps and data, since I guess I don't need that anymore. I see both sides of this issue and although I don't come across that volume of fields, I do come across painful/slow UX in the Repositories with as little as 55 custom fields and no category fields and under 50 expressions (still working on optimizing them, but nonetheless).

I started noticing a decrease in performance when the "enabled/disabled" flag for each expression was introduced, and now another one when the search in dropdowns was introduced. I am suspecting a JS memory leak processing that. Doing a quick DevTools Performance capture, you can see:

• Almost half a million nodes were rendered
• More than 55.000 listeners
• Initializing the select (dropdown) took a good chunk of time

Here's the screencast - noticed that I stopped the recording for more than a minute. The amount of time for this experience during recording was longer than normal because the browser was also busy recording and the computer recording the screencast as well, but it's noticebly slow under "normal" conditions

https://www.screencast.com/t/kZc1d1uewo

I understand that maybe instead of dropdown, we need probably a modal window that is rendered once and then updates the field name and prepopulated when it's triggered?!? Maybe that will only render that dropdown once in the DOM and avoid all of that duplication?

Any optimization on the backend will be welcomed.

Best Regards,

Nuno Aguiar

Hi Adrian and Nuno

Looking at this, I see 3 seperate issues

1. the slimselect javascript based search in select box decreases performance. With many expressions and fields and combinations, it becomes very slow or even dies. We will make an update to not load the javascript component if there are many fields and/or many expressions. That will move the boundary for when it becomes unresponsive, but it will probably still not work with 1000s of fields.
2. One of the other performance problems (frontend) on Adrians solution comes form the facet definition which contains a lot of dimensions. When all of this is in one facet file, all facets will be calculated (how lucene works), and the only way to fix that is by not having that many facets in one go. Solution to that problem is to create multiple facet files and use group based facets maybe along with Dynamic facets to ensure that only part of the facets are calculated on each pageview. On the catalog paragraph only select one facet file with limited number of general facets.
3. No matter the change we will conduct in 1, the query window will probably still break. To solve it, only index needed fields to limit select box length and also consider splitting the query in multiple query definitions instead of one big one.
4. Bonus: Do not use contains extended.

BR Nicolai

Hi Nicolai,

Yep, I agree with all of that. Good to know the js will be addressed and I know there's a threshold for all of this.

I guess the other thing we/Dynamicweb should probably do is to update the docs with some sort of statement as "Performance may be affected whenever you have more than X expressions or X fields in your index, ..."

I know there are no magic number, but being part of the official documentation help us support the claim when talking to customers, specially because on the big customers that would look for it and challenge us on it, there's a technical person on their end.

Best Regards,

Nuno Aguiar

Hi Nicolai,

Thank you very much for the update.

I have already started working on #2 and #3 from your list. It makes sense and I have alwayw tried to separate things as much as I could.
Sometimes it is possible, sometimes it is not. But at least know we know the implications.

I try to avoid as much as possible #4. I understand it's a danger to performance.

Back to the point of documentation, I believe it would be good to mention at least the metrics of the testing scenarios and always aim at testing with a slightly higher figures than average. Not as a limitation but rather as a recommendation.

That could give a starting point for anybody and since it's official, it's gonna be easier for us to handle the crazy scenarios we receive from our customers.

Thank you,

Adrian

Hi Nicolai,

Do you have a TFS # for the 1st item on this list? The SlimSelect performance?

The more customers we upgrade to 9.8 the worst it becomes. We can't work with query in the backend :(

Best Regards,

Nuno Aguiar

No

Ok,

Do you need a new thread for that? This is impacting our development because it's becoming almost unmanegable, when it was previously usable.