Developer forum

Forum » Dynamicweb 10 » Tips for working with large amounts of Facets

Tips for working with large amounts of Facets

Justin Sjouw Dynamicweb Employee
Justin Sjouw
Reply

Hi,

On a project I'm working on there will be a PIM structure with a large number of Attributes, which are all potentially a facet in the frontend. Around 20K attributes/facets are expected

A lot of attrributes are already imported into DW PIM, and the facets are organized in to a couple hundered Facet Groups. I noticed that assigning the facet groups to the Product Catalog App takes quite a while (and it has also crashed before). 

Any tips you can give for working with these kinds of numbers, to try and be ahead of any further performance issues?

I appreciate it's a broad question, but I'd rather ask it too soon than too late ;-)

Thanks.

Justin


Replies

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Hi Justin

I think I need to see the solution, but I am pretty sure we do not have the tools so you can support this scenario. The facets in Dynamicweb is maybe designed for a couple of hundred facets in total at a maximum and you are way beyond that.

You should not assign all the facet groups to the same product catalog instance as that will cause all facets to be calculated on all queries and your solution would not perform.

We need some kind of feature that can add the facets dynamically based on the context you are in. Something down the line of "Fully dynamic facets" described here:

https://doc.dynamicweb.com/documentation-9/repositories/faceting/dynamic-facets

I think the current issue is that this model has to be handled in your template. 

In order to understand the need, I need to understand the logic behind your implementations. How are your products organized, why do you have 20.000 facets, what facets should be in a given search result and why.

BR Nicolai

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Hi Justin

Just talked to Jeppe about this - we would like to have a look. My post above might not be 100% accurate...

You should be able to define a facet group on a product group - and then the catalog should pick that one up. On the catalog you then only have a 'fallback' facet group selected. It might not work as intended - we need to look into that.

That said, It might still not be enough - but we would like to see your solution and dig into possible solutions.

BR Nicolai

 
Justin Sjouw Dynamicweb Employee
Justin Sjouw
Reply

Hey NP,

Thanks for your quick reply. I'll dm you to set something up.

A bit of background is that we are dealing with an ETIM structure, so there are quite a lot of unique attributes per ETIM Class. So although there is not a huge number of products (around 6K) there are quite a lot of attributes, which should all potentialy be used as a facet.

Thanks,

Justin

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Justin and Nicolai,

 

I'd love to hear about the final solution, if you can all share. We sometimes come across large amounts of fields and facets (nothing as big as this though), and I am asked about the limits and solutions we can find.

 

BR,

Nuno Aguiar

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

I will have a look. But it sounds like a solution design that should be re-visited.

It will never make sense to have 20000 facets on 6000 products.

 
Adrian Ursu Dynamicweb Employee
Adrian Ursu
Reply

Hi guys,

I am also interested in hearing about it. We have a potential project using the ETIM standard and it would be useful to understand the limitations.

We have implemented an ETIM standard previously but have asked the customer to limit the number of facets. We have also managed to use reference fields for a large part of the properties, significantly reducing the number of facets. But even in this case, you can still end up with a large number of facets.

It would be good to know the limitations (best practice) of the current system as well as potential alternatives (Elastic search or anything else)

Thank you,
Adrian

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

We have some mentions of limits in this article: https://doc.dynamicweb.com/documentation-9/repositories/faceting/dynamic-facets

You can create a facet group with a max of 15 facets for each product group (either top-level or for each sub-level product group) and then have a limited set of facets (app 7 facets in one group) that applies to all products.

Then add a facet group to each product group with the relevant facets - that would give you 20-25 facet groups on product lists and goin above that level will cause you issues, first and foremost in performance, but also on UX.

As mentioned there might be an issue related to this specific feature on the product catalog for viewmodels.

Also we are exploring the idea to dynamically add the facets based on the datamodel/category specifed on the current group - also so you do not need to setup that many facet groups - as that can be done automagically.

BR Nicolai

 
Adrian Ursu Dynamicweb Employee
Adrian Ursu
Reply

Hi Nicolai,

Thank you for the clarifications.

The Dynamic approach would be extremely helpful. I am looking forward to it :)

Thank you,
Adrian

 

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

@Justin - you have not send anything yet. At least I have not received it.

 
Wiktor Wieczorek
Reply

Hello,

I am working on a project which has dynamic facets, there is 88105 books authors:
Right now there is a limit of 100 facet options displayed at a time

I was asked if it's possible to display all of the authors. Like you said here I suspect it may cause performance issues.
If it's possible, please let me know how to change the limit.
Also, I would appreciate any other advice. I thought about adding some search for the facet options.

Best regards
Wiktor

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Wiktor,

 

Worst case you can do some custom code to get all of the distinct values, instead of having that be a facet. That may solve some performance issues, but you'll still have a big one, which is the generated markup.

 

Browsers may begin to struggle because of having to paint the necessary elements to render 88105 checkboxes and labels. You will not get a good result!

 

What you can do instead, is to have a typeahead dropdown (similar to free text search) that, even if it gets the distinct list of authors, would simply return the top X of the search criteria, and everytime you click on one, it would add the value (similar to tags). That would perform much better.

 

Best Regards,

Nuno Aguiar

 

You must be logged in to post in the forum