New index - Boost Best Practices

Nuno Aguiar

Posted on 11/09/2015 01:30:05

Hi,

We are trying to graps the Boost feature in the new index for eCommerce but cannot make it work 100%. We have multiple issues with exclusion (of fields in the schema extender) and sort

Assumptions:

We are using the Schema Extender, hence no control over boost
We created an additional field for each one the client requested boost/priority value for
- There were around 12 fields, so we set the most important to 1000 and worked our way down in decrements of 10, so the last one had roughtly 850

Result:

I get the results I was expecting them to be sorted according to the prominence
rowlf.dw-demo.com/search-results?Summary=Joao - Since I am searching for "Joao", I expected to see the results inverted, since the 2nd one starts with Joao
http://rowlf.dw-demo.com/search-results?Summary=News+of+Great+Joy! - In this case, products with an exact match should be sorted first
http://rowlf.dw-demo.com/search-results?Summary=I+Will+Sing+Praise - same as above
http://rowlf.dw-demo.com/search-results?Summary=I+Shall+Not+be+Afraid - the scenario is the expected one, by I am guessing dumb luck, because the issue is the same as the 2 above. I am assuming the 3rd result is in third due to the boost factor
Boost does not seem to work 100%
http://rowlf.dw-demo.com/search-results?listview=&Summary=Mark+Hayes - This search should sort products with "Mark Hayes" in composers field (boost=990) on top of all other products except with "Mark Hayes" as the product name as that is not the case. Check attachment

Related issue:

Since I had to set fields (duplicated) in the query to set the boost, I figured it made sense to exclude them from the schema extender, but once I did that, the index returned nothing. I even tried it with AutoID (which seems harmeless) because it's never used anywhere, and still got an empty template. As soon as I delete the exclude options, it works. Using the query sandbox I validated the index did exclude them
The only field I managed to exclude with no consequences was PriceType

Given my issues what are the best practices to set up these scenarios?
Using 8.6.1.14

Best Regards,

Nuno Aguiar

Replies

Nicolai Høeg Pedersen

Posted on 15/09/2015 15:54:16

Hi Nuno

Introduction to how Boost works in Lucene: http://blog.imaginea.com/understanding-lucene-boosting-part-1/

It also has something to do with the analyzer on the fields - and lucene is taking a lot of things into consideration. Your search probably uses "Contains" for the free text search - and it does not look for the exact string. If you need to do that, you need to add the title to a field that is not analyzed - and boosted - and make a search that is an exact match.

Your searches above have a lot of stop words - of, i, be etc. could also have an affect. Your first example - not sure prominence is taken into consideration.

BR Nicolai

Nuno Aguiar

Posted on 22/09/2015 00:04:02

Hi Nicolai,

I am not an expert in Lucene, so I am struggling a little bit with this. I will get better in the future, but for now I need a few straight answers on what possible/not possible if you could help me.

The search is using contains on a handfull of fields which seems to be the proper approach. We did create all of the necessary fields without being analyzed and with proper boost values according to the client's specification. We don't seem to get the expected results, maybe because of the the boost racio is between the fields don't allow the fields we are expecting to be leveraged as much as we want.

We also tried excluding fields from the Schema extender, since that could be an issue (to avoid duplicate data based on the ones we had to create manualy not to be analyzed), but that stops the index from rendering anything (using 8.6.1.16)

The usage of stop words is something we might not be able to control because those examples represent actual product names and that would be the expected user behaviour, or am I missing something?

I understand prominence makes sense from the end user's point of view and would be a nice feature to implement.

What do you think I could change/experiment before going back to the client?

Best Regards,

Nuno Aguiar

Nicolai Høeg Pedersen

Posted on 22/09/2015 16:29:35

Hi Nuno

What to do depends on what you want. And I'm not clear about that.

Do you want that a search on an exact title would be on the top? And then what with the rest of the search result?

What should happen if I search for "Great Joy" and you have a book called "News of Great Joy" and a book called "Great Mountain Joy"?

The way you have made the search here, makes it very difficult to control what happens.

You search a lot of fields in an or statement, see screen 1. You should instead create a summary field on the index and use that for general search (see Screen 2 + 3).

Also on the index, create an extra field for the product name without analyzer and search it using a match all (see screen 4 + 5)

BR Nicolai

Nicolai Høeg Pedersen

Posted on 22/09/2015 16:29:51

And here comes screen 5

Nuno Aguiar

Posted on 22/09/2015 16:34:50

Hi Nicolai,

Thanks this helps a lot. Being a new module, I am still trying to find the best way to do things. We will give it a go and see the client is happy with these results.

Best Regards,

Nuno Aguiar

You must be logged in to post in the forum

Developer forum

New index - Boost Best Practices

Replies