Developer forum

Forum » Ecommerce - Standard features » Freetext-search in product repository

Freetext-search in product repository

Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Hi,

I have a solution running DW 9.7.2, where I would like to make a freetext-search on the product repository.

I've made a summary field on the index, named Freetext. This field is used on the query with a "Contains" - see attached images.

It works fine as long as you search like "fitting safeseal", but not with "fitting with safeseal".

Would this be possible at all with the Lucene index or should I go with the weighted search instead, which has the ability to search on part of the search query?

--
Best regards
Kurt Moskjaer Andersen

lucene-freetext.jpg lucene-query.jpg

Replies

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Kurt,

 

Does your search parameter take a string[] (array) or a string? What you show on the setup seems accurate, and you can definately do that with Lucene.

 

I am assuming that "fitting with safeseal" is the Product Name or Synonym field.

 

Best Regards,

Nuno Aguiar

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Hi Nuno,

My search parameter takes a string array (string[]).

I haven't got any products with the product name "fittings with safeseal" though - only "fittings, safeseal".

Best regards
Kurt

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Hi Nuno,

Regarding search variations, I also need the search to return the product number 71.9923.610, when searching for 719923610 or 71 9923 610.

I've looked into the ability to create my own list of stop-words (with the file stopwords.txt in the Repository-folder), but don't know if that's the way to go.

/Kurt

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply
This post has been marked as an answer

Hi Kurt,

 

  • To figure out the "fitting" + "safeseal" issue, I'd look deeper, using LukeNet to see if the values are properly indexed and analyzed. I don't see how it could be wrong.
    Does it work if you search by a single word only?
    Oh...  try changing the Operator from a Contains to a Match All
     
  • Searching for the numbers will be harder
    I recommend an IndexBuilderExtender that takes the ProductNumber and removes special characters.
    • Create an array
    • Append the ProductNumber replacing periods/dots with an empty string
    • Append the ProductNumber replacing periods/dots
    • Store the array in a new field
    • Add that new field to the summary (not sure if you won't need to use it in a new expression instead of the summary field)

 

Since you're using "Match All", you should be able to get the results you want.

 

Best Regards,

Nuno Aguiar

Votes for this answer: 1
 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Hi Nuno,

No, it doesn't work with a single word either, but will try your suggestions.

Regarding your suggestion with the IndexBuilderExtender, it sounds like the way to go.

Thank you very much.

--
Best regards
Kurt

 
Nicolai Pedersen
Reply
This post has been marked as an answer

Your parameter needs to be String and not String[].

String will be "analyzed" on its way in just like your freetext field will be.

You can run a ?debug=true in your querystring to see the actual query being run. Make the above change, check if it works, and if it does not, run the debug and paste the query here.

BR Nicolai

Votes for this answer: 1
 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

I've made the changes, but without any luck.

I've searched for "fittings with safeseal" and the query is like this: +Active:True +((+Freetext:fittings* +Freetext:safeseal*)) +(+*:* -(+(+ID:prod*)))

The site-language is Danish though and when making a search for the danish terms "fittings med safeseal", the word "med" is added to the query too, as this is not a stopword.

/Kurt

 
Nicolai Pedersen
Reply

So you are trying to find products that does not have a product id that starts with "prod" and that is case sensitive...?

with is left out of the search becauase it is a stopword.

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Yes, correct Nikolai - don't see the case-sensitive part though

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Well, got it working now, if I use my own file stopwords.txt containing "med".

@Nikolaj, if I want to use a custom list of stopwords like above, do I need to include the default ones as well or are they implicit?

/Kurt

 
Nicolai Pedersen
Reply
This post has been marked as an answer

if you create your own stopword list, lucene will not use its own.

Votes for this answer: 1
 
Nicolai Pedersen
Reply
This post has been marked as an answer

It will be case sensitive as Lucene is that. But when you analyze a field, the terms will be lowercased and also the value being searched. So if productid is not analyzed, it will be case sensitive.

Votes for this answer: 1
 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Hi Nuno,

Do you have an example of how to use the IndexBuilderExtender?

I cannot find any examples in the community or the docs.

/Kurt

 
Nicolai Pedersen
Reply
This post has been marked as an answer

Hi Kurt

Here is an example: https://doc.dynamicweb.com/forum/ecommerce-standard-features/ecommerce-standard-features/facet-options-for-variant-show-without-variant-option

Happy coding!

Votes for this answer: 1
 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Perfect, thanks :)

/Kurt

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

I've made an IndexBuilderExtender, compiled it and put it into the bin-folder. After building the index, the new field is not present in the index, if I inspect it in Luke. Shouldn't it be?

My code is show below:

    public class ProductIndexExtender : IIndexBuilderExtender<ProductIndexBuilder>
    {
        public void ExtendDocument(IndexDocument indexDocument)
        {
            if (indexDocument.ContainsKey("Number"))
            {
                string productNumber = (string)indexDocument["Number"];

                productNumber = productNumber.Replace(".", "");

                if (!indexDocument.ContainsKey("ProductNumberCustom"))
                {
                    indexDocument.Add("ProductNumberCustom", productNumber);
                }
                else
                {
                    indexDocument["ProductNumberCustom"] = productNumber;
                }
            }
        }
    }
 
Nicolai Pedersen
Reply
This post has been marked as an answer

Try changing your inherits statement to 

IndexBuilderExtenderBase<ProductIndexBuilder>

instead of 

IIndexBuilderExtender<ProductIndexBuilder>

I can see you got your code from the course. But I think it is an old example and that it is wrong... I will have the docs updated.

Votes for this answer: 1
 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Well, it doesn't change much, I'm afraid - the field is still not present in the index shown in Luke.

indexbuilder-1.jpg indexbuilder-2.jpg indexbuilder-3.jpg
 
Nicolai Pedersen
Reply

Maybe you placed the dll in the wrong /bin?

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Just checked again to be certain, but it is the correct folder :)

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

Hi Kurt,

 

You have to manually add a field in the index. Here's how to do it

https://www.screencast.com/t/UupG8NPgRaJi

 

Best Regards,

Nuno Aguiar

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

A strange thing is, that if I omit the check for the key "Number" in the IndexDocument, the field is populated into the index:

           if (!indexDocument.ContainsKey("ProductNumberCustom"))
            {
                indexDocument.Add("ProductNumberCustom", "1234");
            }
            else
            {
                indexDocument["ProductNumberCustom"] = "1234";
            }

But as soon as I add the check for the Number-field in the IndexDocument, it is not populated:

            if (indexDocument.ContainsKey("Number"))
 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Hi Nuno,

I've already done this :)

/Kurt

 
Kurt Moskjær Andersen
Kurt Moskjær Andersen
Reply

Well, I've found the reason for this. The "Number" field is not called this, but instead called "ProductNumber" :)

/Kurt

 
Nuno Aguiar Dynamicweb Employee
Nuno Aguiar
Reply

I see, on the IndexBuilderExtender. Good catch. Hope the next ones are easier smiley

 

You must be logged in to post in the forum