Developer forum

Hi Marie Louise

You mentioned BeforeQuery and AfterQuery. These are limited, but BeforeQuery can still alter how the Lucene query is composed, even if it doesn’t expose the final product list. You could:

• Intercept and inject a boosted term for your user’s assortment products.

• Use Lucene’s query syntax to give extra weight to assortment items, like:

  assortmentid:user123^5 OR assortmentid:all

  where ^5 boosts your own assortment in ranking.

Another solution could be to do 2 lists - one for my assortment and one for shared assortment. Maybe in tabs. But that would be somewhat different.

Another solution would be creating a combined sorting key - one column for each customer assortment.

Hi, 

We like the idea of injecting our own lucene query to boots the customers own assortment. But we cannot find any property on the BeforeQuery notificationsubscriber to incject raw lucene syntax.
Can you specify how to get further? 

Here is a sample from our docs:

using System;
using System.Collections.Generic;
using System.Linq;
using Dynamicweb.Ecommerce.Shops;
using Dynamicweb.Environment;
using Dynamicweb.Extensibility;
using Dynamicweb.Extensibility.Notifications;
using Dynamicweb.Indexing;
using Dynamicweb.Indexing.Querying;
using Dynamicweb.Indexing.Querying.Expressions;
using Dynamicweb.Security.Permissions;

namespace Dynamicweb.Ecommerce.Indexing
{
    [Subscribe(Dynamicweb.Indexing.Notifications.Query.BeforeQuery)]
    public class BeforeQueryObserver : NotificationSubscriber
    {
        public override void OnNotify(string notification, NotificationArgs args)
        {
            if (ExecutingContext.IsBackEnd())
            {
                Dynamicweb.Indexing.Notifications.Query.BeforeQueryArgs beforeQueryArgs = args as Dynamicweb.Indexing.Notifications.Query.BeforeQueryArgs;
                var query = beforeQueryArgs?.Query;
                if (query is object)
                {
                    var indexService = ServiceLocator.Current.GetInstance<IIndexService>();
                    var index = indexService.LoadIndex(query.Source.Repository, query.Source.Item);
                    if (index.Builds.Any(build => build.Value is ProductIndexBuilder))
                    {
                        var expressions = new List<Expression>(new[] { query.Expression });
                        var permissionService = new UnifiedPermissionService();
                        var disallowedProductGroupIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
                        var disallowedShopIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
                        var allowedGroupIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
                        var allowedShopIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
                        using (var backendPermission = PermissionContext.Backend())
                        {
                            foreach (List<IdWithDefaultPermissionLevel<string>> permissionQueue in backendPermission.PermissionPriority)
                            {
                                foreach (var userPermission in permissionQueue)
                                {
                                    foreach (string allowedProductGroupId in permissionService.GetAllMatchingKeys<Products.Group>(userPermission.Id.ToString()))
                                        allowedGroupIds.Add(allowedProductGroupId);
                                    foreach (string allowedShopId in permissionService.GetAllMatchingKeys<Shop>(userPermission.Id.ToString()))
                                        allowedShopIds.Add(allowedShopId);
                                    foreach (string disallowedGroupId in permissionService.GetAllMatchingKeys<Products.Group>(userPermission.Id.ToString(), PermissionLevel.None))
                                    {
                                        if (!allowedGroupIds.Contains(disallowedGroupId))
                                        {
                                            disallowedProductGroupIds.Add(disallowedGroupId);
                                        }
                                    }

                                    foreach (string disallowedShopId in permissionService.GetAllMatchingKeys<Shop>(userPermission.Id.ToString(), PermissionLevel.None))
                                    {
                                        if (!allowedShopIds.Contains(disallowedShopId))
                                        {
                                            disallowedShopIds.Add(disallowedShopId);
                                        }
                                    }
                                }
                            }
                        }
                        UpdateExpressions(query, expressions, disallowedShopIds, "ShopIDs");
                        UpdateExpressions(query, expressions, disallowedProductGroupIds, "GroupIDs");
                        UpdateExpressions(query, expressions, disallowedProductGroupIds, "ParentGroupIDs");
                        query.Expression = Expression.Group(false, OperatorType.And, expressions);
                    }
                }
            }
        }

        private void UpdateExpressions(IQuery query, List<Expression> expressions, HashSet<string> disalowedValues, string fieldName)
        {
            var existingExpression = FindPermissionExpression(fieldName, query.Expression);
            if (disalowedValues.Any() && existingExpression is null)
            {
                var g = Expression.Group(true, OperatorType.And, new List<Expression>(new[] { Expression.In(Expression.Field(fieldName, fieldName), Expression.Term(disalowedValues.ToArray())) }));
                expressions.Add(g);
            }
        }

        private BinaryExpression FindPermissionExpression(string fieldName, Expression topLevelExpression)
        {
            var groupExpression = topLevelExpression as GroupExpression;
            if (groupExpression is object)
            {
                foreach (var item in groupExpression.Expressions)
                {
                    var childExpression = item as GroupExpression;
                    if (childExpression is object && childExpression.Negate && childExpression.Expressions?.Count() == 1 && childExpression.Operator == OperatorType.And)
                    {
                        var binaryExpression = childExpression.Expressions?.First() as BinaryExpression;
                        if (binaryExpression is object && binaryExpression.Operator == OperatorType.In)
                        {
                            if (binaryExpression.Left is FieldExpression)
                            {
                                var fieldExpression = binaryExpression.Left as FieldExpression;
                                if (fieldName.Equals(fieldExpression?.FieldName, StringComparison.OrdinalIgnoreCase))
                                {
                                    return binaryExpression;
                                }
                            }
                        }
                    }
                }
            }
            return null;
        }
    }
}

Hi again,

We have figured out how to modify the expressions now but haven't found a way to boost any specific assortment. 

The current code without SYS_ALL assortment which work.
But if I manually try to add 'boost' it doesnt work. Any input to what Ive missed?

Ive also tried to add boost in the loop like this:

BR
Marie Louise 

Create boosted subqueries manually (recommended)

Try something like this: “my assortment” (Assortmetns not in all) and “all” assortment expressions into two groups and manually boost one:

var myAssortment = Expression.In(
    Expression.Field("AssortmentIDs", "AssortmentIDs"),
    Expression.Term("MY_ASSORTMENT_ID")
);

var otherAssortments = Expression.In(
    Expression.Field("AssortmentIDs", "AssortmentIDs"),
    Expression.Term("SYS_ALL")
);

// Create separate groups for Lucene parsing
var boostedMyAssortment = Expression.Group(false, OperatorType.And, new[] { myAssortment });
boostedMyAssortment.Boost = 5.0f; // <- boost applies to group

beforeQueryArgs.Query.Expression = Expression.Or(boostedMyAssortment, otherAssortments);

This should set the Lucene Query.Boost at the group level when the Dynamicweb Expression is converted into Lucene queries.

I cannot access any 'Boost' on the Dynamicweb.Indexing.Querying.Expressions.Expression so I cannot seem to boost there :) 

The installed version of the indexing packages are these:

Ok - we need something like this in our query:
(AssortmentIDs:MY_ASSORTMENT_ID)^0 OR (BoostedAssortmentIDs:MY_ASSORTMENT_ID)^5

What you can do is to create a custom field for assortments that basically have a copy of the regular assortment field and then boos that custom field with customer specific assortment keys.

So first extend the index:

public class IndexBuilderExtenderExample : IndexBuilderExtenderBase<ProductIndexBuilder>{

public override void ExtendDocument(IndexDocument document)
{
    if (document.ContainsKey("AssortmentIDs"))
    {
        var assortments = document["AssortmentIDs"] as string[];

        if (assortments != null && assortments.Contains("MY_ASSORTMENT_ID"))
        {
            // Copy to a special boosted field
            document.Add("BoostedAssortmentID", "MY_ASSORTMENT_ID");
        }
    }
}

}

And then do something like this in the notification subscriber
 

var userAssortmentId = "MY_ASSORTMENT_ID";

var fieldNormal = Expression.Field("AssortmentIDs", "AssortmentIDs");
var fieldBoosted = Expression.Field("BoostedAssortmentID", "BoostedAssortmentID");

var matchNormal = Expression.Equal(fieldNormal, Expression.Term(userAssortmentId));
var matchBoosted = Expression.Equal(fieldBoosted, Expression.Term(userAssortmentId));

var combined = Expression.Group(false, OperatorType.Or, new List<Expression> { matchBoosted, matchNormal });
beforeQueryArgs.Query.Expression = Expression.Group(false, OperatorType.And, new List<Expression> { combined });

// Apply boost to the boosted field only
beforeQueryArgs.Settings.Boost.Add("BoostedAssortmentID", 5f);

Of course the MY_ASSORTMENT_ID is what you find from the current user - and you have more of them maybe?

See if this is possible.

BR Nicolai

I will try.

Its possible for some customers to have multiple customerspecific assortments, which all should be boosted to the top :) 

I ran into a new challenge.
The 'AssortmentIDs' is empty when the indexextender is hit. Its also a generated field from DW, so can it be a issue then?

The extender should run in the end of the document creation process - so you should have the AssortmentIDs field.

Can you check if "Skip assortment fields" flag is not set on the product index builder settings?

We ended up adding a new Field PriceListAssortmentIDs to the index and boosted the field by 30 (probably overkill, but it works).
As you suggested we enriched the new field with data using an IndexBuilderExtender. The BeforeQueryObserver however turned out not to be necessary as the settings in the query configuration screen turned out to be enough to solve the problem by grouping the boosted assortment IDs field (PriceListAssortmentIDs) and the normal AssortmentIDs using the OR grouping.
We had to disable group sorting on the shop page, set the sort value to Score and then it works.

Thank you for the assistance Nicolai

PriceListAssortmentIDs field

Query configuration

IndexBuilderExtender

public class ProductIndexBuilderExtender : IndexBuilderExtenderBase<ProductIndexBuilder>
{
    private static readonly ILogger _logger = LogManager.Current.GetLogger("ProductIndexBuilderExtender");

    public override void ExtendDocument(IndexDocument document)
    {
        try
        {
            if (document.ContainsKey("AssortmentIDs"))
            {
                IEnumerable<string> assortments = document["AssortmentIDs"] as IEnumerable<string>;

                if (assortments != null && assortments.Any())
                {
                    var assortmentIds = new List<string>();
                    foreach (var assortment in assortments)
                    {
                        if (assortment != "SYS_ALL")
                        {
                            assortmentIds.Add(assortment);
                        }
                    }
                    document.Add("PriceListAssortmentIDs", assortmentIds);
                }
            }
        }
        catch (Exception ex)
        {
            _logger.Error("Error extending document", ex);
        }
    }
}

Great you made it work - and eventually very simple. Great way of using boost on the same data in different columns to provide a sorting.

Thank you for sharing the final solution.