Developer forum

The product query attached:

<?xml version="1.0" encoding="utf-8"?>
<Query ID="60ce027c-f713-41cb-a36a-eabca4aaae98" Name="ProductQuery.query">
  <Meta />
  <Settings />
  <Source Repository="ProductRepository" Item="ProductIndex.index" Type="Dynamicweb.Indexing.Queries.IndexQueryProvider, Dynamicweb.Indexing" />
  <Expressions>
    <GroupExpression Operator="And">
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Active" />
        </Left>
        <Right>
          <ConstantExpression Value="True" Type="System.Boolean" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="MatchAny" Disabled="false">
        <Left>
          <FieldExpression Field="GroupIDs" />
        </Left>
        <Right>
          <ParameterExpression Name="GroupID" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="LanguageID" />
        </Left>
        <Right>
          <MacroExpression Lookup="Dynamicweb.Ecommerce.Context:LanguageID" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="ShowMasterOrVariantInList" />
        </Left>
        <Right>
          <TermExpression Value="False" DisplayValue="False" Type="System.Boolean" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="In" Disabled="false">
        <Left>
          <FieldExpression Field="Number" />
        </Left>
        <Right>
          <MacroExpression Lookup="Custom.Ecommerce.UserDefinedAssortment:UserDefinedAssortment" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Brand_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="Brand" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Diameter_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="Diameter" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="OuterDiameter_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="OuterDiameter" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Height_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="Height" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="ThreadMetric_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="ThreadMetric" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Length_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="Length" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="ThreadVersion_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="ThreadVersion" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="HeadType_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="HeadType" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Drive_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="Drive" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="Material_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="Material" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="SurfaceTreatment_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="SurfaceTreatment" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="ColorFamily_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="ColorFamily" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="RalNumber_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="RalNumber" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="PackageQuantity_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="PackageQuantity" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="PackageWeight_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="PackageWeight" />
        </Right>
      </BinaryExpression>
      <BinaryExpression Operator="Equal" Disabled="false">
        <Left>
          <FieldExpression Field="DinNorm_Facet" />
        </Left>
        <Right>
          <ParameterExpression Name="DinNorm" />
        </Right>
      </BinaryExpression>
    </GroupExpression>
  </Expressions>
  <ViewFields />
  <ViewLanguages />
  <CompletionRules />
  <CompletionLanguages />
  <Parameters>
    <Parameter Name="GroupID" Type="System.String" DefaultValue="" />
    <Parameter Name="Brand" Type="System.String" DefaultValue="" />
    <Parameter Name="Diameter" Type="System.String" DefaultValue="" />
    <Parameter Name="OuterDiameter" Type="System.String" DefaultValue="" />
    <Parameter Name="Height" Type="System.String" DefaultValue="" />
    <Parameter Name="ThreadMetric" Type="System.String" DefaultValue="" />
    <Parameter Name="Length" Type="System.String" DefaultValue="" />
    <Parameter Name="ThreadVersion" Type="System.String" DefaultValue="" />
    <Parameter Name="HeadType" Type="System.String" DefaultValue="" />
    <Parameter Name="Drive" Type="System.String" DefaultValue="" />
    <Parameter Name="Material" Type="System.String" DefaultValue="" />
    <Parameter Name="SurfaceTreatment" Type="System.String" DefaultValue="" />
    <Parameter Name="ColorFamily" Type="System.String" DefaultValue="" />
    <Parameter Name="RalNumber" Type="System.String" DefaultValue="" />
    <Parameter Name="PackageQuantity" Type="System.String" DefaultValue="" />
    <Parameter Name="PackageWeight" Type="System.String" DefaultValue="" />
    <Parameter Name="DinNorm" Type="System.String" DefaultValue="" />
  </Parameters>
  <SortOrder>
    <Sort Field="ID" Direction="Ascending" />
    <Sort Field="VariantID" Direction="Ascending" />
  </SortOrder>
</Query>

ADD:

This occurs when calling the list view with no group id provided, even with an empty list view template configured.

The list view with a group id provided does work.
The detail view does work.

Thnx ...
I indeed had 9999 configured, which meant as much as "show them all" ... because I don't want a limitation ...
I changed it to 999 and now it works fine ... and we'll probable not reach that number on a single page anyway ...

Hi Jeppe,

If the UI has that limitation on the field, not allowing more than 1000, would prevent users from making these "mistakes" in the first place. How does that sound?

Nuno

Hi Nuno,

I'd rather not have those kinds of limitations in the code. I wouldn't feel comfortable with adding a limit like that because of a techical concern - which this is.

It should be possible for someone to render 10.000 products on a page if they REALLY wanted to. They'd have to deal with the performance that resulted from that decision, but it would be THEIR decision.

I don't think it's unreasonable to expect developers to consider the impact of the page size on the performance, but putting a limit on that field is like me holding your hand because I don't think you're grown up enough to cross the street on your own. Naturally, if I expect the developer to be reasonable, it's only fair that the developer expects me to be reasonble as well. If they set the page size to any number they choose, they shouldn't be greeted with an exception (except maybe a timeout).

Please excuse that I'm arguing from performance concern, when this issue in fact is a technical one, but I hope that I managed to make some sense as to why I don't think adding a limit is the right solution.

- Jeppe

But when I put 10.000 as PageSize value the module doesn't get rendered anymore, even without templates selected.
So how can we deal with performance if there is no output anymore?

Hi Peter,

I think you misunderstood my argument. It wasn't your issue specifically that I was commenting on, rather it was Nuno's suggestion.

If I understand Nuno correctly, his suggestion was to simply limit the page size field to 1000 because the API underneath throws an exception for larger pages. My philosophy is that it shouldn't be some API that dictates a page size limit, just because it can't handle that many products. The page size field is there to allow the developer to fine tune the product catalog to show the products needed while still maintaining an acceptable level of performance. If the developer decides that the performance is less important that a large number of products, it should be the developer's decision. It shouldn't be limited by the API. You may, of course, still get timeouts if the amount of data that needs to be processed gets too big, but that's a separate matter.

Your situation is exactly that you're limited by the API, which is why I've created a task to get it fixed. It isn't fixed at the moment, which is why you can't set the page size larger than 1050 without getting an exception. Once the fix is released, you should be able to set any page size you want and then you can decide on the performance trade-offs.

I hope that makes it clear what I meant.

- Jeppe

Hi Jeppe,

Not a problem at all. What you say makes sense. Thank you for that explanation, it helps understand some of the decisions.

Best Regards,

Nuno

Hi Jeppe,

Thanks for planning a fix for this issue. 

We need the large page size because we have products (screws, bolts, rings, etc) with loads of variants (up to a 100 for 1 product) which we want to display grouped by masterproduct, with paging and with facet filtering. As far as we have been able to find we can only do this by letting our productrepository return only productvariants and then group that result on masterproduct, and create our own custom paging on those masterproducts. As far as we know there is no way to do this with standard DW functionality.

The 1050 limit is not enough for us, since we have product groups with up to 1400 variant products.

Do you have an expectation when the fix for this issue might be released?

Greets Hans

Hi Hans,

While I'd love to be able to tell you exactly when the fix will be available, I don't know at the moment. We're going to look into the issue next week, but depending on the scale of the fix, it may not be out for another couple of weeks. It's not on the backburner, so it'll be out as soon as possible.

I'm sorry that I cannot be more specific.

- Jeppe

Hi Hans,

We've had to deal with similar issues in the past (render only master, but have variants data). Have you tried to develop an index builder extender that targets only master products and adds the relevant variant data to it?

That would allow you to:

• Always and only match Master products
• Get facets to match "existing" variant data (but on the master product) - i.e. available in Color Blue, White and Orange (this could be applied to any custom field of course)
• Being just master products you don't have issues with paging and pagesize and whatnot

Best Regards,

Nuno Aguiar

Hi Nuno,

We have not tried that approach, sounds like it could work, but I fear that in our case it could become somewhat nasty with 70+ product category fields. If it was only color and size we would have gone for it. And our current solution was working fine until our latest update to 9.9.

Greets Hans

Understood Hans

BTW we developed an IndexBuilderExtender that does that for "most fields" automatically. But if you're having to do this for 70 fields (because they are different per variant), sounds like a complex structure as it is. I would encourage pursuing a different solution at least in the future. 

@Nuno, for reference, do you have code to share?

Hi Nicolai,

Check attachment. Here's what that project does:

• Macro (unrelated to this) to get the RealUser (considers if you're impersonating and impersonation settings)
• The IndexBuilder Extender, schemaExtender and whatnot

We also added some notifications (this is a standard base to then develop on top of it), which you'd probably don't need, so disregard those too.

Best Regards,

Nuno Aguiar

Hi Jeppe,

Is there any news about the planning for this issue? I do not see it in the known bug list, and we need to know if we need to escalate it, since its a blocking issue for our go live for our customer.

Greets Hans

Thanks Jeppe!

Hi Hans

The feature was released in Dynamicweb version 9.9.1

This is available in download section:

https://doc.dynamicweb.com/downloads/releases

Kind Regards
Dynamicweb Support
Kristian Kirkholt