Developer forum

Forum » Ecommerce - Standard features » Contains/Equal index query returns a fixed, digit-independent result count for search terms (e.g. "M12" == "M99" == "M777777")

Contains/Equal index query returns a fixed, digit-independent result count for search terms (e.g. "M12" == "M99" == "M777777")

Joakim
Reply

Environment:

  • Dynamicweb.dll: 9.18.1
  • Dynamicweb.Ecommerce.dll: 1.18.1
  • Dynamicweb.Indexing.Lucene.dll: 1.4.6
  • Standard Dynamicweb.Indexing.Queries.IndexQueryProvider repository/index query (not custom code) — Ecommerce product index built via the standard repository/index/query XML setup (Repository + .index + .query files).
     

    Problem: A single-word search term shaped like <one letter><digits> (e.g. M12, B1000) returns a result count that is entirely determined by the letter and completely independent of the digits — including digit strings that don't exist anywhere in the database.

    Reproduction (via the standard product search page/index query, q parameter):

    Query Result count Real DB matches for this exact string
    m1 4275 —
    m12 4275 804
    m99 4275 0
    m777777 4275 0
    b1 3725 —
    b1000 3725 108
    b999999 3725 0
    a2 4113 —
    a1000 4113 —
    z 6 —
    z1000 6 —

    m99 and m777777 have zero literal matches in EcomProducts.ProductName/ProductNumber (confirmed via direct SQL against the underlying database), yet return the identical 4275-result set as the genuinely common m12. Since the result count is invariant across wildly different digit strings — including nonexistent ones — this rules out normal tokenization, stemming, wildcard/prefix expansion, or substring matching, all of which would be sensitive to the actual characters typed. Something in the query pipeline appears to detect the <letter><digits> shape and short-circuit to a per-letter default/fallback bucket, discarding the digits entirely.

    What we've ruled out on our side:

  • Not in our Products.query (XML query definition) — no regex, branching, or letter-specific logic exists anywhere in its <Expressions> block; it's a plain OR of Equal/Contains BinaryExpressions against ProductName_Search, ProductNumber_Search, freetext, and Swedish-stemmed SynSV_* fields.
  • Not in our Products.index field/analyzer configuration — no custom analyzer beyond a standard synonym dictionary, which itself doesn't affect this (confirmed: identical behavior on synonym-free control terms).
  • Not in our front-end templates — they only consume the already-populated ProductListViewModel; none of them construct the eq/q query parameters or contain any digit/letter-shape logic.
  • Multi-word combination logic is not the bug — separately confirmed that Contains correctly ANDs space-separated words together (e.g. skruv alone: 671 hits, mutter alone: 799, skruv mutter together: 239 — a proper intersection-sized reduction). The bug is isolated specifically to single alphanumeric tokens matching this <letter><digits> shape.
  • Impact: This makes any product-number-style search (a huge fraction of real customer searches in an industrial/fastener catalog, e.g. "B1000", "M12x50") return thousands of irrelevant products mixed in with the handful of real matches, since the query silently ignores the actual number typed.

    Question: Is this a known issue in the Lucene indexing query layer? Is there a supported workaround (a different operator, a query/index setting) that avoids this letter/digit collapse, or does this need a platform fix?

    Happy to provide a full request/response trace or repo access details on request.


Replies

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply
This post has been marked as an answer

Thanks for the detailed write-up and the reproduction table. It made this much easier to look into.

We don't believe this is a platform bug or letter/digit detection in the query layer. The behaviour matches an analyzer that discards digits during tokenization on one of the fields in your OR expression. That means you have an analyzed field and do a equals expression on that field.

Why we think so

A tokenizer that splits on non-letters turns "m12", "m99" and "m777777" into the same single term: "m". The digits are gone before the query is executed, so the result is by definition independent of them. Your z / z1000 pair (6 and 6) shows this most clearly: the query is effectively a term query for a standalone "z".

The constant count comes from the OR. Say one branch (for example freetext or one of the SynSV_* fields) uses a letter-only tokenizer. That field indexed "M12" as "m" at index time too, so every genuine m12 hit from your other branches (ProductNumber_Search etc.) is already contained in the "m" result set from that branch. The union is therefore always the full "m" set: 4275, regardless of the digits.

Likely causes in Lucene.Net 3.0.3

- SimpleAnalyzer or StopAnalyzer. Both use LetterTokenizer, which drops digits entirely.
- The analyzer behind the synonym dictionary. This might be it. Your synonym-free control terms don't rule it out, because tokenization happens whether or not a synonym matches.
- The index-level default analyzer. It applies to fields without an explicit analyzer, typically the freetext/summary field. This means if a field is analyzed

StandardAnalyzer and the Swedish Snowball analyzer keep "m12" as one token, so the plain ProductName_Search/ProductNumber_Search fields are probably not the problem, provided they use one of those.

How to confirm

1. Run the query with one OR branch at a time and search for m99. The branch that returns 4275 is the one to look at.
2. Open the index in Luke (the Lucene 3.x version) and look at the terms stored for a product containing "M12" in that field. If you see "m" and no "12" or "m12", that confirms it.
3. Optionally, log or preview the generated Lucene query. You should see the search term reduced to the letter only, e.g. SynSV_Name:m.

Suggested fix

- Change the analyzer on the affected field to Standard, or to a whitespace + lowercase analyzer, and rebuild the index.
- For product-number searches specifically, add a dedicated field that is lowercased but not tokenized, and query it with Equal plus a prefix-style match. This matters for your catalog: even with the Standard analyzer, "M12x50" is indexed as the single token "m12x50", so a plain term match on "m12" will not find it. A prefix match on a keyword-style field will.
- If the stemmed or synonym fields need to keep their current analyzer, consider excluding or down-weighting them for number-like searches so they can't dominate the result.

If the per-branch test doesn't isolate a single field, or Luke shows "m12" correctly stored everywhere, please send the request/response trace together with the .index and .query files and we'll dig further.

Best regards,
Nicolai

Votes for this answer: 1
 
Joakim
Reply

Thank you for that detailed reply, it solved the issue.

 

You must be logged in to post in the forum