Developer forum

Forum » Ecommerce - Standard features » SynonymAnalyzer — normalize term before tokenizing?

SynonymAnalyzer — normalize term before tokenizing?

Wilfred Polderman
Reply

Hi all,

We have a SynonymAnalyzer field type on a Products.index (Whitespace tokenizer), with synonym groups for technical size notations, e.g. 15x21 = 1/2".

This works fine when the search term has no spaces. But 15 x 21 (with spaces) gets split into three separate tokens by the Whitespace tokenizer before the synonym lookup runs, so it never matches — even when 15 x 21 is added as its own <synonym> entry in the same group.

Question: does the SynonymAnalyzer config support a character filter or pattern-replace step before tokenizing? Or is there a built-in way to normalize a query parameter's value (e.g. regex replace) before it hits the matching step? Looking for a config-based solution rather than a custom template/code change.

Thanks in advance!

Kind regards, 

Wilfred


Replies

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

Hi,

Good catch, and useful detail that it works without spaces but not with them — that pointed straight at where the multi-word synonym lookup breaks. I checked the actual implementation (this isn't Lucene's stock SynonymFilter/SynonymMap, Dynamicweb has its own).

Short answer: no, there's no config-based character filter or pattern-replace step available. The SynonymAnalyzer pipeline supports exactly four tokenizers — Standard, LowerCase, Whitespace, Keyword — and there is no character-filter concept (no MappingCharFilter, PatternReplaceCharFilter, or anything pre-tokenization) anywhere in the indexing provider. So there's no XML/backend knob to normalize "15 x 21" before it gets tokenized.

But that's not actually why your multi-word synonym is failing. The Whitespace tokenizer splitting "15 x 21" into ["15","x","21"] isn't the problem by itself — Dynamicweb's synonym engine is specifically built to walk multi-token sequences back together and match them against multi-word synonym entries. Both indexing and querying go through the same analyzer, so the token stream shape is consistent on both sides. The real bug is a case-folding mismatch between the two halves of the pipeline:

The filter chain order for the Whitespace tokenizer is: WhitespaceTokenizer → LowerCaseFilter → (optional StopFilter) → SynonymFilter. So by the time your tokens reach the synonym lookup, they're already lower-cased.
But the synonym dictionary itself is built by SynonymEngine.LoadXmlDocument, which reads each <synonym> entry with synNode.InnerText.Trim() — no lower-casing at all. The multi-word keys it stores in its lookup dictionary are built verbatim from your XML text, case and all.
So if your <synonym> entry is written as 15 x 21 with a lowercase x, it should actually match fine against a 15 x 21 query. But if it's written with any uppercase character anywhere in it — e.g. 15 X 21, or casing anywhere else in that phrase — the dictionary key won't match the lower-cased runtime tokens, and the multi-word walk in SynonymFilter silently falls through (GetStem just returns "not found," there's no logging). Single-word entries can appear to work by coincidence if they happen to already be lowercase, or because a single token still gets some partial matching behavior — but the moment a multi-word phrase key has any case mismatch, the whole phrase-walk fails from the first token.

What to check/try, no code change needed:

Open your synonym XML source and verify every character in every multi-word <synonym> entry is lowercase — including entries like 15 x 21, not 15 X 21.
Double-check for stray extra whitespace in the XML entry itself (tabs, double spaces, non-breaking spaces from copy-paste) — the engine splits strictly on ' ' and rebuilds with single spaces, so "15 x 21" (double space) or a non-breaking space won't match a normal "15 x 21" query either, and this one isn't fixable by casing.
If your stem attribute or other entries in the same group mix case intentionally (e.g. a product code that's meant to stay uppercase), that's fine — it's specifically the literal text of the <synonym> phrase itself that needs to be all-lowercase to survive the LowerCaseFilter step before matching.
If lowercasing the XML source fixes it, that confirms the mechanism. If it's still not matching after that, it's worth sending over the exact <group> XML block so we can check for whitespace/encoding issues in the file itself.

There genuinely isn't a config-based char-filter/pattern-replace mechanism in this analyzer today — if normalizing punctuation (like your 1/2" example) or other formatting variants turns out to be a recurring need beyond just casing, that would need either a custom field-value processing step at index/query time, or would be worth raising as a feature request for the analyzer pipeline.

 
Wilfred Polderman
Reply

Hi Nicolai,

Thanks for the detailed breakdown.

I went through everything you listed:

1. Lowercased the entire synonym source. I had deliberately added uppercase duplicates (e.g. both `15 x 21` and `15 X 21`, `M10L` next to `m10l`) because i assumed the Whitespace tokenizer was case-sensitive. I removed all of them and every `<synonym>` value is now all-lowercase.

2. Checked the file itself for stray whitespace. No tabs, no double spaces, no non-breaking spaces. Encoding is UTF-8.

3. Full rebuild of the index after each change.

Result: single-token synonyms work correctly. Searching `15x21` returns the 1/2" products, and `knelfitting` returns 27 products described as "knelkoppeling". But multi-word entries still never match. `15 x 21` and `5 x 10` return nothing, even though both sit in the same groups as the single-token variants that do work.

So the behaviour is consistent: every single-token entry resolves, every multi-word entry silently fails, regardless of casing.

Config:

xml
<?xml version="1.0" encoding="utf-8" ?>
<SynonymAnalyzer>
  <TokenizerType>Whitespace</TokenizerType>
  <StemmerName></StemmerName>
  <StopWords></StopWords>
</SynonymAnalyzer>

The exact group block you asked for:

xml
<group stem="s1_2">
  <synonym>1/2"</synonym>
  <synonym>15x21</synonym>
  <synonym>15/21</synonym>
  <synonym>15 x 21</synonym>
</group>
 

One thing I would like to rule out on our side: is there anything above the analyzer that could split the parameter value before it reaches the token stream? Our query expressions in Products.query pass the `q` and `eq` parameters into the synonym field. If the query parser tokenizes that value on spaces first, the SynonymFilter would never see the full phrase which would match exactly what we observe. Is that a known factor, or does the parameter value always reach the analyzer intact

Kind regards,

Wilfred

 
Nicolai Pedersen Dynamicweb Employee
Nicolai Pedersen
Reply

I think I can see the issue now and made a fix: https://doc.dynamicweb.dev/documentation/fundamentals/dw10release/releasenotes/workiteminfo.html?workitemid=29176

You might make it work by changing the order of synonyms:
<group stem="s1_2">
  <synonym>1/2"</synonym>
  <synonym>15 x 21</synonym>
  <synonym>15x21</synonym>

  <synonym>15/21</synonym>
</group>

 

You must be logged in to post in the forum