Developer forum

Forum » CMS - Standard features » Rebuilding indexes leads to missing data

Rebuilding indexes leads to missing data

E-mail notifications
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply
Posted on 15/10/2018 09:41:04

We have a site with a Products index and a task set to do a full rebuild every hour. The index has two instances (A and B) and one build that does a Full index. Building manually works fine, and the task also runs successfully every hour. The balancer is set to LastUpdated although we also tried ActivePassive. A full rebuild takes about 10 to 12 seconds.

Everything works OK when we build manually, but from time to time (sometimes a few times a day, sometimes it runs fine for a day or two) the index is empty and we get no results on the Products page. At that point, we have to manually rebuild the index again to get products to show up.

I looked at the Diagnostics folder at the last index build causing the failure seems to be OK. The logs for A and B both contain the same data and amount of products and in the Index folder I find files that look as expected too. This makes me believe the index is fine but there might be something wrong with the balancer determining which index to use or some other kind of problem.

Has anyone seen this before? Are there other logs I can look at to see where the problem can be? We're running on 9.4.15.

Thanks!


Replies

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 24/10/2018 09:04:07
Reply

This continues to be a problem as we lose product data regularly. Is there anything else I can do to analyze this and see what's causing the data to be lost?

 
Nicolai Pedersen
Posted on 24/10/2018 21:51:34
Reply

Hi Imar

It is not in our records as far as I know.

How is the environment? NLB or anything like that?

And what happens if the products are gone and you do a IISRESET? To see if it is some kind of caching issue or it is the index that is broken.

And any chance it can be upgraded to 9.5 which contains a lot more information on index builds for the new monitoring/event system?

Also - either a copy or an access to the solution would be nice if the above does not give us anything.

BR Nicolai

 
Nicolai Pedersen
Posted on 24/10/2018 21:52:12
Reply

And could this be related: https://doc.dynamicweb.com/forum/dynamicweb-9-0-upgrade-issues/dynamicweb-9-0-upgrade-issues/product-index-builder

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 25/10/2018 12:44:06
Reply

Hi Nicolai,

>> How is the environment? NLB or anything like that?
Nope, just a basic custom set up.

>> And what happens if the products are gone and you do a IISRESET? To see if it is some kind of caching issue or it is the index that is broken.
I tried that yesterday before posting here but the products were still gone,

>> And any chance it can be upgraded to 9.5 which contains a lot more information on index builds for the new monitoring/event system?
We build but not in the next couple of weeks yet.

>> And could this be related: https://doc.dynamicweb.com/forum/dynamicweb-9-0-upgrade-issues/dynamicweb-9-0-upgrade-issues/product-index-builder
Not sure but it doesn't look like it as it doesn't fail consistently, It works most of the times.

>> Also - either a copy or an access to the solution would be nice if the above does not give us anything.
I'll send you the link to the site. Also happy top supply a full backup if needed!

Thanks!

Imar

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 25/10/2018 12:59:26
Reply

Could assortments be related to this? For example when the assortments are being built while the index is also being rebuilt?

 
Nicolai Pedersen
Posted on 26/10/2018 00:24:33
Reply
This post has been marked as an answer

I've looked at the solution.

Assortments is probably the reason. It re-indexes every 5 minutes and rebuild assortments also every 5 minutes. Your assortments seem to take up to 3 minutes to rebuild. So if the assortments are in a state of being rebuild, and the index starts, the index has no assortment information - and you have no products because the user has an assortment...

So you should try setting the start time of the index task to 12.00 and assortments to 12.01. Then the assortment has 4 minutes to the next index build. And maybe increase build to every 6 or 7 minutes.

 

Votes for this answer: 1
 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 27/10/2018 16:37:23
Reply

Thanks Nicolai. I think the index task was set to every hour; it's just the repository task scheduler that runs every 5 minutes.

But the problem would be the same at the whole hour: no assortments for the index. I now scheduled each to run once an hour; at the whole hour, the other at the 30 minute mark. That hopefully fixes the issue.

Thanks!

Imar

 
Anders Ebdrup
Anders Ebdrup
Posted on 21/08/2019 10:21:07
Reply

Hi all,

 

We see this issue too - do we have a permanent fix for this?

 
Nicolai Pedersen
Posted on 21/08/2019 11:08:23
Reply

What should that fix be?

 

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 21/08/2019 11:22:51
Reply

Not start an index when an assortment rebuild is running? And vice versa?

 
Anders Ebdrup
Anders Ebdrup
Posted on 21/08/2019 11:25:13
Reply

Or only change the cached assortments when rebuild of assortments is finished, as it seems like the cache of assortments are changed while rebuilding and not only when rebuild is finished

 
Anders Ebdrup
Anders Ebdrup
Posted on 21/08/2019 13:06:43
Reply

Maybe it is related to this issue?

CAS-09422-F7S3B4
investigation number is: 67639 

 

You must be logged in to post in the forum