Developer forum

Forum » Integration » Handling Large XML Feed for EasyCatalog

Handling Large XML Feed for EasyCatalog

Jelle Deridder
Reply

We’ve been running a multilingual XML feed from Dynamicweb 10 to EasyCatalog for quite some time. It has always worked well, but recently the feed has grown so large that it now takes over 2 hours to fully load. This has made live triggering by EasyCatalog impractical and unstable.

The Problem
The XML feed has become too heavy for live generation. We're now exploring alternative approaches to keep it usable and performant.

Options We’re Considering

  1. Pre-generating the XML on a scheduled basis and delivering it as a static file (e.g. via FTP).

  2. Using the 65bit pagination tool for EasyCatalog to reduce the data volume per request (still being evaluated).

  3. Rebuilding the feed logic as a Dynamicweb Integration Activity, though we're unsure if this is suitable given the Razor complexity and volume of joins.

Questions

  • What’s the recommended approach in Dynamicweb for handling very large XML feeds like this?

  • Does anyone have experience with 65bit pagination module in combination with Dynamicweb feeds?

  • Is using an Integration Framework provider a better long-term solution?

Any input is greatly appreciated!

 


Replies

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

I would go with option #1 as it allows you to control when the file is generated. There's a standard scheduled task for it available called the Feed File Export Task:

It uses the same underlying code to generate the feed so you can continue to configure it at the feed level as needed.

Imar

 
Jelle Deridder
Reply

Thanks for the suggestion!

I’ve already tried running our existing Razor-based XML feed through the Feed File Export Task, but unfortunately this results in a runtime error. The template itself works perfectly fine when executed via URL (using a query for a single product). However, when I schedule the same feed via the standard Feed File Export Task, I get the following error:

System.NullReferenceException: Object reference not set to an instance of an object.
   at CompiledRazorTemplates.Dynamic.RazorEngine_xxxxxxxx.Init()
   at CompiledRazorTemplates.Dynamic.RazorEngine_xxxxxxxx.GenerateXML()
   ...

This suggests something goes wrong inside the Init() method, which is defined like this in our Razor mapping:

private void Init()
{
    // Set up config flags
    IncludeStockStatus = false;
    ...

    // Get query string parameter
    var lngParameter = Dynamicweb.Context.Current.Request.QueryString["languageId"]?.ToUpper() ?? "ALL";
    ...
}

From what I can tell, this error likely happens because the Request.QueryString["languageId"] line assumes an HTTP context is available, which is not the case during scheduled execution (no live HTTP request). So Dynamicweb.Context.Current.Request seems to be null in that context, which triggers the NullReferenceException.

 

Is there a clean way to safely access query parameters in Razor templates that will also work in scheduled tasks?

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

There are n o query parameters when executing as a task AFAIK. But you can check for null in the template?

    var lngParameter = Dynamicweb.Context.Current?.Request.QueryString["languageId"]?.ToUpper() ?? "ALL";
The question mark in Current? will then check if Current is null. If it is, it'll default to "All"

Imar

 

You must be logged in to post in the forum