Hi Adrian

You can get a copy of the api endpoint code handling this and create your own version.

BR Nicolai

Hi Nicolai,
Thank you. Before doing that, it would be useful to understand what can be achieved with the standard endpoint and then let the customer decide if they want to invest in a custom endpoint or use the standard.

Can you please help me clarify the standard part?

Thank you,
Adrian

I have given AI the code and some insigths and this is what we get:

 

I dug through the attached ProductsController + the DownloadCart.cshtml template. Here’s what the exports endpoint can and cannot do, and how that maps to Adrian’s 3 pain points.

What /dwapi/ecommerce/products/export actually does (in this controller)

• It builds a ProductListViewModel using one of these strategies (in priority order):

  1. RepositoryName + QueryName

  2. GroupId

  3. ProductAutoIds

  4. ProductIds

• Then it creates a ProductViewModelsExporter and sets:

  • ExportFormat (json/csv/xml/none)

  • optional FeedId

  • LanguageId = request.LanguageId (singular)

  • email template + recipients

  • plus it passes through the full request params as ContextParameters (so the exporter/feed can “see” the submitted params)

• It does not stream a file back to the browser. It triggers an async job and emails a download link (zip).

So this endpoint is basically: build product list → export (optionally with images/assets) → email link.

1) “Export selected languages / multiple shops”

Why your “multiple ShopId / multiple LanguageId parameters” attempt fails

In FillRequest(), the controller copies querystring/form keys into a dictionary like this:

• it iterates Request.Params.AllKeys

• and only adds the first occurrence of each key

• duplicates are ignored

So repeating parameters like:
&LanguageId=LANG3&LanguageId=LANG2
or
&ShopId=SHOP1&ShopId=SHOP2
will effectively collapse to one value (whichever comes first in the request collection).

On top of that, the exporter is explicitly given only:
LanguageId = request.LanguageId (a single value).

What that means in practice

• “Multiple shops in one export request” is not supported by this implementation.

• The UI in your attached template also reflects this reality: it’s a single <select name="LanguageId"> (not multi-select).

Realistic ways to solve it

• Call the export endpoint once per shop/language (loop client-side or server-side), then either:

  • accept “one email per language/shop”, or

  • build your own wrapper endpoint that runs N exports and bundles them into one zip/email.

• If you control the code: change the contract so the endpoint accepts LanguageId[] / ShopId[], then loop internally and export multiple files into the same archive.

If you must stay purely “configuration-driven” with no custom code: the blunt truth is that your “one shop per language” setup fights the assumptions of this feature.

2) “Let user select which fields to export”

You’ve got two knobs, depending on how deep you want to go:

A) Lightweight: FilledProperties / ProductSettings.FilledProperties

The controller explicitly supports using viewmodel “filled properties” (it even documents examples like ProductSettings.FilledProperties=Id,Name,Title).

That means you can:

• add a UI list of fields

• submit them as ProductSettings.FilledProperties=...

• and the product view models should only be populated with those fields

This is the fastest path if your exports are JSON-ish or you’re okay with the exporter simply serializing what’s present.

B) Proper “export definition”: use FeedId

ExportSettings includes FeedId with the comment: “The id of the feed that should be used for generating the product file export.”

That’s a big hint: the intended advanced customization route is eCom Feeds (define which fields are output, ordering, transformations, etc.), then export using that feed.

So: if you want “users pick fields” and you want it reflected cleanly in CSV/XML, you usually end up with either:

• multiple pre-defined feeds (“Minimal”, “Marketing”, “Technical spec”, …), or

• custom code that generates a feed/export config dynamically.

3) “Let user control the label (column header) for each field”

This depends heavily on format:

• JSON: property names tend to be the model property names (labels aren’t really a thing unless you post-process).

• CSV/XML: labels can be supported cleanly if you go through a feed/export-definition layer.

So the best-supported approach is:

• use a Feed where you define output field names / column headers (or whatever your feed system supports for naming).

• If you need totally ad-hoc, per-user labels at runtime: that’s almost certainly custom exporter logic (or post-processing the generated output).

“If I can’t do the above, how do I take control?”

You’ve basically got three levels of control:

1. Template-only (no C# changes)

   • single shop + single language per request

   • choose fields via ProductSettings.FilledProperties (simple)

   • labels: limited (format-dependent)

2. Feed-driven configuration

   • pick a FeedId to control structure/fields/labels

   • still single shop/language per request, unless you orchestrate multiple calls

3. Custom orchestration endpoint (best UX)

   • accept multiple shops/languages

   • accept selected fields + labels

   • generate N exports → bundle → email one link

Next will be the AI proposed changes...

Here’s how I’d wire this cleanly into your actual setup, without breaking the existing export logic.

1️⃣ Multi-language / multi-shop — without fighting the current controller

Your current ProductsController is structurally built around:

• one LanguageId

• one ShopId

• one export execution per request

And it builds the request dictionary from Request.Params in a way that ignores duplicate keys — which is why &LanguageId=...&LanguageId=... doesn’t behave like you hoped.

So we don’t modify it.

Instead:

Add a new endpoint in the same controller (or a sibling controller)

Something like:

[HttpPost]
[Route("dwapi/ecommerce/products/exportbundle")]
public async Task<IActionResult> ExportBundle(ExportBundleRequest model)

Where ExportBundleRequest contains:

public class ExportBundleRequest
{
    public string[] LanguageIds { get; set; }
    public string[] ShopIds { get; set; }
    public string[] Fields { get; set; }
    public Dictionary<string, string> Labels { get; set; }
    public string Format { get; set; }

    public string GroupId { get; set; }
    public string RepositoryName { get; set; }
    public string QueryName { get; set; }
    public string[] ProductIds { get; set; }
}

Then inside:

1. Loop each (shopId, languageId)

2. Reuse the existing export logic (the same service the current endpoint calls)

3. Collect the generated files

4. Zip them

5. Return one download or email one link

We’re orchestrating, not rewriting.

That’s the key move.

2️⃣ UI changes in DownloadCart.cshtml

Right now your template posts:

• LanguageId

• ShopId

• ExportFormat

• etc.

We evolve it.

Change language selector to multi-select

<select name="LanguageIds" multiple>

If one shop = one language, you can hide shop selection and map language → shop server-side.

That keeps UX sane.

Add field selection

In your template:

@{
    var availableFields = new[] { "Id", "Number", "Name", "ShortDescription", "Price.Price" };
}

@foreach (var field in availableFields)
{
    <div>
        <input type="checkbox" name="Fields" value="@field" />
        <label>@field</label>
        <input type="text" name="Labels[@field]" placeholder="Custom label (optional)" />
    </div>
}

Now the form posts:

Fields=Id
Fields=Name
Labels[Id]=Product ID
Labels[Name]=Title

Clean. Deterministic. Safe.

3️⃣ Field filtering in export

Inside your bundle endpoint:

When you build the product list, pass the selected fields into:

ProductSettings.FilledProperties

If your current controller already respects that param (it likely does), you just inject it into the context parameters before export.

If not, you enforce it in your own product list builder.

This ensures only selected fields are populated.

No reflection chaos.

4️⃣ Custom column labels (CSV)

Here’s the elegant hack:

After generating CSV:

var lines = System.IO.File.ReadAllLines(csvPath);
var headers = lines[0].Split(';'); // or ',' depending on format

for (int i = 0; i < headers.Length; i++)
{
    if (model.Labels.TryGetValue(headers[i], out var newLabel))
    {
        headers[i] = newLabel;
    }
}

lines[0] = string.Join(";", headers);
System.IO.File.WriteAllLines(csvPath, lines);

This avoids touching exporter internals.

Simple. Powerful. Boring. Reliable.

5️⃣ Important: PageSize trap in your template

Your current template includes:

PageSize=100

Which means:

You are exporting max 100 products unless overridden.

If this is meant to export “all selected products”:

Either:

• remove PageSize

• or set it very high

• or loop paging internally in your bundle endpoint

Silent truncation is the kind of bug that creates management meetings.

6️⃣ What I would not do

• I would not modify the original export endpoint to suddenly support arrays.

• I would not overload Request.Params parsing.

• I would not dynamically reflect arbitrary field names without whitelist validation.

Because complexity compounds.

And compound complexity is how legacy systems are born.

Final Architecture Snapshot

You’ll have:

• Existing /export endpoint (unchanged)

• New /exportbundle endpoint (orchestrator)

• Slightly upgraded DownloadCart.cshtml

• Optional CSV header rewrite

• Optional feed-based enhancement later

Adrian gets:

✔ Multiple languages
✔ Multiple shops
✔ User-selectable fields
✔ Custom column labels
✔ Single ZIP
✔ One email

And you avoid rewriting the world.

Let’s turn this into pasteable scaffolding that matches your actual controller style (ApiController, [RoutePrefix("dwapi/ecommerce/products")], etc.) and doesn’t depend on undocumented internals of ProductViewModelsExporter.

Below is a drop-in “bundle export” endpoint that:

• accepts multiple ShopIds + LanguageIds

• accepts field selection + optional labels

• exports CSV or JSON

• returns one ZIP download (single request, single response)

It reuses your existing product query logic (ViewModelFactory.CreateView(...)) so selection via Repository/Query, GroupId, ProductIds, etc. works the same way.

1) Add this to ProductsController.cs

Put the DTOs + helpers near the bottom of the controller (or in separate files if you prefer). Then add the endpoint.

DTOs

using System.IO;
using System.IO.Compression;
using System.Net.Http.Headers;
using System.Reflection;
using Newtonsoft.Json; // If you already use Newtonsoft. Otherwise switch to System.Text.Json.

// ...

public class ExportBundleRequest
{
    // Multi selection
    public List<string> LanguageIds { get; set; } = new List<string>();
    public List<string> ShopIds { get; set; } = new List<string>();

    // Export format: "csv" or "json"
    public string Format { get; set; } = "csv";

    // Selected fields (dot-path allowed, e.g. "Price.Price", "ProductFields.Color")
    public List<string> Fields { get; set; } = new List<string>();

    // Optional header labels for fields
    public Dictionary<string, string> Labels { get; set; } = new Dictionary<string, string>();

    // Product selection (same hierarchy you already use)
    public string RepositoryName { get; set; }
    public string QueryName { get; set; }
    public string GroupId { get; set; }
    public List<string> ProductIds { get; set; } = new List<string>();
    public List<string> ProductAutoIds { get; set; } = new List<string>();

    // Optional: file base name
    public string FileName { get; set; } = "products";
}

// A super small response model if you later want "email link" instead of direct download.
public class ExportBundleResult
{
    public string Message { get; set; }
}

The endpoint

[HttpPost, Route("exportbundle")]
[SwaggerResponse(HttpStatusCode.OK, typeof(HttpResponseMessage), Description = "ZIP containing one export file per shop/language.")]
[SwaggerResponse(HttpStatusCode.BadRequest, null, Description = "Invalid parameters.")]
[SwaggerResponse(HttpStatusCode.NotFound, null, Description = "No matching products found.")]
[SwaggerResponse(HttpStatusCode.NoContent, null, Description = "No products match the parameters.")]
public HttpResponseMessage ExportProductsBundle([FromBody] ExportBundleRequest model, [FromUri] ProductListViewModelSettings request)
{
    if (!Security.Licensing.LicenseManager.LicenseHasFeature("DownloadCart"))
        return Request.CreateResponse(HttpStatusCode.PaymentRequired, "You don't have a proper license to support this feature");

    if (model == null)
        return Request.CreateResponse(HttpStatusCode.BadRequest, "Missing body");

    // Defaults / validation
    var format = (model.Format ?? "csv").Trim().ToLowerInvariant();
    if (format != "csv" && format != "json")
        return Request.CreateResponse(HttpStatusCode.BadRequest, "Format must be 'csv' or 'json'");

    if (model.Fields == null || model.Fields.Count == 0)
        return Request.CreateResponse(HttpStatusCode.BadRequest, "Select at least one field to export");

    // Build a list of (shopId, languageId) pairs.
    // If you run "one shop per language", you can just pass LanguageIds and map to shops here instead.
    var pairs = BuildPairs(model.ShopIds, model.LanguageIds);
    if (pairs.Count == 0)
        return Request.CreateResponse(HttpStatusCode.BadRequest, "Select at least one language/shop");

    // Build the SearchRequest just like your existing ExportProductsPost does.
    var searchRequest = new SearchRequest
    {
        RepositoryName = model.RepositoryName,
        QueryName = model.QueryName,
        GroupId = model.GroupId,
        ProductIds = model.ProductIds,
        ProductAutoIds = model.ProductAutoIds
    };

    // NOTE: request is your existing ProductListViewModelSettings (FromUri).
    // We'll clone-per-iteration by copying and setting LanguageId/ShopId.
    using (var zipMs = new MemoryStream())
    using (var zip = new ZipArchive(zipMs, ZipArchiveMode.Create, leaveOpen: true))
    {
        bool anyProducts = false;

        foreach (var (shopId, languageId) in pairs)
        {
            // Make a shallow copy of the settings object (manual copy avoids reflection surprises).
            var iterRequest = CloneRequest(request);
            iterRequest.ShopId = shopId;
            iterRequest.LanguageId = languageId;

            // Optional: If you want field selection to affect viewmodel filling, pass a filled properties param.
            // Many DW viewmodels honor ProductSettings.FilledProperties; if yours does, add:
            // iterRequest.ProductSettings.FilledProperties = string.Join(",", model.Fields);
            // (Only do this if ProductSettings exists on your settings type.)

            // This mirrors your existing ExportProducts(...) selection hierarchy
            var vm = CreateProductListViewModel(searchRequest, iterRequest);
            if (vm == null)
                continue;

            if (vm.Products == null || vm.Products.Count == 0)
                continue;

            anyProducts = true;

            var fileName = MakeEntryFileName(model.FileName, shopId, languageId, format);

            var entry = zip.CreateEntry(fileName, CompressionLevel.Optimal);
            using (var entryStream = entry.Open())
            using (var writer = new StreamWriter(entryStream))
            {
                if (format == "csv")
                {
                    WriteCsv(writer, vm.Products, model.Fields, model.Labels);
                }
                else
                {
                    WriteJson(writer, vm.Products, model.Fields, model.Labels);
                }
            }
        }

        if (!anyProducts)
            return Request.CreateResponse(HttpStatusCode.NoContent);

        zip.Dispose(); // finalize
        var bytes = zipMs.ToArray();

        var response = new HttpResponseMessage(HttpStatusCode.OK);
        response.Content = new ByteArrayContent(bytes);
        response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/zip");
        response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
        {
            FileName = $"{(model.FileName ?? "products")}.zip"
        };

        return response;
    }
}

Helpers used above

private static List<(string shopId, string languageId)> BuildPairs(List<string> shopIds, List<string> languageIds)
{
    shopIds = shopIds?.Where(s => !string.IsNullOrWhiteSpace(s)).Distinct().ToList() ?? new List<string>();
    languageIds = languageIds?.Where(l => !string.IsNullOrWhiteSpace(l)).Distinct().ToList() ?? new List<string>();

    var pairs = new List<(string, string)>();

    // If both are provided, cross-join (explicit).
    if (shopIds.Count > 0 && languageIds.Count > 0)
    {
        foreach (var s in shopIds)
        foreach (var l in languageIds)
            pairs.Add((s, l));
        return pairs;
    }

    // If only one side is provided, create pairs with empty other part (or map later).
    // In your "one shop per language" world, you can map language->shop here.
    foreach (var s in shopIds)
        pairs.Add((s, null));
    foreach (var l in languageIds)
        pairs.Add((null, l));

    return pairs;
}

private ProductListViewModelSettings CloneRequest(ProductListViewModelSettings request)
{
    // Manual shallow clone (copy properties you actually use).
    // Add more properties if your ProductListViewModelSettings has relevant fields.
    if (request == null)
        request = new ProductListViewModelSettings();

    return new ProductListViewModelSettings
    {
        ShopId = request.ShopId,
        LanguageId = request.LanguageId,
        CurrencyCode = request.CurrencyCode,
        CountryCode = request.CountryCode,
        // PageSize etc. if present
        PageSize = request.PageSize,
        PageNumber = request.PageNumber
        // Add others as needed
    };
}

private ProductListViewModel CreateProductListViewModel(SearchRequest searchRequest, ProductListViewModelSettings request)
{
    // Same hierarchy as your ExportProducts()
    if (!string.IsNullOrEmpty(searchRequest.RepositoryName) && !string.IsNullOrEmpty(searchRequest.QueryName))
        return ViewModelFactory.CreateView(request, searchRequest.RepositoryName, searchRequest.QueryName);

    if (!string.IsNullOrEmpty(searchRequest.GroupId))
        return ViewModelFactory.CreateView(request, searchRequest.GroupId);

    if (searchRequest.ProductAutoIds != null && searchRequest.ProductAutoIds.Count > 0)
        return ViewModelFactory.CreateView(request, searchRequest.ProductAutoIds);

    if (searchRequest.ProductIds != null && searchRequest.ProductIds.Count > 0)
        return ViewModelFactory.CreateView(request, searchRequest.ProductIds);

    return null;
}

private static string MakeEntryFileName(string baseName, string shopId, string languageId, string format)
{
    baseName = string.IsNullOrWhiteSpace(baseName) ? "products" : baseName.Trim();
    var s = string.IsNullOrWhiteSpace(shopId) ? "shop" : shopId;
    var l = string.IsNullOrWhiteSpace(languageId) ? "lang" : languageId;
    return $"{baseName}_{s}_{l}.{format}";
}

CSV/JSON writers (field selection + labels)

private static void WriteCsv(TextWriter writer, IList<object> products, List<string> fields, Dictionary<string, string> labels)
{
    // Header
    var header = fields.Select(f => EscapeCsv(labels != null && labels.TryGetValue(f, out var lbl) ? lbl : f));
    writer.WriteLine(string.Join(";", header)); // Use ';' (common in DK/Excel). Change if you need ','.

    foreach (var p in products)
    {
        var row = fields.Select(f => EscapeCsv(Convert.ToString(ReadPath(p, f) ?? "")));
        writer.WriteLine(string.Join(";", row));
    }
}

private static void WriteJson(TextWriter writer, IList<object> products, List<string> fields, Dictionary<string, string> labels)
{
    // Output an array of objects with label-aware keys (labels override field key)
    var rows = new List<Dictionary<string, object>>();

    foreach (var p in products)
    {
        var obj = new Dictionary<string, object>();
        foreach (var f in fields)
        {
            var key = (labels != null && labels.TryGetValue(f, out var lbl) && !string.IsNullOrWhiteSpace(lbl)) ? lbl : f;
            obj[key] = ReadPath(p, f);
        }
        rows.Add(obj);
    }

    // If you don’t want Newtonsoft, swap to System.Text.Json
    var json = JsonConvert.SerializeObject(rows, Formatting.Indented);
    writer.Write(json);
}

private static object ReadPath(object obj, string path)
{
    if (obj == null || string.IsNullOrWhiteSpace(path))
        return null;

    object current = obj;
    foreach (var part in path.Split('.'))
    {
        if (current == null) return null;

        // IDictionary support (Dynamicweb models sometimes expose dictionaries)
        if (current is System.Collections.IDictionary dict)
        {
            if (!dict.Contains(part)) return null;
            current = dict[part];
            continue;
        }

        var t = current.GetType();
        var prop = t.GetProperty(part, BindingFlags.Public | BindingFlags.Instance | BindingFlags.IgnoreCase);
        if (prop == null) return null;

        current = prop.GetValue(current);
    }
    return current;
}

private static string EscapeCsv(string s)
{
    if (s == null) return "";
    if (s.Contains(";") || s.Contains("\"") || s.Contains("\n") || s.Contains("\r"))
        return "\"" + s.Replace("\"", "\"\"") + "\"";
    return s;
}

2) Update DownloadCart.cshtml to use the new endpoint

Minimal changes:

A) Post to the new endpoint

<form method="post" action="/dwapi/ecommerce/products/exportbundle?ShopId=@shopId&LanguageId=@languageId">

Becomes something like:

<form method="post" action="/dwapi/ecommerce/products/exportbundle">

…and send a JSON body (best) or plain form fields. If you want to stay form-only, you can still post form-url-encoded, but JSON is cleaner for arrays + labels.

B) Multi-select languages + shops (form-friendly)

<select name="LanguageIds" multiple>
  @* options *@
</select>

<select name="ShopIds" multiple>
  @* options *@
</select>

C) Field picker + label inputs

@{
  var fields = new[] { "Id", "Number", "Name", "ShortDescription", "Price.Price" };
}

@foreach (var f in fields)
{
  <div>
    <label>
      <input type="checkbox" name="Fields" value="@f" />
      @f
    </label>
    <input type="text" name="Labels[@f]" placeholder="Custom column label (optional)" />
  </div>
}

D) Format

<select name="Format">
  <option value="csv">CSV</option>
  <option value="json">JSON</option>
</select>

E) Keep your existing product selection inputs

Whatever you currently post for GroupId, RepositoryName, QueryName, ProductIds, etc., keep doing that — just ensure they map to ExportBundleRequest.

3) Two practical “DW9 reality” tweaks I’d do immediately

• Kill the PageSize=100 trap for exports. Either remove it for this bundle export or set it explicitly high / implement paging.

• Whitelist fields. Don’t allow arbitrary reflection paths from users unless you enjoy surprise data leaks and long coffees.s type.

Hi Nicolai,
Thank you very much for the detailed response :)

I will have to read it through before I make any changes. But it is very useful.

Thank you.

Adrian

Hi Nicolai,
I got pretty far with your recommendation, thank you very much.
I am now trying to include ProductFields and ProductCategoryFields in the export, but I have no idea how to request them.
I suppose I need to extend the iterRequest object but I could not find the right syntax.
Can you please help?

Thank you,
Adrian

Hi Nicolai,
I tried to make progress with this but I got stuck.
What I am trying to achieve is allow the admin to create a FieldDisplayGroup containing all relevant foelds for export.
Then, render the list of fields in a control similar with what you get for the FieldDisplayGroup setup.
Based on this control, I would collect all requested fields and create the export.

I have a few challenges. The first one is correctly connecting FieldDisplayGroup fields to the viewModel.
The FieldDisplayGroup exposes a FieldIdsList containing systemNames of the fields. For CustomFields and Category fields this is fine. But for standard fields, the system names are different. For example, the FieldDisplayGroup would store ProductName instead of Name which obviously will render an error if I try to export is as ProductName is not a valid ProductViewModel property.
I tried figuring out an easy translation between what I get from FieldDisplayGroup and the ViewModelProperties but I could not find.
Moreover, the ValidPropertyNames method is internal and I cannot even get the valid list of standard properties.

The second part of my issue is requesting ProductFields and ProductCategories fields. I have found an alternative way but I am not very fond of it as it may seem inefficient for large datasets.
Can you please point me in the right direction here?
Thank you,

Adrian
 

Hi Nicolai,
I have another question related to this topic, a bit more critical than my questions above.
It is related to the limitations of this export approach.
The customer in question is still using the standard export approach without any customizations.
They have recently reported that a configured export timed out. Which leads me to my next question: what are the (known) limitations of this export approach? I am talking about the number of products and the number of assets included in the export. 
This is rather critical because the answer may force a different development direction.

Thank you,
Adrian

Hi Adrian

An analysis of your question on top of our code base:

Why it timed out --- what the code actually does

The controller returns Ok() immediately after starting a Task.Run(), so the HTTP request itself does not timeout in the normal sense. What can go wrong is in two places:

1. ViewModelFactory.CreateView() runs synchronously BEFORE the task starts

If the product set is large, this DB/viewmodel call eats into the IIS request timeout (default httpRuntime executionTimeout is 90 seconds). For a DownloadCart scenario with ProductIds this is usually fast, but it is a real ceiling.

Also note in FillRequest():

Default PageSize is 10 if not passed, not "all". The DownloadCart template typically passes 100. If someone bumped this up significantly (or removed the limit), the initial query gets proportionally slower.

2. The background task is fire-and-forget --- and IIS will kill it

The Task.Run() has no CancellationToken, no timeout, no retry, no progress tracking. It runs as a thread pool thread. The problem: IIS app pool recycles will silently kill it mid-export. Default IIS idle shutdown is 20 minutes, recycling intervals vary by hosting config. A large export that takes 25 minutes just disappears --- no email, no error, no log entry (unless the catch fires before recycling).

The customer's "timed out" is almost certainly this: app pool recycled during a long-running export task.

3. Image processing is fully serial and all-in-memory

Inside Export():

Then in AddImagesToZip(), for every image in that combined list:

This is one synchronous ImageConverter.Convert() call per image, one after the other. No parallelism. For a Digital Assets Portal (which is specifically about assets), a product can easily have 10--50 assets. 100 products × 20 assets × average 8MB each = 16GB of image data to convert serially. Memory pressure alone will crash the process. Time alone will exceed any reasonable app pool lifetime.

Practical limits (not hard-coded, but practical)

Conclusion

The standard endpoint is designed for small-to-medium exports --- a few dozen products with a small number of images each, completing well within IIS app pool lifetime. For a Digital Assets Portal where the whole point is exporting many assets, it will hit these walls reliably at scale.

For reliable large-scale exports the architecture needs:

1. A real background job (Hangfire, a IHostedService, or DW's own scheduling) rather than Task.Run() --- so the job survives app pool recycles
2. Paging through products rather than loading all into memory at once
3. Writing images to disk as they are processed (streaming into the zip) rather than collecting all AssetInfo objects first and converting all at once
4. Progress/status feedback so the user knows it is running

The ProductViewModelsExporter.Export() approach (collect all → convert all → zip all) is a single-threaded, fully-in-memory pipeline that simply does not scale to what a Digital Assets Portal workload looks like in production.

Hi Nicolai,
If I read the conclusion correctly, the standard DAP export functionality should be used only for small exports (a few dozen products) and for small to medium asset lists.
Correct?
What is the alternative if the customer asks us to support larger exports?

Thank you,
Adrian
 

You might want to write down all the requirements in one list - what this thing should do. Then it is easier to give you something useful.

The export endpoint is not meant for integration purposes - but I have these 50 products for which I need assets.

If you want to do something more like a full export but as integration - you can do something like below. But might be better to collect all requirements so it is not these wild guesses in all directions.

Viable Architecture: Large Resumable Digital Assets Export (DW9 / Swift)

Context

The standard /dwapi/ecommerce/products/export endpoint uses ProductViewModelsExporter, which:

• Loads all products into memory at once
• Converts all images serially, one after another, all in memory
• Runs in a fire-and-forget Task.Run() with no cancellation, no checkpoint, and no survival past an IIS app pool recycle

This works for small exports. It does not work for 25,000 products with assets.

Core Problems to Solve

Architecture

[Admin UI / DownloadCart template]
        │ POST: start export (GroupId, LanguageId, fields, format...)
        ▼
[Lightweight API endpoint]
 → Writes ExportRequest parameters to a job file on disk
 → Triggers / schedules the DW scheduled task
        │
        ▼
[BaseScheduledTaskAddIn: DigitalAssetsBulkExportAddIn]
 → Load checkpoint (resume from last completed page if interrupted)
 → Page loop (e.g. PageSize = 500 → 50 iterations for 25k products)
    ├── ViewModelFactory.CreateView(settings, groupId)   ← paged
    ├── Write CSV/JSON rows to zip entry
    ├── Write assets per product directly into zip
    └── Save checkpoint after each page
 → Email download link when complete
 → Delete checkpoint

1. Job Runner: BaseScheduledTaskAddIn

DW's scheduler supports CancellationToken and runs outside the HTTP request lifecycle. It survives app pool recycles and appears in DW's task log.

[AddInName("My.DigitalAssets.BulkExportAddIn")]
[AddInLabel("Digital Assets Bulk Export")]
[AddInIgnore(false)]
public class DigitalAssetsBulkExportAddIn : BaseScheduledTaskAddIn
{
    [AddInParameter("Group ID")]
    [AddInParameterEditor(typeof(TextParameterEditor), "inputClass=inputControl")]
    public string GroupId { get; set; }

    [AddInParameter("Language ID")]
    [AddInParameterEditor(typeof(TextParameterEditor), "inputClass=inputControl")]
    public string LanguageId { get; set; }

    [AddInParameter("Shop ID")]
    [AddInParameterEditor(typeof(TextParameterEditor), "inputClass=inputControl")]
    public string ShopId { get; set; }

    [AddInParameter("Page size")]
    [AddInParameterEditor(typeof(TextParameterEditor), "inputClass=inputControl")]
    public int PageSize { get; set; } = 500;

    [AddInParameter("Recipient email")]
    [AddInParameterEditor(typeof(TextParameterEditor), "inputClass=inputControl")]
    public string RecipientEmail { get; set; }

    public override bool Run()
    {
        var checkpoint = LoadCheckpoint();
        int currentPage = checkpoint.LastCompletedPage + 1;

        // First call to determine total pages
        var settings = BuildSettings(1);
        var firstPage = ViewModelFactory.CreateView(settings, GroupId);
        if (firstPage == null || firstPage.TotalProductsCount == 0)
            return true; // nothing to do

        int totalPages = firstPage.PageCount;

        using var zip = OpenOrCreateZip(checkpoint.ZipPath); // ZipArchiveMode.Update if resuming

        for (int page = currentPage; page <= totalPages; page++)
        {
            if (CancellationToken.IsCancellationRequested)
            {
                SaveCheckpoint(checkpoint with { LastCompletedPage = page - 1 });
                return false; // scheduled task infrastructure will retry
            }

            settings = BuildSettings(page);
            var vm = ViewModelFactory.CreateView(settings, GroupId);
            if (vm?.Products == null) continue;

            WriteProductPageToCsv(zip, vm.Products);
            WriteAssetsForPage(zip, vm.Products);

            SaveCheckpoint(checkpoint with { LastCompletedPage = page });
        }

        zip.Dispose(); // finalize the archive
        SendEmail(checkpoint.ZipPath, RecipientEmail);
        DeleteCheckpoint();
        return true;
    }

    private ProductListViewModelSettings BuildSettings(int page) =>
        new ProductListViewModelSettings
        {
            LanguageId = LanguageId,
            ShopId = ShopId,
            PageSize = PageSize,
            CurrentPage = page
        };
}

2. Checkpoint for Resumability

A simple JSON file written after each completed page.

// Stored at: /Files/System/Export/bulkexport_{jobId}.json
public record ExportCheckpoint(
    string JobId,
    string ZipPath,
    int LastCompletedPage,
    DateTimeOffset StartedAt
);

private ExportCheckpoint LoadCheckpoint()
{
    var path = GetCheckpointPath();
    if (!File.Exists(path))
        return new ExportCheckpoint(
            JobId: Guid.NewGuid().ToString("N"),
            ZipPath: GetNewZipPath(),
            LastCompletedPage: 0,
            StartedAt: DateTimeOffset.UtcNow);

    return JsonSerializer.Deserialize<ExportCheckpoint>(File.ReadAllText(path));
}

private void SaveCheckpoint(ExportCheckpoint checkpoint) =>
    File.WriteAllText(GetCheckpointPath(), JsonSerializer.Serialize(checkpoint));

private void DeleteCheckpoint() =>
    File.Delete(GetCheckpointPath());

private string GetCheckpointPath() =>
    SystemInformation.MapPath($"/Files/System/Export/bulkexport_{ScheduledTaskName}.json");

On IIS app pool recycle: the next scheduled run reads the checkpoint and resumes from LastCompletedPage + 1. On completion: checkpoint is deleted.

3. Memory-Bounded Image Writing

Write assets directly into the zip per product — never accumulate a List<AssetInfo> across all products.

private void WriteAssetsForPage(ZipArchive zip, IList<ProductViewModel> products)
{
    foreach (var product in products)
    {
        foreach (var assetCat in product.AssetCategories)
        {
            foreach (var asset in assetCat.Assets)
            {
                var fullPath = asset.Value.StartsWith("/") ? asset.Value : "/" + asset.Value;
                if (!File.Exists(SystemInformation.MapPath(fullPath)))
                    continue;

                var bytes = ConvertImage(fullPath);
                if (bytes == null) continue;

                var entryName = $"{product.Number}/{Path.GetFileName(asset.Value)}";
                // Skip if already in zip (resuming)
                if (zip.GetEntry(entryName) != null) continue;

                var entry = zip.CreateEntry(entryName, CompressionLevel.Optimal);
                using var stream = entry.Open();
                stream.Write(bytes, 0, bytes.Length);
            }
        }
    }
}

Memory at any point = one page of products (e.g. 500) and their images — not 25,000.

4. On-Demand Triggering

The scheduled task can be:

• Triggered manually from the DW backend (Scheduled Tasks UI)
• Triggered via URL using DW's existing UrlAddIn and an HTTP call to the scheduler endpoint
• Triggered from the DownloadCart template by posting to a small API endpoint that writes the job parameters and fires the task

The lightweight API endpoint just writes parameters and triggers — it never does the actual work.

Tuning: Page Size

The page size controls both memory usage and checkpoint granularity. Smaller = more checkpoints but safer on memory.

What This Gives You

What This Does NOT Change

• The existing /dwapi/ecommerce/products/export endpoint is left completely untouched.
• ProductViewModelsExporter is not modified.
• The new add-in is a standalone provider DLL, deployable independently.

Hi Nicolai,
Fair enough. Here is the ideal definition:

1. Support a large volume (thousands) of products and mixed assets (images, videos, PDFs).
2. Support selection of languages for the data exported
3. Support a default set of fields for export and the ability to adjust the list of fields exported
4. CSV/JSON/XML
5. Asset list separated into a dedicated CSV for CSV export, with a dedicated schema
6. Asset files are separated into folders for each product inside the archive
7. Custom schema for JSON and XML
8. Receive the result archive by email

Other considerations: Performance (DAP runs on the same solution as the main webshop), Storage space (if multiple customers are requesting the full catalog around the same time, the archives may fill up space very quickly).

I believe the above captures all requests from our customer.

Thank you,
Adrian

I think this needs some more information - and some thought of making something that is actually possible in production. 

The real issue is scale, not only feature scope.

For 100 products and their assets, synchronous export can be OK.

For “thousands of products and mixed assets,” the first thing we need is more exact volume profile, because viability depends much more on asset count and asset size than product count.

The dangerous combinations are things like:

• 20,000 products × 10 assets each = 200,000 assets. How many products? How many pdfs? How many videos? How many images?
  • If we copy all of this into seperate folders from the origin for each export. I guess you can see the issue.
  • Are images generated for a particular export - that is an extreme amount of image generation - and probably requires dedicated servers.
• if those assets include PDFs and videos, the archive size can become extremely large. Might even run into issues with max zip file sizes or download sizes.
• if multiple customers request full-catalog exports around the same time, both CPU, IO, and storage pressure become serious because DAP runs on the same solution as the webshop

The main technical risks are:

1. Runtime / timeout
A full export with product data, folder creation, manifest generation, schema mapping, and archive compression is unlikely to be safe as a synchronous front-end request once we move into thousands of products.

2. Storage
If each request generates a temporary archive and customers request similar large exports close together, disk usage can spike fast. Retention and cleanup become mandatory.

3. Asset weight
Images are manageable if resized and optimized. PDFs are often much larger. Videos are the biggest concern because even a small number of video files can dominate total archive size.

4. Main webshop impact
Because this runs on the same solution, heavy export jobs can affect normal browsing, search, checkout, and indexing unless queued or offloaded.

5. Repeated duplicate exports
If many users export “full catalog in all languages” with the same settings, we should consider caching or reuse of already-generated archives instead of regenerating everything each time.

So - some numbers are needed - how many products, exports, images etc. It might not be a very good idea to implement this like the current export endpoint. I see a lot of potential issues.  

If the numbers are this extreme it might be an idea to queue the jobs up and run them - either in low priority threads one by one when requested - or during the night (alos one export at a time) so they are ready each morning offloading the job to times that are not busy.

Also - you have defined the solution you believe is needed - but not the problem/pain you are trying to solve. The problem is not how to generate and download an enourmous zip-file. Have other solutions been considered that e.g. does not copy all assets every time? Basically - why should it work like described?

Hi Nicolai,
All are very pertinent topics/questions/concerns.
We will address them with the customer. We got very far with the current implementation, only to fail on volume cases.
The underlying question is how often volume requests occur and if those requests should even be handled through the DAP interface.
Will get back on the topic next week after talking with the customer.
In the meantime, can you help me understand the pattern for requesting ProductFields and ProductCategoryFields through the regular export API endpoint?
Thank you,
Adrian

Hi Nicolai,
I am circling back to this topic as I am trying to optimize what we have so far.
When setting up Feeds, there is an option to include the full url for assets. Is there something similar in the FilledProperties that I can use?
In your example, you suggested that Price.Price would be a supported way of asking for a property of a Class. Is this approach going to work as well for ProductFields and ProductCategory fields? Is there a valid pattern I can use?
Thank you,
Adrian