Batch Integration

A batch integration is a file-based integration between a Dynamicweb solution and a remote system of some sort – typically an ERP system. A batch integration consists of:

  • One or more data integration jobs which move data between Dynamicweb and the file system
  • One or more batch integration activities which connects to a remote system, requests data, and saves it to disk as a file

Before starting work on the Dynamicweb side of an integration project, you must mock, retrieve, or receive an XML file with data per feature included in the integration. We recommend you retrieve the data using the Dynamicweb Connector TestTool.

A data integration job is a set of instructions for taking data and putting it somewhere, either in the solution database (import) or in a file (export).

It has two components:

  • A source provider which matches the data source – usually an XML file (import) or the Dynamicweb database (export)
  • A destination provider matching the data destination - also usually an XML file (export) or the Dynamicweb database (import)

Each provider can also be configured, tweaking the behavior and e.g. deleting data which is nor part of the source file, and so on. You can read more about data integration jobs and providers in the Data Integration Module documentation.

In typical scenarios, you will create a number of separate jobs -  often one per feature - and schedule them at intervals which make sense for the feature and the solution. For instance, currencies are typically not updated very often, whereas prices and stock levels are updated frequently.

When starting an integration project, you likely won’t have any ‘real’ data in the system to begin with. So, for each feature you must do an initial data import to populate the database with data from the remote system – we recommend using the Dynamicweb Connector TestTool to produce XML files with the correct data.

Once you have retrieved or received that data do the following:

  • Upload the appropriate XML file to the /Files/System/Integration folder on the solution
  • Create a new integration job matching the feature
    • Select the XML Provider as source provider
    • Select the appropriate XML file as the source file
    • Click Next and select the appropriate destination provider
    • Check any relevant checkboxes – see the Integration Provider articles for details
    • Click Next
    • Select which source tables to import
    • Map them to an appropriate target table (this usually happens automatically)
    • Click Next
    • Name the job
  • Review the column mappings for each table in the job
  • Save
  • Run the job

For each job, review the results thoroughly. Any custom data on the remote side must be handled on the Dynamicweb side as well, and placed in either standard field or a custom field of the appropriate data type.

Remember, you have access to a number of useful standard features such as context-sensitive values, conditionals, scripting, etc. 

A batch integration activity is a scheduled task which does three things when executed:

  • Connects to a remote system – directly or via the DynamicwebConnector service – and submits requests for data using a scheduled task add-in
  • Receives a response in XML from the remote system and saves it to disk
  • Triggers an integration job, which processes the XML file and places the data in the Dynamicweb database

To create a batch integration activity:

  • Go to Settings > Integration > Integration Framework Batch
  • Click Add in the toolbar and fill in the general information (Figure 3.1)
Figure 3.1 Creating a new batch integration activity

You must:

  • Name the task
  • Select a begin and end time
  • Specify an intervalonce, hourly, weekly, etc.
  • Use the Type dropdown to select and configure an appropriate scheduled task add-in

scheduled task add-in connects to a remote system and submits a request for data whenever the scheduled task is run. Each add-in comes with a set of parameters which are required by the remote system and/or Dynamicweb.

Out of the box, we supply the following scheduled task add-ins for batch integration tasks:

Name

Use with

Notes

Active Directory Import data add-in

Microsoft Active Directory

Used to import users and validate user credentials

Export data add-in

Any remote system

Used to e.g. export order data

Import data add-in

Any remote system

Basic add-in for users & data (products, prices). Used with Integration Framework v.1

Import Data from Multiple Requests Add-in

Any remote system

Used to request data from a ConnectorService Add-in with multiple end-points. For custom integrations.

Import data with custom request add-in

Any remote system

Used to submit custom request against e.g. a modified remote system plugin. Otherwise very basic.

Import data with paging add-in

Any remote system

Used to connect to remote systems with a plugin that can react to paging requests, reimport requests, etc.

 

This is true for our AX2012R2 and D365 plugins.

Import Perfion Images Add-in

Perfion

Retrieves image data from a Perfion solution by querying the Perfion API Service

Sync queued orders using Live Integration

Any remote system with Live Integration and queued orders enabled

This add-in is used to sync queued Live Integration orders with the remote system, for scenarios where the connection has been unavailable for a while

In general, system-specific add-ins connect to remote systems which are passive, and where the returned data must be transformed on the Dynamicweb-side of the integration.

For the Integration Framework v2 code units, we recommend that you use the Import data with paging add-in as it has been specifically developed for this purpose.