Batch Integration

The batch integration framework uses scheduled tasks to transfer data and maintain data consistency between a Dynamicweb solution and a remote system – typically a Dynamics NAV or AX solution, a CRM system or a product information management (PIM) system.

A batch integration consists of periodic updates, where data is retrieved from a remote system and imported into the Dynamicweb database – and exported from Dynamicweb and imported into a remote system. 

Figure 1.1 A batch integration

To outline the dataflow:

• During an import, a call to the web service exposed by the Dynamicweb Connector retrieves the data to be imported and saves the file in XML format on disk – the content is then imported into the Dynamicweb database using the Data Integration module.
• During an export, the Data Integration module is used to pull data from the database and save it in XML format in the file archive. The saved data is then sent via the web service to the remote system. When the remote system has received the data (e.g. new orders or users), it sends an acknowledgement back to Dynamicweb containing a list of IDs that the data items were saved with in the remote system. The remote system IDs are then added to the newly exported records in the Dynamicweb database.

In this article we will cover three things:

• How to setup a batch integration on your remote system and in Dynamicweb
• What kind of data formats Dynamicweb exports – and expects to receive
• How to modify the data flowing between the remote system and Dynamicweb

Creating a batch integration involves:

• Installing and configuring a plugin/code unit and the Dynamicweb Connector on a server which can access your remote system
• Testing the connections using the Dynamicweb Service Test app
• Creating and configuring integration activities in Dynamicweb
• Scheduling the activities to run at appropriate intervals

Below, you can read through a typical implementation scenario to help you get started.

The first thing you need to do is ready your remote system – your NAV or AX solution, your CRM, or your PIM – for an integration with Dynamicweb.

In most cases that involves:

• Installing and configuring a plugin on your remote system
• Installing and configuring the Dynamicweb Connector web service on your remote environment

The setup procedure varies according to the particular remote system you want to integrate to – find the how-to for your system below.

Once you’ve completed the appropriate how-to, you can test the connection using the Dynamicweb Service Test app.

The Dynamicweb Service Test app (Figure 5.1) is a small tool for testing the web services exposed by the DynamicwebConnector, and by Microsoft NAV, AX and CRM when using the example plugins.

Which service you are testing the connection to is determined by which tab is active.

Figure 5.1 The Dynamicweb Service Test app

You can use the app for two things:

• Testing the connection to NAV, CRM or AX during setup
• Testing the connection to the DynamicwebConnector

When testing the connection to NAV, CRM or AX, the configuration (endpoint, username, password, etc.) is taken from the configuration file. This works identically to the configuration of the DynamicwebConnectorService. Basically, if you can connect to your AX, NAV or CRM web service using the Test App, you can copy the configuration directly into you DynamicwebConnectorService configuration file, and it should work.

When testing the connection to the DynamicwebConnector, the settings used are the ones visible in the "DynamicwebServiceTest" tab. The contents of the test app configuration file is ignored. The URL and secret fields should contain the same URL and secret you defined in the DynamicwebConnectorService configuration file.

The Dynamicweb Service Test app must be installed and run from the server on which the DynamicwebConnector is installed.

When you’ve completed the setup guide appropriate to your remote system, you are ready to set up your Dynamicweb solution.

This involves:

• Populating the Dynamicweb database
• Setting up continuous integration
• Creating activities for continuous integration
• Set up scheduled tasks for the initial and continuous integrations

To get things started you must first populate your Dynamicweb database with data from your remote system, i.e.:

• Import products, languages, currencies, etc.
• Import users

To do so you must create data integration activities and create and run a batch integration task.

To create the data integration activities:

• Go to Settings > Data Integration
• Click New activity from template
• Create an activity based on the ErpDataImport template
• Create an activity based on the ErpUserImport templates

The templates are set to use automatic mapping, which means that as long as the input data is named identically to the tables and columns in the database everything will work out of the box.

When you’ve created the integration activities you must create and run a batch integration task:

• Go to Settings > Integration > Integration Framework Batch
• Click Add in the toolbar to create a new batch integration task (Figure 7.1)

Figure 7.1 A batch integration task

To configure the batch integration task you must:

• Name the task
• Specify a start and end date (and time)
• Specify at which interval you want the task to be run (once, every hour, every week, etc.). Since you are populating the database, run the job manually instead of scheduling it.

Using the type dropdown, select the import data add-in (Figure 7.2) and configure it

Figure 7.2 Configuring a batch integration task add-in

To configure the import add-in you must:

• Enter the web service URL exposed by the DynamicwebConnector (which you also used in the test app). By default this is http://YourServerNameOrIP:8090/DynamicwebService .
• Enter the security key which is the secret key from the DynamicwebConnector service configuration file (which you also used in the test app)
• Select a user import activity and/or a data import activity to use (the ones you just created)

You can use the user import settings and data import settings to send data to the root node of the request XML. Make sure it is formatted as valid XML attributes, e.g. ShopId=”Shop1”, which will be sent as <GetEcomData ShopId=”Shop1><tables>…</tables></GetEcomData>. In the Data Integration activity settings field you should also correspondingly enter shopID=”SHOP1”.

In the initial phase you should enable logging – whereas email notifications are more useful for production scenarios. Log files can end up taking up a lot of space, so disable as soon as you have a stable setup.

It is important that you run the Data Integration activities via this batch integration task, or you will not properly connect to the DynamicwebConnector service, and your import will not behave as intended (old date, no updates, and fail without an error).

After the initial import, you may want to schedule import activities for weekly, daily or hourly updates from the remote system – so you have current and updated prices, stock levels, users, etc.

In short, this is done by creating integration activities similar to the ones used for populating the database, but adjusted to only update the desired fields.

The activities should usually cover:

• Importing products
• Importing users
• Exporting users
• Exporting orders

The Data Integration module contains templates for each of these activities, but you probably need to change or replace them with custom activities in some cases. For instance:

• If you want to use (preserve) descriptions created in Dynamicweb, you should import products but not map to the ProductShortDescription and ProductLongDescription fields.
• If you have a very large product catalog, and you only really need to update stock info and prices, you may want to create a custom import job which maps only to the key fields (ProductID, ProductVariantID & ProductLanguageID) and to the ProductStock and ProductPrice fields, in order to improve performance.

In principle, you can import and export anything you’d like using the Dynamicweb Provider – you just need to make sure your remote system is equipped to handle the data accordingly.

The activities for continuous integration are scheduled and run in exactly the same way as the initial import was – and the export data add-in is configured in the same manner as the import add-in.

All data entering and leaving Dynamicweb follows the same general structure:

Additionally, most solutions employ some form of custom data – whether it’s custom product fields, custom user fields, or something else entirely – and this is especially true for solutions with integration, as you often need to create custom properties or fields for storing data from your remote system which are not present in Dynamicweb by default.

At any point in time, you can create a data integration activity (using the Dynamicweb provider as the source provider and the XML provider as the destination provider) and export the relevant parts of your Dynamicweb database to view the exact markup for your solution

When importing data, it’s sometimes necessary to create data integration activities using manual mapping, in order to make sure the data from the remote system is mapped correctly into the custom fields in Dynamicweb. However, if you are able to convince the remote system people to name the data in their custom fields exactly as they are named in the Dynamicweb database you can still use automatic mapping.

To help you get started, we’ve described some of the most common integration use case scenarios below.

For assortments, the data format is as follows:

The mandatory fields for each table are:

•  EcomAssortmentGroupRelations:
  • AssortmentGroupRelationAssortmentID (ID of assortment)
  • AssortmentGroupRelationGroupID (ID of product group)
• EcomAssortmentPermissions:
  • AssortmentPermissionAssortmentID (ID of assortment)
  • AssortmentPermissionCustomerNumber (customer number the assormtent is for)
• EcomAssortmentProductRelations:
  • AssortmentProductRelationAssortmentID (id of assortment)
  • AssortmentProductRelationProductID (id of product in the assortment)
• EcomAssortments:
  • AssortmentID
  • AssortmentActive
• EcomAssortmentShopRelations:
  • AssortmentShopRelationAssortmentID
  • AssortmentShopRelationShopID

For currencies the data format is as follows:

The only mandatory field is CurrencyCode (EUR, DKK, etc.).

For order and order lines, the data format is as follows:

The mandatory fields are:

• EcomOrderlines
  • orderlineID (must be unique)
• EcomOrders
  • orderID (must be unique)

In some cases, you will need to modify the data flowing between Dynamicweb and your remote system – either because you want to import or export data which is not covered by our standard functionality, because you need to send a non-standard set of parameters to your remote system, or because you are creating an integration with a remote system which is not included in the framework by default.

Broadly speaking there are three places to accomplish that:

• Custom Scheduled task add-ins
  While there are several standard scheduled task add-ins available for batch integration, these may not be adequate for your needs. In that case, you can create custom scheduled task add-ins, which can handle and modify the data being sent to and received from the Dynamicweb Connector.
• Data Integration tasks & custom data integration providers
  Both imported and exported data is handled by the Data Integration module. This means that you can customize the data flow here as well, either by configuring your data integration tasks – or by creating a custom Data Integration provider.
• Custom Dynamicweb Connector add-ins
  If you customize the Dynamicweb Connector add-in you have full control over which data is returned by the Dynamicweb Connector, since your custom add-in code handles the dataflow to and from the remote system. Custom add-ins are usually only relevant if you are integrating with a remote system for which we do not provide a standard add-in, or if the remote system is heavily customized.

If you need to implement functionality which is not possible using our standard batch integration task add-ins – e.g. retrieve product images from an external system – you can create custom scheduled task add-ins.

A template for creating custom batch integration task add-ins are included in the Visual Studio templates available from the Downloads page. The template contains the full source code for one of the standard scheduled task add-ins that come with Dynamicweb – it should be fairly straight forward to adjust as needed.

You will be using the ErpServiceCaller helper class to communicate with the DynamicwebConnector.

Batch Integration tasks are executed using the Data Integration module, which means that you can modify data in all the standard ways associated with that module:

•  Use the built-in mapping tools, such as conditionals, scripting and XSLT transformations
• Use a different Data Integration provider, e.g. the Ecom provider, which can auto-generate product information from incomplete data
• Creating a custom data integration provider – or extending an existing one

The Dynamicweb Connector receives and transmits data to and from a remote system and Dynamicweb.

The Dynamicweb Connector add-in interface is fairly simple; it has one method, which takes a string as an argument, and returns a string.

When you create your own version of the add-in, you must

• Handle the incoming xml sent by Dynamicweb
• Return xml in the format which Dynamicweb expects

A template for creating custom connector add-ins is included in the Dynamicweb templates for Visual Studio.

The request for data from the remote system is fairly simple:

Your code should capable of returning data for just one of the tables requested. So for an example like…

..you should only return data for users – and not for products, currencies or any other tables.

You can safely ignore requests for data not used in your implementation (e.g. if you do not import currencies and languages but create them directly in Dynamicweb) – you should get no exceptions.

It is also ok to include more data than is requested when it makes sense – e.g. when you request ‘products type = “all”’, you can include groups, variants, variant options, etc. – as this data may be necessary to present products correctly.

The attribute “type” is used to indicate if this is a batch request (where you want to retrieve all data for a given table) or a live request (where you want to retrieve only a subset of the data, e.g. for the products shown in the page you are rendering).

Since batch integration imports data using the Data Integration module, the return format can be any format accepted by the data integration provider used – please see the Data Integration module documentation for a general overview of the standard providers, and each provider article for provider particular information. The basic format for needed for the return data can be seen in AllTables.xml and Users.xml .

Please note that these example XML files contain the columns accepted by Dynamicweb, they are not all mandatory - however:

• When importing users using the default User Provider activity, the AccessUserUserName must be present and unique for each user, and the same applies to AccessUserExternalID (if you use the Dynamicweb provider, you can define your own keys)
• When importing products the minimum amount of data which makes sense is a ProductID, alongside either ProductPrice or ProductStock. Usually employs the Dynamicweb Provider, although the Ecom Provider may be a valid alternative, depending on your needs.

Exporting orders from Dynamicweb to the remote system is done with PutEcomOrders – see PutEcomOrders.xml for an example. The example contains a single order, but could just as well contain more, and each order should be handled.

The return XML format which Dynamicweb expects to get back is fairly simple. If we export two orders (ORDER 19 and ORDER20), we expect to receive XML looking like this:

The ExternalOrderIDs are then saved to the Dynamicweb database, and used to ensure that the same order isn’t exported multiple times. Additionally, the external orderId is visible in the Dynamicweb backend, in order to make it easy to see how the orders in Dynamicweb relate to the orders in the remote system.

Exporting users from Dynamicweb to a remote system is done with PutUsers – see PutUsers.xml for an example.

The return xml format which Dynamicweb expects to get back is fairly simple. If we export two users, with AccessUserID 1 and 2, we expect to receive XML looking like this:

The ExternalUserIDs are then saved to the Dynamicweb database.

Any settings you need in your custom add-in – e.g. connection strings for databases, references, login credentials, etc. – can be saved and loaded from the config file associated with the DynamicwebConnector.

The settings are retrieved as usual, from the configuration. Assuming you have a setting called “ConnectionString” in a section called “ExampleConnectorAddin”, you can retrieve it using the following code:

When you have created a custom add-in:

• Copy the dll to the folder where your DynamicwebConnectorService is located
• Open the DynamicwebConnectorService.exe.config file and locate the <appSettings> node
• Change the value of <add key="ErpConnectorType" value="NavConnectorAddIn.NavConnector"/> to the fully qualified name of your custom connector, e.g.: