Developer forum

Forum » Integration » Better error handling for OData Provider

Better error handling for OData Provider

Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

Hi there,

We use the OData Provider to synchronize orders from DynamicWeb to BC. However, we run into issues from time to time with the setup. Because with BC OData we need to use the V2 and V4 end points, and because we need to send separate entities to distinct end points (orders to the orders end point, followed by lines to the order lines end point), the integration doesn’t use an atomic transaction as Live Integration with the code unit does, we run into trouble when the job fails mid-way.

Here's a simplified version of the integration set up:

Job 1

  1. Using V2, we find all orders in the New state and synchronize to the salesOrders end point
  2. We use Response mapping to save the order integration ID and number back to the order, and set the state to “In progress”
  3. In the same job, we find all order lines belonging to orders in the “In progress” state and send them to BC.
  4. We use Response mapping to save the Sequence ID from BC back on the individual lines

Job 2

  1. Using V2 we update all order lines and set “Allow invoice discount to true”

Job 3

  1. Using V2 we set the total order discount on the order

We frequently (every other week or so) run into an issue where step 4 results in a timeout (or other error) and the sequence IDs are not saved correctly from the Response mapping. However, as part step 2, it looks like the order state for the current job has already been changed (so the lines mapping in step 3 and 4 filters correctly ) but not saved to the database . We then get errors from BC that we cannot change the sequence IDs when we run the job again because we send 0 as a value, while BC already has orders lines with different IDs, causing the job to fail with the error:

The value for sequence cannot be modified. Delete and insert the line again.

This in itself is already a bit of a hassle for one order, but the problem gets worse because the next time the job runs, it’ll pick up the same (faulty) order again, which then fails the entire job again. This means we end up with hundreds of orders that cannot be synced over the weekend, when we have one that is failing.

I noticed the “Continue on error” option on the provider which may help with this, but I am a bit reluctant to turn it on as it then may continue with broken data. Do you have any recommendations as to when to turn this on or not?

I think the better solution would be to have an “Error mapping” or so that allows me to change the order that is being processed when an error occurs. That way, I could set the order state to Failed and skip it in subsequent runs.

Would love to have a better, more robust solution for this as it drives the customer nuts when they have a whole bunch of unsynced orders when only one has failed.

Running on DW 9 but I assume this will be the same on 10.

Thanks!

Imar


Replies

 
Matthias Sebastian Sort Dynamicweb Employee
Matthias Sebastian Sort
Reply

Hi Imar,

What source provider are you using in job 1, if you are not using the Order Provider is it Views? then you could do a “order by” OrderLineAutoId desc then the faulty once are always last in the queue for the export?

“Continue on error” is an option where the OData Provider kinda tries to ignore errors from the endpoint and move on to the next codeline (mainly when waiting for a response), we’ve added a retry handler when OData Provider is destination on version 3.0.11 (only on DW9 for now) when it gets an error from the endpoint it retries 5 times (when the error is a type of HttpRequestException or WebException) and if it keeps getting an error it then throws this and stops the job. The OData Provider calls the endpoint twice, first to see/check if object/element exists (via your selected key-columns) secund time is to do the actual POST/PATCH call. So, I will only recommend this option, when the data you export is more “simple/not that complex” as Order+OrderLines are.

I do like your idea of “Error mapping” and that could solve this issue and even others in the future, but we are not doing feature requests on DW9, so this would be something we implement on DW10 only.

BR

Matthias Sort

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

>> if you are not using the Order Provider is it Views? then you could do a “order by” OrderLineAutoId desc 

Yes, it's views. Sorting is something we could consider, although it's bit black magic. Also, not sure if it'll work. If we order descending, then just the latest order becomes the first faulty one.

>> So, I will only recommend this option, when the data you export is more “simple/not that complex” as Order+OrderLines are.

I see. That's what I was afraid of.

>> we’ve added a retry handler when OData Provider is destination on version 3.0.11

I looked at the implementation but I don't think that will help us in our scenario. The retry is at the rest level while in our case the job fails when saving back data we got from BC higher up the stack. With this retry change, we'll just submit the same faulty data five times with the same result, with a second of waiting time in between.

>> I do like your idea of “Error mapping” and that could solve this issue and even others in the future, but we are not doing feature requests on DW9

Understood; that makes sense, thanks!

Imar

 

You must be logged in to post in the forum