Developer forum

Forum » Integration » Make "Last run" more reliable

Make "Last run" more reliable

E-mail notifications
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply
Posted on 28/10/2025 09:20:17

Hi there,

I have a data integration job using the OData Provider with delta replication. That means that when a request is made to the ERP, a filter like the following is appended to the URL:

$filter=lastDateTimeModified+gt+2025-10-27T21%3A15%3A29.000z

That's working really well when it works as it should and only returns rows since that date, which is the previous date and time the job ran successfully.

The implementation, however, has two flaws:

  1. It depends on the file JobName_lastrun.log that keeps track of the last date and time this job ran. When then the log file is deleted (due to cleanup for example), there's no date and therefore no filter which means all data is pulled back. Due to a stupid bug with log retention, this happens more often than I like.
  2. It depends on the previous run completing successfully. Here's a flow that's causing me issues:
    1. Job runs with data filter and completes successfully, date is written to log file
    2. Job runs again and uses date filter. However, the job fails (for example because the ERP is down). 
    3. The job runs again; because of the previous failure no date filter is applied

Can this be solved in a different way? 

Imar

 

 


Replies

 
Matthias Sebastian Sort Dynamicweb Employee
Matthias Sebastian Sort
Posted on 28/10/2025 12:55:28
Reply

Hi Imar,

Our OData Provider is currently looking at the Job.LastSuccessfulRun function witch only returns a datetime if the last run was a type of "JobResult.Completed" or "JobResult.CompletedWithError" or if the last run job is not completed it will do a loop through all logfiles for the job in decending order of File.CreationTime looking at the last line written in the file and if it endswith "Batch succeeded." or "Job succeeded." it will use the CreationTime from that file, if there is no match the $filter will not be added to the URL and result in a full replication/import.

I can think of saving the date into the Job, but since our feature management for schema management is not "forced" enabled yet, it will cost too much to do a Job.Save everytime the job is completed.

BR

Matthias Sort

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Posted on 29/10/2025 12:06:56
Reply

Performance isn't really my worry here, either with or without schema management. But I think writing to the job at run-time is a bad idea anyway. Oftentimes, these jobs are source-controlled and auto-deployed, so with the date in the job you would reset the date every time a deployment takes place.

What about global settings like /Integration/DataIntegration/Jobs/JobName/LastRunDate ?

Imar

 

You must be logged in to post in the forum