Hello,
In DW9, how does the platform handle delta replication within a single OData integration job using the provider when:
- Multiple source tables are included, and
- Each table uses a different field to track the last-modified timestamp?
For KZ (https://int-kzrv.mydwsite3.com/admin/), we currently extract between three and six OData entities within a single integration job and load them into staging tables as the first step of the integration process. However, when implementing delta replication we use multiple different fields across the entities to track. See example below:
https://app.screencast.com/WKFfmAjnmKjoN
One theoretical option would be to split these entities into separate integration jobs so that the appropriate last-modified field can be defined per entity. However, this approach introduces significant operational drawbacks.
Specifically:
- Current OData integration jobs: 13
- Current scheduled tasks: 12
- Projected integration jobs if split: 27
- Projected scheduled tasks: 33
This level of fragmentation would more than double the number of integration jobs and nearly triple the number of scheduled tasks we must create, monitor, and maintain. As a result, administrative overhead would increase substantially, performance could be impacted, monitoring requirements would expand, and additional points of failure would be introduced. Given these impacts, splitting the jobs is neither sustainable nor scalable.
What other options are available to support delta replication in this scenario without introducing unnecessary operational overhead or long-term maintenance burden?
