Developer forum

Hi Justin,
can you try to use an alternative approach with Odata queries instead? It shold be possible to setup the Endpoint and configure the Odata provider with the page size option
and then do the import.
BR, Dmitrij

Hey Dmitrij,

Sure, I can do that.

I made a start this morning, I configured the oData service which will fetch around 10K products, as a test case i added a $top, 10 headers, and with that I get a snappy response in the endpoint editor.

A problem I have now is that when I work on the mappings in the activity screen, selecting a mapping, and connecting the source to a destination column, takes around 20 seconds per column.

The activity is set to Full Replication, with a maximum page size of 10 as well.

Is there anything I can configure to make the column mapping process more responsive? It basically seems that for some reason every action i take in the activity, that check something on the source is taking 20+ seconds...

Thanks,

Justin

Hi Justin,

We are going a bit off topic here, but there shouldn't be any lag when making column mappings as the schema is saved in the job file and the endpoint is only called when you run the job or reload schema. Could you make a video of what you are experiencing? 

BR Rasmus 

Hey Rasmus,

Thanks, sorry about that, this is where reality seems to be taking us :-)

I do have some curious behaviour. As you can see in the screencast https://screenrec.com/share/EN7hHr1s3l 

When I select an oData activity, and try to manage the mappings, my log is filled with exceptions that the AddIn manager cannot find the Import Data with Paging Addin. On another machine I tried the same, with the same result. I tried deleting all other activities, and recycling the app, but it keeps showing this behaviour.

What more can I check for references to this dll? I have to the best of my knowledge not added something explicitly. Anbd I also tried removing the entire "Import Data Custom Request" app, thinking it might be related to that. Not sure 

On a side note, I was attempting to get a fresh view with the Event Viewer, so deleted all files in \System\Log\EventViewer, but the event list still shows a first page, when I select "All" it says there is nothing to show, this list is also still shown after a full reboot of the machine (some very persistent caching in the EventViewer?)

Thanks,

Justin  

Hi Justin,
can you check your scheduled tasks? The ImportDataWithPaging addin wasn't moved/developed for DW10, so it can not be referenced by anything from DW10. Maybe the problem is that you have the scheduled tasks (in the database ScheduledTask table or in the /Files/Integration/ScheduledTasks/ folder) that were created in DW9 and have a references to the add-ins that doesn't exist in DW10. Could you check that?
The OdataProvider or EcomProviders don't have references to the scheduled task add-ins.
BR, Dmitrij

Hey Dmitrij,

Turns out there was a orphaned scheduled task in the database, which was created probably because one of the other issues we discussed on the forum regading saving a Parent task with childen. I deleted it and the errror regarding this are gone. but the lag in the mappings for an oData activity still occur. 

If I change the oData endpoint in the source to a service that return only about a 100 lines I do get a quick response, when I change it back to the endpoint that returns 10K rows the mapping is slow again, so it does seem like it is related to calling the service anyway? see a test with bot a small and large table: https://screenrec.com/share/84zoYeCBvd 

Before updating the solution I did check https://doc.dynamicweb.dev/documentation/fundamentals/dw10release/upgradingfromDW9.html which currently does not mention anything about the CustomRequestWithPaging addin, it would make sense to add it there I think.

Hi Justin,

I am a little confused about your problem, so correct me if I am wrong.

You are currently working on a DW10 solution, and looking for the "Import data with Paging add-in" as a scheduled task.

Is that because of an upgrade from a DW9 to DW10? in that case, you need to find a different solution for it, like Dmitriy says, you can setup an activity, where you have the OData Provider as Source to whatever destination Provider, and in the settings for the OData Provider you can define the "Maximum page size" to take the wanted amount of elements pr response from the OData Endpoint.

If you want to schedule this activity you can do that directly in the Overview screen of the activity:

I've tested on my localhost with latest version of DW10, against F&O to see if i could trigger any load issues on the mapping screen, and I seems not to experience the same as you, even when I have more than 3k tables in the list for SourceTables to select from.

DW10 should work just the same as DW9 when it comes to loading the Schemas for the Source & Destination (it stores it into the job-file). But if you are copy the jobs from a DW9 solution into a DW10 solution, then I would recommend you to create the jobs over again directly in DW10.

If I am seeing ghosts, then I need to know a little more about your setup, what is it you want to achieve? :)

BR

Matthias Sort

Hey Matthias,

Thanks for folluwing up. I think I saw a ghost ;-) because indeed I upgraded from 9. 

Before updating the solution I did check https://doc.dynamicweb.dev/documentation/fundamentals/dw10release/upgradingfromDW9.html which currently does not mention anything about the CustomRequestWithPaging addin, it would make sense to add it there to the list of deprecated modules, with the note to replace with oData.

I do still have the problem with a slow mapping screen, but I will create a seperate thread for that later today...

Thanks,

Justin