Developer forum

Forum » Integration » Add option to allow of data

Add option to allow of data

Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

Hi there,

It would be nice if there was a column-level setting on a data import job that would allow truncation of data to the maximum length. Sometime we get data that is longer than the target column's width. Currently you get an exception when the data doesn't fit but with this new setting the data would be truncated to the max length of the column.

Thanks for considering!

Imar


Replies

 
Rasmus Sanggaard Dynamicweb Employee
Rasmus Sanggaard
Reply

Hi Imar,

This could potentially lead to a lot more problems, especially relating to primary keys etc. 

Do you have some examples of the issues you run into, maybe the solution is something else. 


Best regards 
Rasmus Sanggaard

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

As an example a street name that is longer than 255 characters to be imported into AccessUserAddress. It's not uncommon to have junk in there like "old address do not use, use bla bla instead, watch out for the dog and call Bob before delivering your package". This currently crashes the entire integration and the only way to fix that is:

1. Fix the data. This is of course the best solution but cannot always be done on time for an integration engineer to continue working

2. A Table Script to trim the data. Lots of work for something as simple as truncating data

With this new option turned on, truncating would happen automatically and cap the length at 255.

And it won't be an issue for PK as they would not exceed the max length normally and you would not turn on this option for a PK column anyway. And if they would exceed the max length and you did turn on that setting, I think you have other problems to worry about also :-)

Imar

 
Rasmus Sanggaard Dynamicweb Employee
Rasmus Sanggaard
Reply

Hi Imar,

If it's a common issue that street names are typically longer than 255 char. then I think we should fix that instead, even though it was just an example. I believe it's a slippery slobe to allow for incomplete data and would just lead to other problems down the line. 

It's not something that is going on our backlog at this time, but I will keep it in mind if I get similar requests.

 

Best Regards

Rasmus Sanggaard 

 
Lars Hejgaard Sørensen Dynamicweb Employee
Lars Hejgaard Sørensen
Reply

Hi Imar,

There is an alternative by setting the "Persist successful rows and skip failing rows" option, so that records where e.g. the length of the address exceeds the maxlength of the database field will be skipped, while other records will be imported.

The feedback from the system is not the best, but perhaps that could be improved displaying the failing records, so the data issues can be handled.

Br.

Lars

 

 

 

 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

Thanks Lars; we use that today already, but sometimes it's nice to be able to trim and still store the data.

What about a code provider then so I can assign something like @Code(value.Trim()) on a column where value is the input value of the column? That would open up lots of other interesting possibilities. It allows us to change data without having to write (and more importantly, deploy) a table script. It also makes that code more visible than a table script is today.

Imar

 
Lars Hejgaard Sørensen Dynamicweb Employee
Lars Hejgaard Sørensen
Reply
This post has been marked as an answer

That would be useful. I have another approach, though, that can be used as an alternative. By importing the data to a staging table instead of the production table you can create a view that e.g. trims the data or joins with some other sources. Then, you import the view data to the production tables. This is extremely powerful, because you have so much fleksibilitet with SQL functions in the view, and you can even filter by destination data. Let's say you import inventory from BC for products that have been combined in DW. You can use a tablescript for that, but by importing to a temporary storage in the DW database you can join the inventory table with EcomProducs on item No./ProductNumber and select the ProductId/ProductVariantId combination to insert the data in EcomStockUnits. Even though you have inventory entries in BC for items that are not in DW, you hereby only import the relevant entries to the production table, while the others remain in the staging table.

 

Br.

Lars

Votes for this answer: 1
 
Imar Spaanjaars Dynamicweb Employee
Imar Spaanjaars
Reply

Late reply, but I really like that approach Lars. Will give that a try the next time.

Imar

 

You must be logged in to post in the forum