Avoid errors when importing CSV files

Tomas Gomez

Posted on 10/09/2021 11:53:29

Hi,

We get the data from our customer in CSV files that we import in DW by integration tasks. There are many files and they contain lot of data, so errors are frequent.

For instance, we notice that a task error is thrown if there are not valid CSV files at the same foder (even if they are not used by the integration). As well, an error is thrown if there are empty columns inserted within the data (for visual separation of data blocks).

In the Source setting of the task, there is an option to "Skip troublesome rows" but it only avoid some kind of errors.

Is the any othe option to avoid these errors when integrating CSV files? Which are your suggestion?

Regards,
Tomas

Replies

Tomas Gomez

Posted on 30/09/2021 11:14:25

Hi again,

Beside the aformentioned issues, there are other human-mistakes desiderable to check. For instance, avoid non-printable characters (tab, return), custom format of cell contents, etc.

At your exprience, how would you automatically detect these CSV errors? I can think of several alternatives

In the Excel file, create a macro to check the format within the CSV file
In DW content, create an private page to read the file and display the errors
In DW tasks, create a task to do it. Similar as previous, maybe more porwerful but need compilation

Regards,
Tomas

Adrian Ursu

Posted on 30/09/2021 12:30:26

This post has been marked as an answer

Hi Tomas,

I agree with what you are saying. We have experienced similar issues before and we had to manually check every file before we could identify all the troublesome data.

It would be good to have this type of check built-in in the CSV provider (and probably also in the Excel provider too).

Maybe it is worths adding a feature request?

Adrian

Votes for this answer: 1

Tomas Gomez

Posted on 04/10/2021 11:38:37

Thanks Adrian for your suggestion.

I posted a new feture request at Avoid errors when importing Excel/CSV files

Regards,
Tomas

You must be logged in to post in the forum

Developer forum

Avoid errors when importing CSV files

Replies