How to reject a corrupted file if a column is not found?

[kotisyamala]

Dear Adobe Team,

We are unable to reject a file in a technical workflow if the specified csv file columns are not found in a file. Is there a way to reject the file to corrupted file in the file transfer or in data loading to send as a complimentary file?

Any suggestions would be helpful.

Thanks,

Koti.

[CedricRey · Accepted solution]

Hi @kotisyamala ,

There is not easy way to manage corrupted file with the fileImport activity.

If you know how many line the file could conains, one way could be to  :

 - make mandatory all colunms of the file (don't allow null values for each column instead of "Adobe Campaign Default", 5th columns of the file format wizard)

 - put the maximum errors on the max error count parameter (large number if possible), check the option to get the reject in a file that will create a new transition. If the main transition count 0 lines that could indicate the input file has a missing column. This could be easy to test (test activity, vars.recCount == 0) and you'll get all the lines into the error file (indicated by the vars.filename of the complementary transition).

I'll be interested if anyone has a more elegant solution.

Cedric

[Krishnanunni]

Hi @kotisyamala ,

I don't think you might be able to check the file header efficiently using data loading itself. I would suggest you use a shell script activity or a JS activity for this evaluation. The shell script could be as following:

fileA="path to the file downloaded"
fileB="path to file with header"

head -1 $fileA | grep "$(cat $fileB)"
if [[ $? -ne 0 ]] ;
then
echo "fileA header does not match with fileB"
exit 1;
fi

You may also use an Advanced JavaScript code activity with the following code:

var fileA="path to the file downloaded"
var fileB="path to file with header"

var file = new File(fileA);
file.open("r", File.CODEPAGE_UTF8);

var headerOfDownload = file.readln();
var originalHeader = loadFile(fileB,"utf-8")
logInfo(headerOfDownload);
logInfo(originalHeader);
if(headerOfDownload.trim() == originalHeader.trim()) {
task.postEvent(task.transitionByName('match'));
} else {
task.postEvent(task.transitionByName('nomatch'));
}

You can use the 'nomatch' transition to alert the respective operators. Hope it helps!

[Krishnanunni]

Hi @kotisyamala ,

I don't think you might be able to check the file header efficiently using data loading itself. I would suggest you use a shell script activity or a JS activity for this evaluation. The shell script could be like
fileA="path to the file downloaded"
fileB="path to file with header"

head -1 $fileA | grep "$(cat $fileB)"
if [[ $? -ne 0 ]] ;
then
echo "fileA header does not match with fileB"
exit 1;
fi

You may use an Advanced JavaScript code activity instead of shell script activity, with the following code:

var fileA="path to the file downloaded"
var fileB="path to file with header"

var file = new File(fileA);
file.open("r", File.CODEPAGE_UTF8);

var headerOfDownload = file.readln();
var originalHeader = loadFile(fileB,"utf-8")
logInfo(headerOfDownload);
logInfo(originalHeader);
if(headerOfDownload.trim() == originalHeader.trim()) {
  task.postEvent(task.transitionByName('match'));
} else {
    task.postEvent(task.transitionByName('nomatch'));
}

You can use the 'nomatch' transition to alert the respective operators. Hope it helps.

[Indra123]

Hi @Krishnanunni  ,

How can we read sample file and input file, is there any way to count no of filled column??.

Thanks,

Indra Verma

[Krishnanunni]

Hi @Indra123 ,

Using shell script, you could read a csv file from SFTP and check the number of empty columns as 

cat file.csv | head -n 2 | tail -n 1 | grep -c ',,'

As the number of columns would be static, you could calculate the non empty columns.