Highlighted

How to zip .csv file over SFTP or in Export Activity

Avatar

Avatar

ketangp28686788

Avatar

ketangp28686788

ketangp28686788

21-07-2017

Hi,

How to Zip .csv file which is on SFTP or

is it possible to zip .csv file in export activity so as to compress it.

Thanks

Replies

Highlighted

Avatar

Avatar

Adhiyan

Employee

Avatar

Adhiyan

Employee

Adhiyan
Employee

21-07-2017

Hi Ketan,

There is an unzip utility called gunzip which can be installed in a linux server and then used to compress and decompress files in Adobe campaign.

For example you can use gzip -filename to compress the file or gzip -d filename.gz to uncompress it.

Please note that the gunzip utility needs to be present on the server and it will zip files in the .gz format and not.zip format

Regards,

Adhiyan

Highlighted

Avatar

Avatar

Amit_Kumar

MVP

Avatar

Amit_Kumar

MVP

Amit_Kumar
MVP

23-07-2017

Hi Ketan,

Apart from what Adhiyan suggested you can use a script to zip your file in export activity.

var zip = new ZipFile("fileName.zip");

var content = new MemoryBuffer();

content.fromString("string if you have data as string else other options to get files content");

zip.setEntry(fileName, content)

content.dispose();

zip.save("", true);

//Use file io to export if you do not want to use export activity

Regards,

Amit

Highlighted

Avatar

Avatar

ketangp28686788

Avatar

ketangp28686788

ketangp28686788

25-07-2017

Thanks Adhiyan,Amit.

I have tried both ways first solution requires gunzip utility which is currently not installed over server.

I tried second solution and made some changes to it, below is the script

var zip = new ZipFile("/path/Data.zip")

var content = new MemoryBuffer();

content.load("/path/Data.csv");

zip.setEntry("/path/Data.csv",content);

content.dispose();

zip.save("", true);

In above script it is throwing file too big error. As file size of CSV is 1.2GB.

Is there any way out for this ?

Thanks

Highlighted

Avatar

Avatar

Diego_Fontan

Avatar

Diego_Fontan

Diego_Fontan

26-07-2017

You should resolve this problem considering split the file in differents chunks and the merge everything at the end.

There are many ways, libraries to implement  this:

  • Java Scanner streaming
  • Streaming Apache Commons IO
  • Java reader.lines()
  • Any library, fmk for big data works as well. 
Highlighted

Avatar

Avatar

Yashdeep08

Avatar

Yashdeep08

Yashdeep08

30-08-2017

Hi Diego,

I am using following code and I am getting error when the records in the file is above 1 million. The error is "Function 'fromString', argument 0: cannot convert argument to a string". 

var content= new MemoryBuffer();

var x=loadFile(vars.filename);

content.fromString(x);

var zipFileName= "Name of File";

zip.setEntry(zipFileName,content);

zip.save(zipFileName);

zip.dispose();

Can you please let me know in brief how to use these libraries

Highlighted

Avatar

Avatar

Jean-Serge_Biro

MVP

Avatar

Jean-Serge_Biro

MVP

Jean-Serge_Biro
MVP

31-08-2017

Hi Ketang,

Regarding the Javascript size limitation of Amit's solution (although powerful in many cases), perhaps you can raise the MAX memory of Javascript parameter in serverConf.xml or instance conf.xml.

But, to have no limitations at all and have much more options to do all compression tasks, what I used to do for my own cases is to use 7zip tool (free tool) installed on the server, and because my Adobe Campaign servers are under Windows it is more common to use such tools rather gzip. Windows integrated tool for compressing/uncompressing is not so efficient as 7zip, most of all 7zip command lines options are powerful.

This solution is similar of gzip/gunzip suggested by Adhiyan, and supposed that you have a On Premise hosting and able to install/execute system tools.

So, in your workflow, after the file extraction activity, you can add an external task activity and set it as is:

1289891_pastedImage_0.png

In the command line of the External task, you write this:

7z a [pathname]\[filenamewithoutextension].zip [pathname]\[filenametocompress]

For instance

7z a D:\Temp\campaignXXX_<%= formatDate(new Date(), "%2D_%2M_%4Y") %>.zip  D:\Temp\campaignXXX__<%= formatDate(new Date(), "%2D_%2M_%4Y") %>.csv

Please note that since build 8795 it is not allowed by default to use External task / System script, for security reasons, except by allowing specifically in serverConf options. So see with your system administrator if he can allow to install and run 7zip, and only On Premise hosting mode is allowed (not Cloud hosting):

Adobe Campaign Release Notes (v7.0)

Adobe Campaign Release Notes (v6.1.1)

Regards.
J-Serge