How to zip .csv file over SFTP or in Export Activity | Community
Skip to main content
Level 4
July 21, 2017

How to zip .csv file over SFTP or in Export Activity

  • July 21, 2017
  • 4 replies
  • 7905 views

Hi,

How to Zip .csv file which is on SFTP or

is it possible to zip .csv file in export activity so as to compress it.

Thanks

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

4 replies

Adhiyan
Adobe Employee
Adobe Employee
July 21, 2017

Hi Ketan,

There is an unzip utility called gunzip which can be installed in a linux server and then used to compress and decompress files in Adobe campaign.

For example you can use gzip -filename to compress the file or gzip -d filename.gz to uncompress it.

Please note that the gunzip utility needs to be present on the server and it will zip files in the .gz format and not.zip format

Regards,

Adhiyan

Amit_Kumar
Level 10
July 24, 2017

Hi Ketan,

Apart from what Adhiyan suggested you can use a script to zip your file in export activity.

var zip = new ZipFile("fileName.zip");

var content = new MemoryBuffer();

content.fromString("string if you have data as string else other options to get files content");

zip.setEntry(fileName, content)

content.dispose();

zip.save("", true);

//Use file io to export if you do not want to use export activity

Regards,

Amit

Level 4
July 25, 2017

Thanks Adhiyan,Amit.

I have tried both ways first solution requires gunzip utility which is currently not installed over server.

I tried second solution and made some changes to it, below is the script

var zip = new ZipFile("/path/Data.zip")

var content = new MemoryBuffer();

content.load("/path/Data.csv");

zip.setEntry("/path/Data.csv",content);

content.dispose();

zip.save("", true);

In above script it is throwing file too big error. As file size of CSV is 1.2GB.

Is there any way out for this ?

Thanks

Diego_Fontan
Level 4
July 27, 2017

You should resolve this problem considering split the file in differents chunks and the merge everything at the end.

There are many ways, libraries to implement  this:

  • Java Scanner streaming
  • Streaming Apache Commons IO
  • Java reader.lines()
  • Any library, fmk for big data works as well. 
Yashdeep08
August 30, 2017

Hi Diego,

I am using following code and I am getting error when the records in the file is above 1 million. The error is "Function 'fromString', argument 0: cannot convert argument to a string". 

var content= new MemoryBuffer();

var x=loadFile(vars.filename);

content.fromString(x);

var zipFileName= "Name of File";

zip.setEntry(zipFileName,content);

zip.save(zipFileName);

zip.dispose();

Can you please let me know in brief how to use these libraries

Jean-Serge_Biro
Level 10
August 31, 2017

Hi Ketang,

Regarding the Javascript size limitation of Amit's solution (although powerful in many cases), perhaps you can raise the MAX memory of Javascript parameter in serverConf.xml or instance conf.xml.

But, to have no limitations at all and have much more options to do all compression tasks, what I used to do for my own cases is to use 7zip tool (free tool) installed on the server, and because my Adobe Campaign servers are under Windows it is more common to use such tools rather gzip. Windows integrated tool for compressing/uncompressing is not so efficient as 7zip, most of all 7zip command lines options are powerful.

This solution is similar of gzip/gunzip suggested by Adhiyan, and supposed that you have a On Premise hosting and able to install/execute system tools.

So, in your workflow, after the file extraction activity, you can add an external task activity and set it as is:

In the command line of the External task, you write this:

7z a [pathname]\[filenamewithoutextension].zip [pathname]\[filenametocompress]

For instance

7z a D:\Temp\campaignXXX_<%= formatDate(new Date(), "%2D_%2M_%4Y") %>.zip  D:\Temp\campaignXXX__<%= formatDate(new Date(), "%2D_%2M_%4Y") %>.csv

Please note that since build 8795 it is not allowed by default to use External task / System script, for security reasons, except by allowing specifically in serverConf options. So see with your system administrator if he can allow to install and run 7zip, and only On Premise hosting mode is allowed (not Cloud hosting):

Adobe Campaign Release Notes (v7.0)

Adobe Campaign Release Notes (v6.1.1)

Regards.
J-Serge