How to zip .csv file over SFTP or in Export Activity

Avatar

Avatar
Validate 10
Level 3
ketangp28686788
Level 3

Likes

23 likes

Total Posts

73 posts

Correct reply

1 solution
Top badges earned
Validate 10
Validate 1
Boost 5
Boost 3
Boost 10
View profile

Avatar
Validate 10
Level 3
ketangp28686788
Level 3

Likes

23 likes

Total Posts

73 posts

Correct reply

1 solution
Top badges earned
Validate 10
Validate 1
Boost 5
Boost 3
Boost 10
View profile
ketangp28686788
Level 3

21-07-2017

Hi,

How to Zip .csv file which is on SFTP or

is it possible to zip .csv file in export activity so as to compress it.

Thanks

Replies

Avatar

Avatar
Establish
Employee
Adhiyan
Employee

Likes

241 likes

Total Posts

343 posts

Correct reply

124 solutions
Top badges earned
Establish
Give Back 50
Give Back 25
Give Back 10
Give Back 5
View profile

Avatar
Establish
Employee
Adhiyan
Employee

Likes

241 likes

Total Posts

343 posts

Correct reply

124 solutions
Top badges earned
Establish
Give Back 50
Give Back 25
Give Back 10
Give Back 5
View profile
Adhiyan
Employee

21-07-2017

Hi Ketan,

There is an unzip utility called gunzip which can be installed in a linux server and then used to compress and decompress files in Adobe campaign.

For example you can use gzip -filename to compress the file or gzip -d filename.gz to uncompress it.

Please note that the gunzip utility needs to be present on the server and it will zip files in the .gz format and not.zip format

Regards,

Adhiyan

Avatar

Avatar
Springboard
MVP
Amit_Kumar
MVP

Likes

329 likes

Total Posts

649 posts

Correct reply

231 solutions
Top badges earned
Springboard
Ignite 5
Ignite 3
Ignite 1
Validate 10
View profile

Avatar
Springboard
MVP
Amit_Kumar
MVP

Likes

329 likes

Total Posts

649 posts

Correct reply

231 solutions
Top badges earned
Springboard
Ignite 5
Ignite 3
Ignite 1
Validate 10
View profile
Amit_Kumar
MVP

23-07-2017

Hi Ketan,

Apart from what Adhiyan suggested you can use a script to zip your file in export activity.

var zip = new ZipFile("fileName.zip");

var content = new MemoryBuffer();

content.fromString("string if you have data as string else other options to get files content");

zip.setEntry(fileName, content)

content.dispose();

zip.save("", true);

//Use file io to export if you do not want to use export activity

Regards,

Amit

Avatar

Avatar
Validate 10
Level 3
ketangp28686788
Level 3

Likes

23 likes

Total Posts

73 posts

Correct reply

1 solution
Top badges earned
Validate 10
Validate 1
Boost 5
Boost 3
Boost 10
View profile

Avatar
Validate 10
Level 3
ketangp28686788
Level 3

Likes

23 likes

Total Posts

73 posts

Correct reply

1 solution
Top badges earned
Validate 10
Validate 1
Boost 5
Boost 3
Boost 10
View profile
ketangp28686788
Level 3

25-07-2017

Thanks Adhiyan,Amit.

I have tried both ways first solution requires gunzip utility which is currently not installed over server.

I tried second solution and made some changes to it, below is the script

var zip = new ZipFile("/path/Data.zip")

var content = new MemoryBuffer();

content.load("/path/Data.csv");

zip.setEntry("/path/Data.csv",content);

content.dispose();

zip.save("", true);

In above script it is throwing file too big error. As file size of CSV is 1.2GB.

Is there any way out for this ?

Thanks

Avatar

Avatar
Validate 1
Level 4
Diego_Fontan
Level 4

Likes

29 likes

Total Posts

62 posts

Correct reply

12 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile

Avatar
Validate 1
Level 4
Diego_Fontan
Level 4

Likes

29 likes

Total Posts

62 posts

Correct reply

12 solutions
Top badges earned
Validate 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile
Diego_Fontan
Level 4

26-07-2017

You should resolve this problem considering split the file in differents chunks and the merge everything at the end.

There are many ways, libraries to implement  this:

  • Java Scanner streaming
  • Streaming Apache Commons IO
  • Java reader.lines()
  • Any library, fmk for big data works as well. 

Avatar

Avatar
Boost 3
Level 1
Yashdeep08
Level 1

Likes

3 likes

Total Posts

3 posts

Correct reply

0 solutions
Top badges earned
Boost 3
Boost 1
View profile

Avatar
Boost 3
Level 1
Yashdeep08
Level 1

Likes

3 likes

Total Posts

3 posts

Correct reply

0 solutions
Top badges earned
Boost 3
Boost 1
View profile
Yashdeep08
Level 1

30-08-2017

Hi Diego,

I am using following code and I am getting error when the records in the file is above 1 million. The error is "Function 'fromString', argument 0: cannot convert argument to a string". 

var content= new MemoryBuffer();

var x=loadFile(vars.filename);

content.fromString(x);

var zipFileName= "Name of File";

zip.setEntry(zipFileName,content);

zip.save(zipFileName);

zip.dispose();

Can you please let me know in brief how to use these libraries

Avatar

Avatar
Validate 1
MVP
Jean-Serge_Biro
MVP

Likes

353 likes

Total Posts

464 posts

Correct reply

153 solutions
Top badges earned
Validate 1
Boost 50
Boost 5
Boost 3
Boost 250
View profile

Avatar
Validate 1
MVP
Jean-Serge_Biro
MVP

Likes

353 likes

Total Posts

464 posts

Correct reply

153 solutions
Top badges earned
Validate 1
Boost 50
Boost 5
Boost 3
Boost 250
View profile
Jean-Serge_Biro
MVP

31-08-2017

Hi Ketang,

Regarding the Javascript size limitation of Amit's solution (although powerful in many cases), perhaps you can raise the MAX memory of Javascript parameter in serverConf.xml or instance conf.xml.

But, to have no limitations at all and have much more options to do all compression tasks, what I used to do for my own cases is to use 7zip tool (free tool) installed on the server, and because my Adobe Campaign servers are under Windows it is more common to use such tools rather gzip. Windows integrated tool for compressing/uncompressing is not so efficient as 7zip, most of all 7zip command lines options are powerful.

This solution is similar of gzip/gunzip suggested by Adhiyan, and supposed that you have a On Premise hosting and able to install/execute system tools.

So, in your workflow, after the file extraction activity, you can add an external task activity and set it as is:

1289891_pastedImage_0.png

In the command line of the External task, you write this:

7z a [pathname]\[filenamewithoutextension].zip [pathname]\[filenametocompress]

For instance

7z a D:\Temp\campaignXXX_<%= formatDate(new Date(), "%2D_%2M_%4Y") %>.zip  D:\Temp\campaignXXX__<%= formatDate(new Date(), "%2D_%2M_%4Y") %>.csv

Please note that since build 8795 it is not allowed by default to use External task / System script, for security reasons, except by allowing specifically in serverConf options. So see with your system administrator if he can allow to install and run 7zip, and only On Premise hosting mode is allowed (not Cloud hosting):

Adobe Campaign Release Notes (v7.0)

Adobe Campaign Release Notes (v6.1.1)

Regards.
J-Serge