Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

HOW TO EXPORT A HUGE FILE THROUGH GENERIC IMPORTS AND EXPORTS IN JOBS?

Avatar

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile
krishnakanthpat
Level 1

20-12-2016

Hi All,

We have a schema with huge amount of records and the size of the data is greater than 3 G.B.

We are trying to export this data using Jobs into our local machine but, every time when we are exporting the file we are getting the following error: 

12/20/2016 5:46:24 AM    XSV-350122 An error occurred and the process has been stopped.

Also the file is getting downloaded but only up to 2 GB.

Is there a way to export complete data through Jobs or Workflow script approach.

Please provide your suggestion or solutions.

Note: Through SFTP we were able to download 3 GB file but this could take a lot time for both download and upload.

Thanks

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Give Back 50
Level 10
florentlb
Level 10

Likes

236 likes

Total Posts

1,109 posts

Correct Reply

239 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 200
View profile

Avatar
Give Back 50
Level 10
florentlb
Level 10

Likes

236 likes

Total Posts

1,109 posts

Correct Reply

239 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 200
View profile
florentlb
Level 10

21-12-2016

Ok, indeed the generic export function is not designed to handle big files like this. Maybe you can try using Targeting workflows like this instead:

  1. Use a Query activity to get the data you need.
  2. Optionally if you want to segment the data so that the final are smaller, add a Split activity and create as much segments as you need to divide the data incoming from the Query.
  3. Add Data extraction (file) activities at the end of each outbound transition coming from the Split activity. In this activity, you can specify the name of the file, the data you need to extract, etc.

As there is no easy way to directly download files on the local machine from workflow activities, you can export it on the Campaign server so that it can be accessed in HTTP, or use a file transfer activity to make it available on a FTP server.

Hope this helps,

Florent.

Answers (5)

Answers (5)

Avatar

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile
krishnakanthpat
Level 1

23-12-2016

Hi Amit,

Yes, I'm Trying to export in CSV format. However, I feel the approach which I'm trying to do appears to be a bad practice. Working on a local file of Huge size(3GB) is not advisable. This may cause loss of data during any importing activities.

Thank you for your help and effort fleltuerto and Amit_Kumarsmiley

Avatar

Avatar
Validate 10
MVP
Amit_Kumar
MVP

Likes

329 likes

Total Posts

649 posts

Correct Reply

231 solutions
Top badges earned
Validate 10
Validate 1
Establish
Give Back 50
Give Back 5
View profile

Avatar
Validate 10
MVP
Amit_Kumar
MVP

Likes

329 likes

Total Posts

649 posts

Correct Reply

231 solutions
Top badges earned
Validate 10
Validate 1
Establish
Give Back 50
Give Back 5
View profile
Amit_Kumar
MVP

22-12-2016

What kind of file format are you using i.e. CSV? I have implemented the similar thing by breaking a file in small files 500 MB by using a JS script to write the file in zipped format to a network file system which is quite good. if this sis something you want i can help you with that.

Avatar

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile
krishnakanthpat
Level 1

21-12-2016

So far I tried to export the file directly through Data Extraction but may be due to firewall restrictions it is not getting exported.

Even I tried to connect to Putty and it didn't work.

Finally the last thing which I did is used a SFTP software (xftp) to transfer files between the servers and it worked.

But the concern here is even though it copies the file to destination, it requires client side network connectivity as it downloads a temporary file while transferring the data.

 

If at all there is any other approach apart from the above. Please let me know.  As this might help reduce a lot of effort.

Thanks smiley

Avatar

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile

Avatar
Validate 1
Level 1
krishnakanthpat
Level 1

Likes

0 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Validate 1
View profile
krishnakanthpat
Level 1

20-12-2016

fleltuerto wrote...

Hello,

Have you tried segmenting the data to export 2 or 3 smaller files to see if this issue could be caused by the size of the final file? I'll check if there is a known limit or an easy work around.

Florent.

 

Hi Florent,

I have tried segmenting the data but as there is no split option in Jobs, I'm unable to achieve the expected result. We have filtering conditions where we can filter data but cannot Segment the data (the is as per my understanding, please correct me if I'm wrong).

I have tried to setup an external SFTP account between two instances but was not successful. 

It would be helpful if you suggest any other approach where we can directly export the data to local.

Thanks

Avatar

Avatar
Give Back 50
Level 10
florentlb
Level 10

Likes

236 likes

Total Posts

1,109 posts

Correct Reply

239 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 200
View profile

Avatar
Give Back 50
Level 10
florentlb
Level 10

Likes

236 likes

Total Posts

1,109 posts

Correct Reply

239 solutions
Top badges earned
Give Back 50
Give Back 5
Give Back 3
Give Back 25
Give Back 200
View profile
florentlb
Level 10

20-12-2016

Hello,

Have you tried segmenting the data to export 2 or 3 smaller files to see if this issue could be caused by the size of the final file? I'll check if there is a known limit or an easy work around.

Florent.