Moving large amounts of content between AEM instances

[elizabethgboss]

Hi - I have about 100 GB of content (DAM and Pages) that I need to move between Production and Non-Production (and vice versa). I am using the 'vlt' process and it takes days (~3). Is there any way to speed this up? I am using AEM 5.6.1.

Thanks,

Elizabeth

[Opkar_Gill]

Hi Elizabeth,

looking at the test numbers form this article[0], your numbers do seem very slow. Although the type of content can slow down the process. 

However, other factors to look into are the specified and actual network speed between the two servers.

Are the servers on site, in the cloud, the same or different data centres, same region?

Regards,

Opkar

[0]http://cq-ops.tumblr.com/post/43179911102/how-to-efficiently-copy-large-amounts-of-content

[leeasling · Accepted solution]

I used ReCap to transfer about 400GB of data between numerous servers and it took less than a day.

http://adamcin.net/net.adamcin.recap/

I will also add that I needed to turn off the workflows for updating renditions, otherwise the system would get very slow.  Maybe that's the cause of your issues?

[elizabethgboss]

The servers are in the AWS cloud, managed by Adobe. They are in the same region. 

[elizabethgboss]

I turned off the workflows. I can try this out. Thanks!

[Opkar_Gill]

Elizabeth Gallo wrote...

The servers are in the AWS cloud, managed by Adobe. They are in the same region. 

Have you discussed the performance with your AMS cloud engineer?

[elizabethgboss]

Yes, of course 🙂

I found a tool that Time Warner used to speed it up but have not had enough time to make it work:

https://github.com/TWCable/grabbit

http://blogs.adobe.com/experiencedelivers/experience-management/time-warner-cable-shares-open-source...

Are you familiar with this tool? This seems to be a common industry problem. I don't think network latency is the issue.

[Abhinav_Sap]

Grabbit should be the quickest. Is there any specific issue that you are facing setting it up ?

[elizabethgboss]

The last time I tried was about 1 month ago....I followed the instructions but could not get the code to run on the source machine. I don't remember the error I was getting. I only spent about an hour on it before I needed to move onto other things.