Migrating 15TB Asset to AEM cloud.

[udaya_kumar1]

Hi All,

We have a large amount of assets (around 15 TB) to migrate to the AEM cloud from S3. What's the best approach to migrating the assets, and how long is required to migrate?

How many assets can be migrate per hour using the asset bulk importer?

Thanks,

Udaya

[Shashi_Mulugu]

@udaya_kumar1 if you have your source assets in s3, then pleae engage Adobe Customer support and their engineering team will help copy assets from customer s3 to Adobe s3, ofcourse given a one time read access to Adobe s3 user. This is fast and efficient way.. post that you can use acs metadata bulk importer to import metadata. 

Taking into assumptions that the folder structure is not changed between source and AEM..

[shubhanshu_singh · Accepted solution]

Hi @udaya_kumar1 , If you are concerned about the import time duration you may have migration batches as per your use case, lets say 5TB and repeat the process.

When using bulk import tool to ingest assets on AEMaaCloudService assets undergo processing via asset compute services. You should be able to import assets at the rate of ~20000 assets/hour but it depends upon your binary size as well.

The exact progress percentage can be seen on the running import job status, using that you may get a rough estimate of completion.

References: 
Q&A: Adobe Developers Live 2021, Session | Asset Bulk I... - Adobe Experience League Community - 394474
Documentation: Using Bulk Import with AEM Assets | Adobe Experience Manager

[udaya_kumar1]

Got it. Thanks Shubhanshu

[Rohan_Garg]

@udaya_kumar1 - Asset Bulk Importer is the best way to go for S3 data sources.

Regarding performance query, in Adobe's Test, they were able to import 2000 assets per hour.

In one of their customer implementations, they were able to import ~ 20,000 assets per hour for total ~ 23 TB of real customer data.

Source - Adobe Developers Live | Asset Bulk Ingestion
URL - https://www.youtube.com/watch?v=pY6A9hzKN2M (Timestamp - 08:00 min)

Hope this helps!