Expand my Community achievements bar.

Enhance your AEM Assets & Boost Your Development: [AEM Gems | June 19, 2024] Improving the Developer Experience with New APIs and Events
SOLVED

Migrating 15TB Asset to AEM cloud.

Avatar

Level 1

Hi All,

 

We have a large amount of assets (around 15 TB) to migrate to the AEM cloud from S3. What's the best approach to migrating the assets, and how long is required to migrate?

 

How many assets can be migrate per hour using the asset bulk importer?

 

Thanks,

Udaya

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

Hi @udaya_kumar1 , If you are concerned about the import time duration you may have migration batches as per your use case, lets say 5TB and repeat the process.

When using bulk import tool to ingest assets on AEMaaCloudService assets undergo processing via asset compute services. You should be able to import assets at the rate of ~20000 assets/hour but it depends upon your binary size as well.

The exact progress percentage can be seen on the running import job status, using that you may get a rough estimate of completion.

References: 
Q&A: Adobe Developers Live 2021, Session | Asset Bulk I... - Adobe Experience League Community - 394474
Documentation: Using Bulk Import with AEM Assets | Adobe Experience Manager

View solution in original post

4 Replies

Avatar

Community Advisor

@udaya_kumar1 if you have your source assets in s3, then pleae engage Adobe Customer support and their engineering team will help copy assets from customer s3 to Adobe s3, ofcourse given a one time read access to Adobe s3 user. This is fast and efficient way.. post that you can use acs metadata bulk importer to import metadata. 

 

Taking into assumptions that the folder structure is not changed between source and AEM..

Avatar

Correct answer by
Community Advisor

Hi @udaya_kumar1 , If you are concerned about the import time duration you may have migration batches as per your use case, lets say 5TB and repeat the process.

When using bulk import tool to ingest assets on AEMaaCloudService assets undergo processing via asset compute services. You should be able to import assets at the rate of ~20000 assets/hour but it depends upon your binary size as well.

The exact progress percentage can be seen on the running import job status, using that you may get a rough estimate of completion.

References: 
Q&A: Adobe Developers Live 2021, Session | Asset Bulk I... - Adobe Experience League Community - 394474
Documentation: Using Bulk Import with AEM Assets | Adobe Experience Manager

Avatar

Community Advisor

@udaya_kumar1 - Asset Bulk Importer is the best way to go for S3 data sources.

Regarding performance query, in Adobe's Test, they were able to import 2000 assets per hour.

In one of their customer implementations, they were able to import ~ 20,000 assets per hour for total ~ 23 TB of real customer data.

Source - Adobe Developers Live | Asset Bulk Ingestion
URL - https://www.youtube.com/watch?v=pY6A9hzKN2M (Timestamp - 08:00 min)

 

Rohan_Garg_0-1684821818008.png

 

Hope this helps!