Migrating 15TB Asset to AEM cloud. | Community
Skip to main content
U
udaya_kumar1
May 23, 2023
Solved

Migrating 15TB Asset to AEM cloud.

  • May 23, 2023
  • 3 replies
  • 1050 views

Hi All,

 

We have a large amount of assets (around 15 TB) to migrate to the AEM cloud from S3. What's the best approach to migrating the assets, and how long is required to migrate?

 

How many assets can be migrate per hour using the asset bulk importer?

 

Thanks,

Udaya

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by ShubhanshuSi2

Hi @udaya_kumar1 , If you are concerned about the import time duration you may have migration batches as per your use case, lets say 5TB and repeat the process.

When using bulk import tool to ingest assets on AEMaaCloudService assets undergo processing via asset compute services. You should be able to import assets at the rate of ~20000 assets/hour but it depends upon your binary size as well.

The exact progress percentage can be seen on the running import job status, using that you may get a rough estimate of completion.

References: 
Q&A: Adobe Developers Live 2021, Session | Asset Bulk I... - Adobe Experience League Community - 394474
Documentation: Using Bulk Import with AEM Assets | Adobe Experience Manager

3 replies

Shashi_Mulugu
Community Advisor
Shashi_MuluguCommunity Advisor
Community Advisor
May 23, 2023

@udaya_kumar1 if you have your source assets in s3, then pleae engage Adobe Customer support and their engineering team will help copy assets from customer s3 to Adobe s3, ofcourse given a one time read access to Adobe s3 user. This is fast and efficient way.. post that you can use acs metadata bulk importer to import metadata. 

 

Taking into assumptions that the folder structure is not changed between source and AEM..

    S
    Community Advisor
    ShubhanshuSi2Community AdvisorAccepted solution
    Community Advisor
    May 23, 2023

    Hi @udaya_kumar1 , If you are concerned about the import time duration you may have migration batches as per your use case, lets say 5TB and repeat the process.

    When using bulk import tool to ingest assets on AEMaaCloudService assets undergo processing via asset compute services. You should be able to import assets at the rate of ~20000 assets/hour but it depends upon your binary size as well.

    The exact progress percentage can be seen on the running import job status, using that you may get a rough estimate of completion.

    References: 
    Q&A: Adobe Developers Live 2021, Session | Asset Bulk I... - Adobe Experience League Community - 394474
    Documentation: Using Bulk Import with AEM Assets | Adobe Experience Manager

      U
      udaya_kumar1Author
      May 23, 2023

      Got it. Thanks Shubhanshu

       

      Rohan_Garg
      Community Advisor
      Rohan_GargCommunity Advisor
      Community Advisor
      May 23, 2023

      @udaya_kumar1 - Asset Bulk Importer is the best way to go for S3 data sources.

      Regarding performance query, in Adobe's Test, they were able to import 2000 assets per hour.

      In one of their customer implementations, they were able to import ~ 20,000 assets per hour for total ~ 23 TB of real customer data.

      Source - Adobe Developers Live | Asset Bulk Ingestion
      URL - https://www.youtube.com/watch?v=pY6A9hzKN2M (Timestamp - 08:00 min)

       

       

      Hope this helps!

       

        Terms & ConditionsAccessibility statement

        Loading footer...