Migration of 10TB of assets into AEMaaCS

[ayyappavenkatesh]

I'm looking for suggestions on how to handle the 10TB of assets from a vendor base to AEMaaCS. As the data is huge and confidential, no feasibility to involve any 3rd parties like tools/http calls etc. It should be cloud(like AWS/AZURE/GCP etc) to cloud(AEMaaCS). Please share inputs if anyone handled such use-cases or in and around the same, would be very much appreciated.

[Saravanan_Dharmaraj]

@ayyappavenkatesh  Please check the below 2 posts and links inside of those.

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/migrating-15tb-asset-to-ae...

https://medium.com/@bsaravanaprakash/how-we-migrated-a-million-assets-into-aem-cloud-service-dam-ef7...

[ayyappavenkatesh]

Thanks for sharing the examples.

The quick POC worked for me to migrate the assets from S3 to AEMaaCS. But the real challenge which was identified as on, how to deal with the metadata. In simple terminology, obviously the metadata_client.csv file column names doesn't match with the metadata_aemaacs.csv(which was created using Export Metadata option). So is there any Adobe recommended approach on how to handle the metadata(like mapping/insertion/manipulation etc) of TB's of assets during the Bulk Import by any chance or is it something to deal with some custom scripts?

Another question, Is it possible to map the metadata of the assets with the correct one's inside AWS before we go with Bulk Import or else first migrate the assets from AWS S3 to AEMaaCS and then deal with the respective metadata, if yes then is there any approach recommended by any chance?

Thanks in advance!

[Avinash_Gupta_]

Hi @ayyappavenkatesh 

In my past project, I migrated nearly 60TB of assets in AEMasCS. The approach was to first focus on migrating asset binaries and later metadata.

1. For the assets binary migration, the best and most efficient approach is to first upload all the assets from the legacy application to the S3 bucket.

2. Once all the assets are uploaded to S3, migrate these assets from S3 to AEMasCS using the bulk import service. Configure the S3 account details and the AWS source path and AEMasCS dam as destination folder path.

https://experienceleague.adobe.com/en/docs/experience-manager-learn/cloud-service/migration/moving-t... 

3. Handle the import of metadata separately. Create the metadata sheet by providing the asset path with all the metadata namespaces and finally import this metadata sheet in AEM

https://experienceleague.adobe.com/en/docs/experience-manager-learn/assets/metadata/metadata-import-... 

Hoping all these steps will help you in importing these assets in bulk without any glitch.

Thanks,

Avinash

[Vishal_Anand]

Please follow below steps:

1. Upload assets to an AWS S3 bucket. Alternative option being (Azure, GCP)
2. Use AWS Direct Connect to establish a private, high-speed connection to AEMaaCS.
3. Configure AEMaaCS to ingest assets directly from the S3 bucket using Adobe’s APIs or cloud-native integration tools.

[kautuk_sahni]

@ayyappavenkatesh just checking in! Were you able to get this resolved? If one of the replies above helped—whether it completely solved the issue or simply pointed you in the right direction—marking it as accepted can make it much easier for others with the same question to find a solution. And if you found a different way to fix it, sharing your approach would be a great contribution to the community. Your follow-up not only helps close the loop but also ensures others benefit from your experience. Thanks so much for being part of the conversation!