Migrate Assets form Onedrive, Sharepoint to AEM DAM on AEM as Cloud

[Asutosh_Jena_]

We are planning to migrate a huge amount of assets available at different locations, such as Onedrive, Sharepoint, Dropbox and few on S3.

For S3 we can use the Bulk Asset upload mechanism and we can pull the binaries. But for all other modes, what will be the best way to pull the assets into AEM?

Assets size are almost around more than 200 TB combining all the sources.

[achennapragada · Accepted solution]

Hi @Asutosh_Jena_ 

I would prefer moving these assets which are in multiple source systems to a single system(preferably cloud storage like S3/Azure) process it and then use bulk importer tools to pull this to AEM.

Thanks,

Anil.

[Asutosh_Jena_]

Hi @achennapragada 

Yes we have already thought about this but it's a tedious process to gather all the data from different locations and move it into S3 in a organized manner and then process using bulk uploaded. Also it will have additional cost involved.

[achennapragada]

I believe apart from the above these are the only other options

• https://github.com/adobe/aem-upload
• https://medium.com/adobetech/introduction-to-aem-as-a-cloud-service-asset-upload-http-api-fc560ffd72...

Though the first process initially takes time and involves cost, considering the size of the total asstes leveraging bulk importer tools to import assets in batches might be better option.

Thanks,

Anil. 

[Mayank_Gandhi]

@achennapragada That will be double the cost to the system but doable.

[kautuk_sahni]

@Asutosh_Jena_ I have asked the internal AEM Assets SME to get back to you here. 

[kautuk_sahni]

@Asutosh_Jena_ 

Suggestion 1:->

Be sure to consider file names and paths. Avoid more than 1000 nodes per parent (see DAM performance guidelines). The file path and name is the unique reference, they may want to scan for duplicate images first.
For large migrations, Adobe consultants usually use scripts and, perform in batches, and throttle to allow AEM to process the assets.

Suggestion 2:->

And please do have a look at: https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/assets/manage/add-a... 

and 

https://experienceleague.adobe.com/docs/experience-manager-learn/cloud-service/migration/bulk-import... 

From what I know, there's no 'direct' method to connect to OneDrive/Sharepoint - but I suppose one of these options could potentially be used.also, oftentimes it is a number of assets rather than their cumulative size that can help pick the right tool. For example, if a number of assets is small, but each asset itself is large-sized, Desktop App running on a fast enough network might be a better fit than the bulk importer.

PS: I'll keep you posted with internal answers!!

[kautuk_sahni]

Suggestion 3:->

The first option listed (AEM-Upload) is a sample app for uploading into AEM using Node — and if they’re cost-conscious then it might be worthwhile to investigate creating a custom program that reads their files from OneDrive and/or Sharepoint (realistically it’s probably the same API for both) and use the sample code to upload the assets.  The S3 and Azure connectors are also good general-purpose options but I don’t think that storage cost is a heavy factor if the storage buckets are used quickly.  Disk space is charged as an average per day, so if they upload a chunk of files, import them, and then purge them from the blob storage I think they can minimize cost very effectively.  That might be a cheaper option than developing a custom migration approach entirely to avoid blob storage options already available.

Suggestion 4:->

The bulk Import tool is the right approach for the content in S3 buckets. For others, it may be better to leverage the AEM Desktop App, or move all the other content to S3 and then use a bulk import tool for all of it

Suggestion 5:->

The best would be to consolidate all in the S3 bucket and import from there with our bulk importer.

[spotade]

I Agree on the additional cost and effort required but the cleanest and recommended approach would be to move all the assets into S3, perform cleanup there and then use Bulk Import to move assets into AEMaaCS. This will require least amount of development effort.

Also Bulk Importer ingest assets as-is from S3, So i would recommend having it in some folder structure. Dont import everything into AEM and then try to move/restructure that would be much more expensive and slower operation.

Also consider is your using Dynamic Media (DM). If so having assets imported in correct/final location makes most sense. With AEMaaCS there is no need to turn off workflow (like previously recommended with AEM 6.5 or AMS). Asset Compute should handle all the operations async.

Hope this helps.

Thanks

Sammit