Migrate Assets form Onedrive, Sharepoint to AEM DAM on AEM as Cloud | Community
Skip to main content
Asutosh_Jena_
Community Advisor
Asutosh_Jena_Community Advisor
Community Advisor
August 8, 2022
Solved

Migrate Assets form Onedrive, Sharepoint to AEM DAM on AEM as Cloud

  • August 8, 2022
  • 3 replies
  • 4626 views

We are planning to migrate a huge amount of assets available at different locations, such as Onedrive, Sharepoint, Dropbox and few on S3.


For S3 we can use the Bulk Asset upload mechanism and we can pull the binaries. But for all other modes, what will be the best way to pull the assets into AEM?

 

Assets size are almost around more than 200 TB combining all the sources.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by achennapragada

Hi @asutosh_jena_ 

I would prefer moving these assets which are in multiple source systems to a single system(preferably cloud storage like S3/Azure) process it and then use bulk importer tools to pull this to AEM.

 

Thanks,

Anil.

3 replies

A
achennapragadaAccepted solution
Level 2
August 8, 2022

Hi @asutosh_jena_ 

I would prefer moving these assets which are in multiple source systems to a single system(preferably cloud storage like S3/Azure) process it and then use bulk importer tools to pull this to AEM.

 

Thanks,

Anil.

    Asutosh_Jena_
    Community Advisor
    Asutosh_Jena_Community AdvisorAuthor
    Community Advisor
    August 8, 2022

    Hi @achennapragada 

     

    Yes we have already thought about this but it's a tedious process to gather all the data from different locations and move it into S3 in a organized manner and then process using bulk uploaded. Also it will have additional cost involved.

      A
      achennapragada
      Level 2
      August 8, 2022

      I believe apart from the above these are the only other options

      Though the first process initially takes time and involves cost, considering the size of the total asstes leveraging bulk importer tools to import assets in batches might be better option.

       

      Thanks,

      Anil. 

        kautuk_sahni
        Community Manager
        kautuk_sahniCommunity Manager
        Community Manager
        August 9, 2022

        @asutosh_jena_ I have asked the internal AEM Assets SME to get back to you here. 

        Kautuk Sahni
          kautuk_sahni
          Community Manager
          kautuk_sahniCommunity Manager
          Community Manager
          August 11, 2022

          @asutosh_jena_ 

           

          Suggestion 1:->

          Be sure to consider file names and paths. Avoid more than 1000 nodes per parent (see DAM performance guidelines). The file path and name is the unique reference, they may want to scan for duplicate images first.
          For large migrations, Adobe consultants usually use scripts and, perform in batches, and throttle to allow AEM to process the assets.

           

          Suggestion 2:->

          And please do have a look at: https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/assets/manage/add-assets.html?lang=en#bulk-upload 

          and 

          https://experienceleague.adobe.com/docs/experience-manager-learn/cloud-service/migration/bulk-import.html?lang=en 

          From what I know, there's no 'direct' method to connect to OneDrive/Sharepoint - but I suppose one of these options could potentially be used.also, oftentimes it is a number of assets rather than their cumulative size that can help pick the right tool. For example, if a number of assets is small, but each asset itself is large-sized, Desktop App running on a fast enough network might be a better fit than the bulk importer.

           

          PS: I'll keep you posted with internal answers!!

          Kautuk Sahni
            kautuk_sahni
            Community Manager
            kautuk_sahniCommunity Manager
            Community Manager
            August 12, 2022

            Suggestion 3:->

            The first option listed (AEM-Upload) is a sample app for uploading into AEM using Node — and if they’re cost-conscious then it might be worthwhile to investigate creating a custom program that reads their files from OneDrive and/or Sharepoint (realistically it’s probably the same API for both) and use the sample code to upload the assets.  The S3 and Azure connectors are also good general-purpose options but I don’t think that storage cost is a heavy factor if the storage buckets are used quickly.  Disk space is charged as an average per day, so if they upload a chunk of files, import them, and then purge them from the blob storage I think they can minimize cost very effectively.  That might be a cheaper option than developing a custom migration approach entirely to avoid blob storage options already available.

             

            Suggestion 4:->

            The bulk Import tool is the right approach for the content in S3 buckets. For others, it may be better to leverage the AEM Desktop App, or move all the other content to S3 and then use a bulk import tool for all of it

             

            Suggestion 5:->

            The best would be to consolidate all in the S3 bucket and import from there with our bulk importer.

             

            Kautuk Sahni
              S
              Adobe Employee
              spotade-adobeAdobe Employee
              Adobe Employee
              August 12, 2022

              I Agree on the additional cost and effort required but the cleanest and recommended approach would be to move all the assets into S3, perform cleanup there and then use Bulk Import to move assets into AEMaaCS. This will require least amount of development effort.

              Also Bulk Importer ingest assets as-is from S3, So i would recommend having it in some folder structure. Dont import everything into AEM and then try to move/restructure that would be much more expensive and slower operation.

              Also consider is your using Dynamic Media (DM). If so having assets imported in correct/final location makes most sense. With AEMaaCS there is no need to turn off workflow (like previously recommended with AEM 6.5 or AMS). Asset Compute should handle all the operations async.

              Hope this helps.

              Thanks

              Sammit

                Terms & ConditionsAccessibility statement

                Loading footer...