Move Assets from Box to AEM

[SarahKa3]

Hi all,

My team is trying to move a large amount of creative assets from Box.com storage to AEM. However we have found that manually moving them is far too time consuming and the files are often too large.

Does anyone know of a plug in or alternative option to assist with this migration of assets?

Thank you!

[DPrakashRaj]

If you are using aem cloud service then Aem asset bulk import feature or aem upload cli plugin can be used else you can look at acs-commons 

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/bulk-upload-import-assets-...

Acs common 

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/bulk-upload-import-assets-...

[rawvarun]

You can use Bulk import tool to move Assets from Box.com provided it's underlying storage is AWS or Azure.

https://experienceleague.adobe.com/docs/experience-manager-learn/cloud-service/migration/bulk-import...

[Rohan_Garg]

@SarahKa3 - We have previously migrated assets from Box to AEM Assets.

This involved writing a migration script and transferring assets in batches. There are no direct plugins but you can migrate assets to S3 or Azure Blob Storage as an intermediary and then use Bulk Import Tool if the cost of using this is not a deterrent.

In our case we were able to migrate approximately 5 TB of files from Box to AEM but had the below constraints - 

• Multi Language Handling as File names were in multiple languages including spanish and french with characters that could interrupt the migration script. The migration script was tweaked to handle these cases.
• There is a rate throttling limit implemented from BOX allowing 1000 concurrent connections to the system. This meant even with multi threaded approach we could effectively transfer 1000 assets at a time. I would advise you to use multi threading vs single thread approach to bump up transfer speed.
• With 1000 concurrent connections being limited to single user we created a pool of users and switched connections between users when the rate throttling limit was reached.

Doing all this, we were able to get a modest amount of speed and were able to transfer approx. 500 assets per hour (this can vary of course but few assets had videos with size more than 5GB+ as well) as far as I can recall. 

Hope this helps!

[kaikubad]

I will suggest that you write your own code to sync assets from box.com through api. We currently have a similar thing with bynder.com.

We have a scheduler which triggers in every 30 min. Its syncs our assets with bynder assets through api and api key.

We have separate log file where we can see which assets has been synced and which are not. 

[tusharbias · Accepted solution]

For AEM CS, 

Assets provide the following upload methods. Adobe recommends that you understand your use case and applicability of an upload option before using it. You can choose based on your need. 

Upload method                       When to use?                                          Primary Persona

Assets Console user interface	Occasional upload, ease of press and drag, finder upload. Do not use to upload many assets.	All users
Upload API	For dynamic decisions during upload.	Developer
Experience Manager desktop app	Low volume asset ingestion, but not for migration.	Administrator, Marketer
Adobe Asset Link	Useful when creatives and marketers work on assets from within the supported Creative Cloud desktop apps.	Creative, Marketer
Asset bulk ingestor	Recommended for large-scale migrations and occasional bulk ingestions. Only for supported datastores.	Administrator, Developer

For onprem - 

The bulk asset functionality is not available in the on-premise version of AEM 6.5. there are tools which you can utilize:

"CSV Asset Importer" from ACS AEM Tools: https://adobe-consulting-services.github.io/acs-aem-tools/features/csv-asset-importer/index.html

"Data Importer" from ACS AEM Commons: https://adobe-consulting-services.github.io/acs-aem-commons/features/mcp-tools/data-importer/index.h...

Refer this for more information on migrating large no. of assets: https://experienceleague.adobe.com/docs/experience-manager-65/assets/administer/assets-migration-gui...

This guide provides valuable insights into various aspects of the asset upload process, including rendition generation, smart handling of metadata, and more.

Custom Utility 

Moving a large amount of creative assets from Box.com storage to AEM manually can indeed be time-consuming and challenging, especially when dealing with large files. To streamline and expedite the migration process, you can consider the following steps:

1. Automated Migration Tools: Explore the possibility of using automated migration tools specifically designed for moving assets from Box.com to AEM. These tools can handle large-scale transfers and often provide options for scheduling and monitoring the migration process.

2. Batch Processing: Instead of transferring assets one by one, organize them into batches based on file types, folders, or other criteria. This way, you can migrate multiple assets simultaneously, reducing the time and effort needed.

3. Optimized File Formats: Check if there are opportunities to optimize the file formats without compromising the quality. For example, compressing images or videos can significantly reduce their size and speed up the migration.

4. Cloud-Based Transfer Services: Some cloud-based transfer services offer faster data transfer speeds and robust handling of large files. Consider using such services to expedite the migration.

5. Parallel Processing: If possible, use multiple threads or parallel processing to move assets simultaneously, increasing the migration speed.

6. Verify Asset Dependencies: Before the migration, ensure that you have identified and accounted for any asset dependencies in AEM. This includes ensuring that references, links, or embeds to the assets in Box.com are appropriately updated in AEM after migration.

7. Quality Checks: Plan for regular quality checks during the migration process to verify that all assets have been transferred successfully and are accessible in AEM.

8. Backup and Rollback Plan: Prioritize data integrity by creating backups of the assets before migration. Have a rollback plan in place in case of any unexpected issues during the migration.

9. Collaboration with IT/DevOps: Work closely with your IT or DevOps team to optimize network and server configurations to support the migration process efficiently.

10. Phased Migration: If the sheer volume of assets is too overwhelming, consider breaking the migration into smaller, manageable phases. This approach allows you to focus on specific subsets of assets at a time.

By combining these strategies and leveraging appropriate tools, you can significantly improve the efficiency of the asset migration process from Box.com to AEM, ensuring a smoother transition of your creative assets.