Expand my Community achievements bar.

SOLVED

Best Approach to upload large Assets in AEM

Avatar

Level 2

What is best approach to upload large size(in TB) of assets in AEM? Should we use content package/sling servlet/webdav or any other approach?

1 Accepted Solution

Avatar

Correct answer by
Employee Advisor

Hi,

ok, let me assume, that I have to import 2 TB of assets at once into the AEM DAM. First that means, that this importing itself is a project on its own. It requires design, coding, testing and validation; and in the end you need to have a system, which is capable to process this huge amount of data. Let's further assume, that we alredy have done that, and that we want to focus on the way how the assets are actually uploaded in the DAM.

First I would have a disucssion with the users about each and every step of the DAM asset update workflow. You really should eliminate all steps which are not required. Because a step less saves you time. And then I would create a custom mini-application inside AEM, which loads the assets from local disk (or any other location) to the DAM using the Asset API. This application will constantly monitor the system to prevent an overload, and it also writes constantly progress information in a logfile. It will allow you pause and resume the process. If it fails, it can recover gracefully and does not re-upload the same assets again.

In the end you will likely have a process, which takes some days to finish.

kind regards,
Jörg

View solution in original post

8 Replies

Avatar

Level 10

There is not a single documented way that states this way is best performance to handle 1000s of assets. However - one thing I do know is it's NOT efficient of having someone manually upload  1000s of digital assets. That is, it is not practical to upload them one at a time using the AEM web client or Curl.

Instead - you can build an app that iterates through assets and posts them automatically to AEM. See:

https://helpx.adobe.com/experience-manager/using/multiple-digital-assets.html

Avatar

Employee Advisor

Hi,

when you need to upload terabytes at a time, you have to create individual batches, and upload them in batches. Just to give the system enough room to breathe and run all the workflows. Also consider the disk space, as you might need quite a number of temporary space.

There are a number of ways to upload the files, and they depend on your approach:

  • You have the option to upload them via webdav, which is a nice way to allow endusers to upload the files.
  • You can use the REST-based SlingPostServlet, which is nice, when your source system has the capability to export assets via HTTP POST requests.
  • Or you can build some custom code, which gives you most control over all of these aspects.

So there is no single best way. And you should really care about monitoring the system :-)

kind regards,
Jörg

Avatar

Level 6

I'm also interested on the best approach for this scenario, if using multiple content packages is better than using post servlet.

Is there any documentation on this?

Avatar

Level 10

Hi there,

Thanks for reaching out to Adobe Community.

You might want to follow the below mentioned article which explains the procedure of uploading assets to DAM:

https://helpx.adobe.com/experience-manager/using/uploading-files-aem1.html

Hope this helps!

Thanks!

Avatar

Level 2

Cannot comment on TB size, but we uploaded some high amount of GB assets using WebDav. Also, we have tried earlier content bundles of 500Mb size.

Avatar

Level 2

Hi Shekhar,

Thanks for reply.

URL shared by you is an example on how to upload using slingallmethodservlet but my question is regarding best approach to follow for large asset size. Should it be loaded using content package or SlingAllMethodServlet or any other approach that is best suited for this scenario.

Avatar

Level 2

Thanks Jorg for explanation. But if you have to select one method then which one would you prefer and why?

Avatar

Correct answer by
Employee Advisor

Hi,

ok, let me assume, that I have to import 2 TB of assets at once into the AEM DAM. First that means, that this importing itself is a project on its own. It requires design, coding, testing and validation; and in the end you need to have a system, which is capable to process this huge amount of data. Let's further assume, that we alredy have done that, and that we want to focus on the way how the assets are actually uploaded in the DAM.

First I would have a disucssion with the users about each and every step of the DAM asset update workflow. You really should eliminate all steps which are not required. Because a step less saves you time. And then I would create a custom mini-application inside AEM, which loads the assets from local disk (or any other location) to the DAM using the Asset API. This application will constantly monitor the system to prevent an overload, and it also writes constantly progress information in a logfile. It will allow you pause and resume the process. If it fails, it can recover gracefully and does not re-upload the same assets again.

In the end you will likely have a process, which takes some days to finish.

kind regards,
Jörg