Level 2

Solved

Best Approach to upload large Assets in AEM

Forum|Forum|10 years ago
October 16, 2015
8 replies
10149 views

What is best approach to upload large size(in TB) of assets in AEM? Should we use content package/sling servlet/webdav or any other approach?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Best answer by joerghoh

Hi,

ok, let me assume, that I have to import 2 TB of assets at once into the AEM DAM. First that means, that this importing itself is a project on its own. It requires design, coding, testing and validation; and in the end you need to have a system, which is capable to process this huge amount of data. Let's further assume, that we alredy have done that, and that we want to focus on the way how the assets are actually uploaded in the DAM.

First I would have a disucssion with the users about each and every step of the DAM asset update workflow. You really should eliminate all steps which are not required. Because a step less saves you time. And then I would create a custom mini-application inside AEM, which loads the assets from local disk (or any other location) to the DAM using the Asset API. This application will constantly monitor the system to prevent an overload, and it also writes constantly progress information in a logfile. It will allow you pause and resume the process. If it fails, it can recover gracefully and does not re-upload the same assets again.

In the end you will likely have a process, which takes some days to finish.

kind regards,
Jörg

smacdonald2008

Level 10

There is not a single documented way that states this way is best performance to handle 1000s of assets. However - one thing I do know is it's NOT efficient of having someone manually upload 1000s of digital assets. That is, it is not practical to upload them one at a time using the AEM web client or Curl.

Instead - you can build an app that iterates through assets and posts them automatically to AEM. See:

https://helpx.adobe.com/experience-manager/using/multiple-digital-assets.html

joerghoh

Adobe Employee

Hi,

when you need to upload terabytes at a time, you have to create individual batches, and upload them in batches. Just to give the system enough room to breathe and run all the workflows. Also consider the disk space, as you might need quite a number of temporary space.

There are a number of ways to upload the files, and they depend on your approach:

You have the option to upload them via webdav, which is a nice way to allow endusers to upload the files.
You can use the REST-based SlingPostServlet, which is nice, when your source system has the capability to export assets via HTTP POST requests.
Or you can build some custom code, which gives you most control over all of these aspects.

So there is no single best way. And you should really care about monitoring the system :-)

kind regards,
Jörg

francisco_ribei

Level 6

I'm also interested on the best approach for this scenario, if using multiple content packages is better than using post servlet.

Is there any documentation on this?

WhoaShekhar

Level 10

Hi there,

Thanks for reaching out to Adobe Community.

You might want to follow the below mentioned article which explains the procedure of uploading assets to DAM:

https://helpx.adobe.com/experience-manager/using/uploading-files-aem1.html

Hope this helps!

Thanks!

M

Morpheus_1729

Level 2

Cannot comment on TB size, but we uploaded some high amount of GB assets using WebDav. Also, we have tried earlier content bundles of 500Mb size.

rajeev6969Author

Level 2

Hi Shekhar,

Thanks for reply.

URL shared by you is an example on how to upload using slingallmethodservlet but my question is regarding best approach to follow for large asset size. Should it be loaded using content package or SlingAllMethodServlet or any other approach that is best suited for this scenario.

rajeev6969Author

Level 2

Thanks Jorg for explanation. But if you have to select one method then which one would you prefer and why?

joerghoh

Accepted solution

Adobe Employee

Hi,

ok, let me assume, that I have to import 2 TB of assets at once into the AEM DAM. First that means, that this importing itself is a project on its own. It requires design, coding, testing and validation; and in the end you need to have a system, which is capable to process this huge amount of data. Let's further assume, that we alredy have done that, and that we want to focus on the way how the assets are actually uploaded in the DAM.

First I would have a disucssion with the users about each and every step of the DAM asset update workflow. You really should eliminate all steps which are not required. Because a step less saves you time. And then I would create a custom mini-application inside AEM, which loads the assets from local disk (or any other location) to the DAM using the Asset API. This application will constantly monitor the system to prevent an overload, and it also writes constantly progress information in a logfile. It will allow you pause and resume the process. If it fails, it can recover gracefully and does not re-upload the same assets again.

In the end you will likely have a process, which takes some days to finish.

kind regards,
Jörg

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded