Expand my Community achievements bar.

Dive into Adobe Summit 2024! Explore curated list of AEM sessions & labs, register, connect with experts, ask questions, engage, and share insights. Don't miss the excitement.
SOLVED

DAM Offloading often choking up with high volume of uploads

Avatar

Level 5

I have a issue where my IT team has to keep rebuilding offloading agents. The problem occurs when we upload high amounts of 

XML content roughly about 50 000 - 100 000 files. Only OOTB workflows are being run on it (DAM Update Asset). AEM processes couple thousands of files extremely fast

then it eventually gets slower and slower where it just comes to a halt. What can be the issue here? Adobe Tech guy said it should be able handle small files like that in large bulks.

I'm uploading it through a desktop client which sends the files to an OSGi Servlet. 

 

Thanks all!

1 Accepted Solution

Avatar

Correct answer by
Level 5

So I've decided to just reduce my concurrent workflows to 20

however AEM still shows more than 20 workflows in the 'RUNNING' status.

I tested with about 200 files. My setup currently consists of one author & two offloading instances.

I changed the config of Apache Sling Job Queue Configuration...set the job topic to:

com/adobe/granite/workflow/job*

And maximum parallel jobs to 20.


Thanks.

View solution in original post

7 Replies

Avatar

Level 10

 If you find the load too much - try reducing load. 

Avatar

Level 2

Have you taken any thread dumps when the system is slow/hung to see what thread(s) may be causing it?  What does the heap look like?  Is there a lot of GC activity?

Avatar

Level 10

I think the system is slow because they are trying to post thousands of DAM assets @ once. I recommend breaking the job into manageable chunks of assets so AEM can handle it. 

Avatar

Level 5

ZygW wrote...

Have you taken any thread dumps when the system is slow/hung to see what thread(s) may be causing it?  What does the heap look like?  Is there a lot of GC activity?

 

 


I'll do this and post results. Can someone explain why this would cause workflows to be processed slower though? I imagine they are just in a queue and get processed in a X amount of times in parallel.

However the case here is that each individual workflow process time gradually gets slower and slower.

Avatar

Correct answer by
Level 5

So I've decided to just reduce my concurrent workflows to 20

however AEM still shows more than 20 workflows in the 'RUNNING' status.

I tested with about 200 files. My setup currently consists of one author & two offloading instances.

I changed the config of Apache Sling Job Queue Configuration...set the job topic to:

com/adobe/granite/workflow/job*

And maximum parallel jobs to 20.


Thanks.

Avatar

Level 10

Are you seeing better AEM performance? 

Avatar

Level 5

Yes it did seem to perform better however when I checked the config node in the back end it was still set to '-1' which

I find quite odd...