DAM Offloading often choking up with high volume of uploads | Community
Skip to main content
TheBigRed
TheBigRed
Level 4
March 7, 2016
Solved

DAM Offloading often choking up with high volume of uploads

  • March 7, 2016
  • 7 replies
  • 1517 views

I have a issue where my IT team has to keep rebuilding offloading agents. The problem occurs when we upload high amounts of 

XML content roughly about 50 000 - 100 000 files. Only OOTB workflows are being run on it (DAM Update Asset). AEM processes couple thousands of files extremely fast

then it eventually gets slower and slower where it just comes to a halt. What can be the issue here? Adobe Tech guy said it should be able handle small files like that in large bulks.

I'm uploading it through a desktop client which sends the files to an OSGi Servlet. 

 

Thanks all!

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by TheBigRed

So I've decided to just reduce my concurrent workflows to 20

however AEM still shows more than 20 workflows in the 'RUNNING' status.

I tested with about 200 files. My setup currently consists of one author & two offloading instances.

I changed the config of Apache Sling Job Queue Configuration...set the job topic to:

com/adobe/granite/workflow/job*

And maximum parallel jobs to 20.


Thanks.

7 replies

smacdonald2008
smacdonald2008
Level 10
March 8, 2016

 If you find the load too much - try reducing load. 

M
Adobe Employee
MeTooTwoAdobe Employee
Adobe Employee
March 8, 2016

Have you taken any thread dumps when the system is slow/hung to see what thread(s) may be causing it?  What does the heap look like?  Is there a lot of GC activity?

smacdonald2008
smacdonald2008
Level 10
March 8, 2016

I think the system is slow because they are trying to post thousands of DAM assets @ once. I recommend breaking the job into manageable chunks of assets so AEM can handle it. 

TheBigRed
TheBigRedAuthor
Level 4
March 8, 2016

ZygW wrote...

Have you taken any thread dumps when the system is slow/hung to see what thread(s) may be causing it?  What does the heap look like?  Is there a lot of GC activity?

 

 


I'll do this and post results. Can someone explain why this would cause workflows to be processed slower though? I imagine they are just in a queue and get processed in a X amount of times in parallel.

However the case here is that each individual workflow process time gradually gets slower and slower.

TheBigRed
TheBigRedAuthorAccepted solution
Level 4
March 9, 2016

So I've decided to just reduce my concurrent workflows to 20

however AEM still shows more than 20 workflows in the 'RUNNING' status.

I tested with about 200 files. My setup currently consists of one author & two offloading instances.

I changed the config of Apache Sling Job Queue Configuration...set the job topic to:

com/adobe/granite/workflow/job*

And maximum parallel jobs to 20.


Thanks.

smacdonald2008
smacdonald2008
Level 10
March 9, 2016

Are you seeing better AEM performance? 

TheBigRed
TheBigRedAuthor
Level 4
March 9, 2016

Yes it did seem to perform better however when I checked the config node in the back end it was still set to '-1' which

I find quite odd...

Terms & ConditionsAccessibility statement

Loading footer...