DAM Offloading often choking up with high volume of uploads

[TheBigRed]

I have a issue where my IT team has to keep rebuilding offloading agents. The problem occurs when we upload high amounts of 

XML content roughly about 50 000 - 100 000 files. Only OOTB workflows are being run on it (DAM Update Asset). AEM processes couple thousands of files extremely fast

then it eventually gets slower and slower where it just comes to a halt. What can be the issue here? Adobe Tech guy said it should be able handle small files like that in large bulks.

I'm uploading it through a desktop client which sends the files to an OSGi Servlet. 

Thanks all!

[smacdonald2008]

 If you find the load too much - try reducing load. 

[ZygW]

Have you taken any thread dumps when the system is slow/hung to see what thread(s) may be causing it?  What does the heap look like?  Is there a lot of GC activity?

[smacdonald2008]

I think the system is slow because they are trying to post thousands of DAM assets @ once. I recommend breaking the job into manageable chunks of assets so AEM can handle it. 

[TheBigRed]

ZygW wrote...

Have you taken any thread dumps when the system is slow/hung to see what thread(s) may be causing it?  What does the heap look like?  Is there a lot of GC activity?

I'll do this and post results. Can someone explain why this would cause workflows to be processed slower though? I imagine they are just in a queue and get processed in a X amount of times in parallel.

However the case here is that each individual workflow process time gradually gets slower and slower.

[TheBigRed · Accepted solution]

So I've decided to just reduce my concurrent workflows to 20

however AEM still shows more than 20 workflows in the 'RUNNING' status.

I tested with about 200 files. My setup currently consists of one author & two offloading instances.

I changed the config of Apache Sling Job Queue Configuration...set the job topic to:

com/adobe/granite/workflow/job*

And maximum parallel jobs to 20.

Thanks.

[smacdonald2008]

Are you seeing better AEM performance? 

[TheBigRed]

Yes it did seem to perform better however when I checked the config node in the back end it was still set to '-1' which

I find quite odd...