Avatar

Correct answer by
Level 6

Hi @Ken_Qrious, sorry for the delay, for me, you need first to be sure that those continuous deliveries are not integrated in an operation. If this is the case, regarding how to troubleshoot here is the process I suggest you: open the technical workflow "operationMgt" and open the "scheduler" js activity.

 

On line 1: you have a hint for bad solution (I'll come back to it later): it takes the value of the option "NmsOperation_LimitConcurrency" in order to know the max operation activities that can be run in parallel. So, dirty solution should to increase option value higher. Now, heatmap is not necessarily helping here as we not talking about just workflows but about workflows/deliveries/import/simulation/hypothesis activities.

 

But first, let's add a logInfo('alert: max activities(' + maxRunnings +') reached - Jobs in queue: ' + iJobCount). Normally it should near line 338 (in the "iJobCount >= maxRunnings" condition) If you see this log Message in the workflow logs when you have the issue => you've pin pointed the problem

 

The best thing to do is then to understand why you reach the max... In your case, the highest probability is about creating continuous deliveries in some operations. I would then recommend you to ask your tech lead to find another implementation method as this is not part of a best practice at all for this reason

 

As a temporary workarround your can increase the NmsOperation_LimitConcurrency option value... But it should be only temporary:

having a huge number of operations activities in parallel is not recommended for global performance platform. This solution will not change the fact that you'll need to implement in a different way as this type of implementation is a structural problem and will not get over with time

View solution in original post