Adobe Campaign Classic - Delayed trigger of External...

[Ken_Qrious]

For Campaign SMS deliveries we are utilizing a routing option to direct SMS deliveries to an external server via an external account definition and technical work flow.  Routinely we noticed that there is a delay of one to four days for some deliveries and we are unable to determine why this is occurring.  Since many SMS deliveries are time sensitive this has become a real issue.

High Level Flow

• Campaign Delivery - The SMS delivery node properties specifies : Routing => External Mobile Routing (External Accounts Definition)
• External Mobile Routing (External Accounts Definition) - Defined as type => routing, channel => mobile SMS, delivery mode => external, post processing => technical work flow and activity => signal
• Technical work flow - The workflow is initiated via an External Signal node

The delay occurs between the delivery node in the campaign being executed and the External Signal in the technical work flow detecting the delivery event.  

Any suggestions as to what might be causing the issue would be welcome.

[LaurentLam]

Hello,

simple naive question: is campaign module involved?

If yes, it can be an issue of implementation related to operationMgt: by default, you can have 50 paralled activities related to operations (workflow operation / deliveries in operation).

operationMgt manages the queue: if the 50 paralled activities are taken, it waits for a / some release(s).

This is why you should not use for example continuous deliveries inside operation as it hardly release the token

[CedricRey]

Also, @Ken_Qrious, Is there some lines in the delivery logs ? What are the dates for each lines ? It could be usefull to follow the differents steps, and see if the end of the delivery is long time before the post processing call, or if the problem is in the delivery itself ? Cedric

[Ken_Qrious]

We do utilize continues deliveries for several campaign deliveries so there may be something to your suggestion. A new question, is there a way to tell how many threads are in use? The heat map shows the current processes and that's way below 50.

[LaurentLam · Accepted solution]

Hi @Ken_Qrious, sorry for the delay, for me, you need first to be sure that those continuous deliveries are not integrated in an operation. If this is the case, regarding how to troubleshoot here is the process I suggest you: open the technical workflow "operationMgt" and open the "scheduler" js activity.

On line 1: you have a hint for bad solution (I'll come back to it later): it takes the value of the option "NmsOperation_LimitConcurrency" in order to know the max operation activities that can be run in parallel. So, dirty solution should to increase option value higher. Now, heatmap is not necessarily helping here as we not talking about just workflows but about workflows/deliveries/import/simulation/hypothesis activities.

But first, let's add a logInfo('alert: max activities(' + maxRunnings +') reached - Jobs in queue: ' + iJobCount). Normally it should near line 338 (in the "iJobCount >= maxRunnings" condition) If you see this log Message in the workflow logs when you have the issue => you've pin pointed the problem

The best thing to do is then to understand why you reach the max... In your case, the highest probability is about creating continuous deliveries in some operations. I would then recommend you to ask your tech lead to find another implementation method as this is not part of a best practice at all for this reason

As a temporary workarround your can increase the NmsOperation_LimitConcurrency option value... But it should be only temporary:

having a huge number of operations activities in parallel is not recommended for global performance platform. This solution will not change the fact that you'll need to implement in a different way as this type of implementation is a structural problem and will not get over with time

[Ken_Qrious]

Thanks for the feed back Laurent. I'll work with you suggestions to see if I can get to the bottom of the issue.

[Ken_Qrious]

Hello Laurent, the current setting for NmsOperation_LimitConcurrency is 15 so already higher than the default 10. I've added the alert as noted and get a continuous alert of "14/06/2021 14:25:45 scheduler alert: max activities(15) reached - Jobs in queue: 15" so this looks like the issue. Now I just need to work out how to identify what is included in the count. Looking at the logic it's not straight forward.

[Ken_Qrious]

Thanks Laurent, you comment about the the alert to the scheduler activity did the trick. We were able to identify 12 delivery and hypothesis processes which were active but stalled so only 3 slots of our 15 allowed were available. We cleared the stuck jobs which had been in place for a while and now all is good.

[LaurentLam]

Hi @Ken_Qrious, I didn't see your last updates. I'm glad that you've been able to narrow the issue. I don't know why your option is set to "15" as normally it's set to "50" by default. It means that someone downsized the value and it could be interesting to understand why. Managing DB perf? I think putting it back to 50 and managing the perf could be something to test

[Ken_Qrious]

Hey @LaurentLam , When you talk about the default parallel setting being 50, did you meant NmsOperation_LimitConcurrency? Previously I found a reference to the default being 10 at https://experienceleague.adobe.com/docs/campaign-classic/using/sending-messages/monitoring-deliverie...

[LaurentLam]

@Ken_Qrious: my bad, when I was in Adobe Support (until early 6.1 versions), the default value of this option was 50... But no Hypothesis feature was available for example

[Ken_Qrious]

Thanks @LaurentLam . I'm happy with the info you provided, it's helped me trace the source of our issue. I'd like to set your response as a correct reply but it looks like if I choose that option it would be your initial comment which would be flagged rather than the final solution.

[LaurentLam]

@Ken_Qrious... Doesn't matter: it reminds me my last position in adobe: I was hating to beg for a survey :)))))) If I think I can help and that it doesn't take too much time it's always a pleasure to share information... And be sure this kind of knowledge base is also huge for me in order to see how other colleagues handled issue I've faced off or that I will face off