Process data received by External Signal

[dipendu_g]

Hello All,

We have a common workflow with an External signal at the start.

This workflow is called multiple times by various other workflows, sometimes 15/20 per minute.

We want to process each request 1 by 1, rather than in bulk. i.e When one request gets completed the next request should be picked up. I understand that External Signal queue's up the request, is there a way to handle this one at a time, any reference would really help

Regards,

DG

[Marcel_Szimonisz]

Hello @dipendu_g,

you want to speed up the through put over this workflow right?

I would spawn the workflow, based on template which you create out of everything after the signal activity, each time request comes. Signal activity initialization script will spawn new workflow, which will process the signal data input. This way the queue wait time will decrease.

Additionally you can then create clean up workflow which will clean up everything older than hour and finished.

Marcel Szimonisz

MarTech Consultant
for more tips visit my blog
https://www.martechnotes.com/

[dipendu_g]

Hi @Marcel_Szimonisz,

Any example workflow, that I can have a look, with the code.

Can you please elaborate on this part "Additionally you can then create clean up workflow which will clean up everything older than hour and finished."

Any working example or tidbits might help me to start

Regards.

DG

[Marcel_Szimonisz · Accepted solution]

Hello @dipendu_g ,

basically each time you call the workflow with the signal activity in it, will create new workflow, adds your input data params to it and will run it.

If you call signal activity 10 times it will create 10 workflows

You will need to:

• create template that contains all activities after you signal activity.
  But instead of signal activity you put start change the name to eg "start00". Set the execution folder somewhere where all the workflows will be dumped so you have them nicelly separated. Save the template as "someNameWorkflow"
• in the original workflow  you are only left with signal activity. Where you send data from somwhere right?
  Go into the intiialization script and add following code

var initVars = "vars.myvar1='" + vars.myvar1 + "'\n;"+
               "vars.myvar2='" + vars.myvar1 + "'\n;";

var workflowVars = 
<workflow label={"Some nice name " + formatDate(new Date,"- %4Y%2M%2D%02H:%02N:%02S")}
<activities>
  <start name="start00">
     <initScript>{initVars}</initScript>
   </start>
</activities>
</workflow>;
NLWS.xtkWorkflow.spawn(
'someNameWorkflow',
<workflow>
      <activities>
        <start name="start00">
         <initScript></initScrip>
        </extern>
      </activities>
    </workflow>
)​

• then when finished and you send signal to the wokflow with the signal activity it will create new workflow  everytime. After some time you will end up with thousands of workflows. Ideally is to create clean up service. It will be another workflow sheduled to run every let's say 30m to remove all workflows from the folder that are finished and last start time is more than 30m. then simply we call update data activity with delete option. THis will only keep all failed workflows for you to fix and restart. and worfklows that are not older than 30m

Marcel

[dipendu_g]

HI @Marcel_Szimonisz,

Fantastic , Thanks for such a detailed explanation. I will try it out !!!.

One follow-up question:

Ideally when multiple data signal comes to a workflow, adobe campaign manages this automatically, then in which case do we need to spawn.

Also is there a way to serially process the records as well, when data comes to a signal, where 2nd record only picked up when the first one is done

Regards,

DG

[Marcel_Szimonisz]

The signal will process all incoming calls one by one. Only improovement is here that the processing is then done in separate process somehow creating threads in adobe campaign 🙂 

Maybe the signal init activity will not have access to the variables you have sent ovar the post event.. so if that does not work add second javascript activity that will spawn the technical process.

also there might be some ifs for example if you are updating some recods and you get same record one after another and no the latest changes will prevail in the update because one workflow finished sooner than the other.

 

[dipendu_g]

Hi @Marcel_Szimonisz,

That is precisely the reason was trying for an option that would only start processing once the first workflow has completed

[Marcel_Szimonisz]

it should work then without anything it will be processed one after another.

[dipendu_g]

Actually what is happening is if 1 record/data is getting processed by the workflow with the external signal and during that time another data comes, it interferes, as there are cases where the second data is dependent on what the first set updates.

We were trying use the lock, but as the workflow has multiple end points with various combinations, releasing lock also becomes a challenge.

I added the query to check if any task is running, and using a subsequent wait and again back to second task, but I see data getting lost sometimes in that process

Regards,

DG

[Marcel_Szimonisz]

Hello @dipendu_g,

have you tried to save each call to database, use it as queue, and then process it from there?

Marcel