Expand my Community achievements bar.

SOLVED

Process data received by External Signal

Avatar

Level 5

Hello All,

 

We have a common workflow with an External signal at the start.

This workflow is called multiple times by various other workflows, sometimes 15/20 per minute.

 

We want to process each request 1 by 1, rather than in bulk. i.e When one request gets completed the next request should be picked up. I understand that External Signal queue's up the request, is there a way to handle this one at a time, any reference would really help

 

 

Regards,

DG

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

Hello @dipendu_g ,

basically each time you call the workflow with the signal activity in it, will create new workflow, adds your input data params to it and will run it.

If you call signal activity 10 times it will create 10 workflows

You will need to:

  • create template that contains all activities after you signal activity.
    But instead of signal activity you put start change the name to eg "start00". Set the execution folder somewhere where all the workflows will be dumped so you have them nicelly separated. Save the template as "someNameWorkflow"
  • in the original workflow  you are only left with signal activity. Where you send data from somwhere right?
    Go into the intiialization script and add following code
var initVars = "vars.myvar1='" + vars.myvar1 + "'\n;"+
               "vars.myvar2='" + vars.myvar1 + "'\n;";

var workflowVars = 
<workflow label={"Some nice name " + formatDate(new Date,"- %4Y%2M%2D%02H:%02N:%02S")}
<activities>
  <start name="start00">
     <initScript>{initVars}</initScript>
   </start>
</activities>
</workflow>;
NLWS.xtkWorkflow.spawn(
'someNameWorkflow',
<workflow>
      <activities>
        <start name="start00">
         <initScript></initScrip>
        </extern>
      </activities>
    </workflow>
)​
  • then when finished and you send signal to the wokflow with the signal activity it will create new workflow  everytime. After some time you will end up with thousands of workflows. Ideally is to create clean up service. It will be another workflow sheduled to run every let's say 30m to remove all workflows from the folder that are finished and last start time is more than 30m. then simply we call update data activity with delete option. THis will only keep all failed workflows for you to fix and restart. and worfklows that are not older than 30m

 

Marcel

View solution in original post

9 Replies

Avatar

Community Advisor

Hello @dipendu_g,

you want to speed up the through put over this workflow right?

I would spawn the workflow, based on template which you create out of everything after the signal activity, each time request comes. Signal activity initialization script will spawn new workflow, which will process the signal data input. This way the queue wait time will decrease.

 

Additionally you can then create clean up workflow which will clean up everything older than hour and finished.

 

 

Marcel Szimonisz

MarTech Consultant
for more tips visit my blog
https://www.martechnotes.com/

Avatar

Level 5

Hi @Marcel_Szimonisz,

 

Any example workflow, that I can have a look, with the code.

 

 

Can you please elaborate on this part "Additionally you can then create clean up workflow which will clean up everything older than hour and finished."

 

Any working example or tidbits might help me to start

 

Regards.

DG

Avatar

Correct answer by
Community Advisor

Hello @dipendu_g ,

basically each time you call the workflow with the signal activity in it, will create new workflow, adds your input data params to it and will run it.

If you call signal activity 10 times it will create 10 workflows

You will need to:

  • create template that contains all activities after you signal activity.
    But instead of signal activity you put start change the name to eg "start00". Set the execution folder somewhere where all the workflows will be dumped so you have them nicelly separated. Save the template as "someNameWorkflow"
  • in the original workflow  you are only left with signal activity. Where you send data from somwhere right?
    Go into the intiialization script and add following code
var initVars = "vars.myvar1='" + vars.myvar1 + "'\n;"+
               "vars.myvar2='" + vars.myvar1 + "'\n;";

var workflowVars = 
<workflow label={"Some nice name " + formatDate(new Date,"- %4Y%2M%2D%02H:%02N:%02S")}
<activities>
  <start name="start00">
     <initScript>{initVars}</initScript>
   </start>
</activities>
</workflow>;
NLWS.xtkWorkflow.spawn(
'someNameWorkflow',
<workflow>
      <activities>
        <start name="start00">
         <initScript></initScrip>
        </extern>
      </activities>
    </workflow>
)​
  • then when finished and you send signal to the wokflow with the signal activity it will create new workflow  everytime. After some time you will end up with thousands of workflows. Ideally is to create clean up service. It will be another workflow sheduled to run every let's say 30m to remove all workflows from the folder that are finished and last start time is more than 30m. then simply we call update data activity with delete option. THis will only keep all failed workflows for you to fix and restart. and worfklows that are not older than 30m

 

Marcel

Avatar

Level 5

HI @Marcel_Szimonisz,

 

Fantastic , Thanks for such a detailed explanation. I will try it out !!!.

One follow-up question:

 

Ideally when multiple data signal comes to a workflow, adobe campaign manages this automatically, then in which case do we need to spawn.

 

Also is there a way to serially process the records as well, when data comes to a signal, where 2nd record only picked up when the first one is done

 

Regards,

DG

Avatar

Community Advisor

The signal will process all incoming calls one by one. Only improovement is here that the processing is then done in separate process somehow creating threads in adobe campaign  

Maybe the signal init activity will not have access to the variables you have sent ovar the post event.. so if that does not work add second javascript activity that will spawn the technical process.


also there might be some ifs for example if you are updating some recods and you get same record one after another and no the latest changes will prevail in the update because one workflow finished sooner than the other.

 

Avatar

Level 5

Hi @Marcel_Szimonisz,

 

That is precisely the reason was trying for an option that would only start processing once the first workflow has completed

Avatar

Community Advisor

it should work then without anything it will be processed one after another.

Avatar

Level 5

Actually what is happening is if 1 record/data is getting processed by the workflow with the external signal and during that time another data comes, it interferes, as there are cases where the second data is dependent on what the first set updates.

 

We were trying use the lock, but as the workflow has multiple end points with various combinations, releasing lock also becomes a challenge.

 

I added the query to check if any task is running, and using a subsequent wait and again back to second task, but I see data getting lost sometimes in that process

 

Regards,

DG

Avatar

Community Advisor

Hello @dipendu_g,

have you tried to save each call to database, use it as queue, and then process it from there?

 

Marcel