Avatar

Level 4

Hi Team,

We have some hundreds of workflows running in parallel everyday through a schedular activity which is triggered daily.
So each of these workflows having a wait activity in between the workflow like below:

 

Ramaswami_0-1590004706451.png

 

 

The wait period is around 2 days due to which number of concurrent workflows is exceeding the limit which is specified as every workflow has a wait activity it continuously holding the execution.

So our idea is to split up the workflow into two
1. first workflow will run only up to the email activity 1 and then sends the export of recipient results to some S3 bucket.
2. the next workflow will pickup the recipient results from S3 bucket after two days and will process the next execution( that is sending the next email).

Here my problem is i have exported the results in csv format to S3 bucket sucessfully so part 1 is done.
part 2: i have downloaded the recipient file from S3 bucket and when i am loading the data and sending the results to continously delivery. it is throwing the below error:

Part 1: Extracting the results and sending email and storing the result file in S3 bucket: which went sucessfully

Ramaswami_1-1590004783097.png

 

part 2: 

Ramaswami_2-1590004856076.png

 

I am downing the file with columns used as additional data in part 1 and uploading the data in part 2. i am using update data activity in order to linkup the targetting columns with recipient table and then sending an email. it should send only 4 emails but it is continously sending me emails like 20,21,22 and so on. 

 

When i don't use the update data activity. i am getting an error like below : 

if i dont use update data activity : My data loading result will be like this : 

Ramaswami_3-1590005071386.png

and i will feed this to continous delivery i am getting this erro: 

Ramaswami_4-1590005406978.png

Let me know if any ready made solution or any alternative we have for this. 

 

 

hi @Milan_Vucetic 

 

i have followed your second suggestion. kept a intersection query with data loading query: 

 

Ramaswami_0-1590162127309.png

 

So i am getting the below error : 

Ramaswami_1-1590162162929.png

 

Am i doing mistake anywhere. i guess this intersection takes some time as the intersection query should run all the records first and then intersect with the temp table.