"Failed to create thread" while calling API thousands of times

[#1]

Hello,

I have a simple workflow that is failing on the JS Code box.

I am getting "Failed to create thread" when I try to process more than 18,000 recipients.

Here is what the wokflow does: 

It targets recipients, add empty columns and populate these columns by calling an external API.

In the javascript code, I am basically looping over all the targets and make an API call with this function :

function processResponse(response, userid){
    var response = response[1];
    response = JSON.parse(response);

// ... parse JSON reponsive and collect data

    var query = "UPDATE ....";
// ... update empty columns in temporary table with retrieved data

    sqlExec(query);
}

function callService(userid) {
    var command = "wget -qO- --header 'Authorization: Basic XXXX' https://example.com/contacts/"+userid;
    var response = execCommand(command);
    processResponse(response, userid);
}

for each(var usertId in sqlSelectUserIds) {
  callService(usertId[0]);
}

cnx.dispose(); 

Do you know what's the problem?

Thank you for your help.

R

[roro_coeur · Accepted solution]

Hi @roro_coeur - This is a known issue with JS activities. You only have a finite amount of memory available to execute the javascript and depending on how much processing you want to do, you can run out of available memory to do processing or to spawn execCommand processes like you found. I would see if there is a way you can batch these external requests rather than doing individual requests.

Another option is to only process 10k records at a time in your JS activity and then loop around again in the workflow back to the same JS activity and then process the next lot of records. This will hopefully reset the available memory available to the JS activity.

Cheers Darren

[roro_coeur · Accepted solution]

Hi @Darren_Bowers, thank you for your answer! Wow, this is quite surprising that ACC has this limitation... 10K is not that much. Adding a Loop to the workflow to reset the memory sounds a lil bit sketchy and overkill. Thanks!

[Darren_Bowers · Accepted solution]

I have definitely ran into the same restrictions as you doing individual REST calls from the temporary workflow schema, which is why for transactions like yours we have opted to batch these using S3 buckets and process them externally. If you use something like AWS Lambda + S3 this shouldnt be too difficult to do in real time.