"Failed to create thread" while calling API thousands of times

Avatar

roro_coeur

02-07-2020

Hello,

I have a simple workflow that is failing on the JS Code box.

I am getting "Failed to create thread" when I try to process more than 18,000 recipients.

 

Capture.PNG

 

Here is what the wokflow does: 

It targets recipients, add empty columns and populate these columns by calling an external API.

In the javascript code, I am basically looping over all the targets and make an API call with this function :

 

function processResponse(response, userid){
    var response = response[1];
    response = JSON.parse(response);

// ... parse JSON reponsive and collect data

    var query = "UPDATE ....";
// ... update empty columns in temporary table with retrieved data

    sqlExec(query);
}

function callService(userid) {
    var command = "wget -qO- --header 'Authorization: Basic XXXX' https://example.com/contacts/"+userid;
    var response = execCommand(command);
    processResponse(response, userid);
}

for each(var usertId in sqlSelectUserIds) {
  callService(usertId[0]);
}

cnx.dispose();

 

 

Do you know what's the problem?

 

Thank you for your help.

 

R

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Darren_Bowers

MVP

02-07-2020

Hi @roro_coeur - This is a known issue with JS activities. You only have a finite amount of memory available to execute the javascript and depending on how much processing you want to do, you can run out of available memory to do processing or to spawn execCommand processes like you found. I would see if there is a way you can batch these external requests rather than doing individual requests.

Another option is to only process 10k records at a time in your JS activity and then loop around again in the workflow back to the same JS activity and then process the next lot of records. This will hopefully reset the available memory available to the JS activity.

Cheers Darren

Answers (0)