HttpClientRequest volume limitation?

[davidh2892249]

Hi there,

I have a requirement where I need to be able to make around 5000 API calls out from a workflow.

It is a different URL for each request (kind of the same, but a different ID to return appropriate data in JSON format).

I have build out a Javascript activity that uses a loop to iterate through the result of a queryDef collection that has the IDs.

All is working fine except that I seem to be getting an error after around 1,017 requests, which makes the workflow error.

IOB-090008 Failure while resolving address 'www.example.com' (errno=-11)

I have checked and it is an Adobe Campaign rather than the API endpoint having a volume constraint of requests it can accept.

Below is a stripped back version of the code (where I have hardcoded the url of the request) just so I could test a loop and it errors at the same volume (which it does)

// Create variable to use in loop

var numberOfCalls = 5000;

//LOOP X TIMES
for (var i = 0; i < numberOfCalls; i++) {

var httpRequest = new HttpClientRequest("https://www.example.com/api/xyz");
httpRequest.header["User-Agent"] = "Adobe Campaign";
httpRequest.method = 'GET';
httpRequest.execute();
var response = httpRequest.response;
var responseBody = httpRequest.response.body;
var responseBodyString = httpRequest.response.body.toString();
var responseJSON = JSON.parse(responseBodyString);
logInfo("Counter: "+i)
logInfo(responseJSON.name)
}

Can anyone advise how we can remedy this?

Is it a known limitation?

I know you need a loop to handle processing in 10,000 chunks, but I wasn't aware of a limitation on HttpClientRequest

I've seen the "keep alive" example here: https://experienceleague.adobe.com/developer/campaign-api/api/c-HttpClientRequest.html , but in our real scenario we need to include a variable in the URL of the request, meaning each is different - so I don't think that's an option?

Any help gratefully received.

Thanks

David

[Darren_Bowers · Accepted solution]

Hi @davidh2892249 - you might be running into a Javascript memory limitation with the amount of API calls you are making in a single Javascript activity. If you are an on-prem install, you can try bumping up the Javascript memory allocation : https://experienceleaguecommunities.adobe.com/t5/adobe-campaign-standard/javascript-query-limitation...

I have run into this a few times and have moved from a model where Campaign does all the API requests individually to offloading the requests in batch to a micro-services architecture and that performs the calls in an AWS Lambda instance. It seems to be much more reliable.

Cheers

Darren

[Sukrity_Wadhwa]

Hi @davidh2892249,

Was the given solution helpful to resolve your query or do you still need more help here? Do let us know.

Thanks!