Hi there,
I have a requirement where I need to be able to make around 5000 API calls out from a workflow.
It is a different URL for each request (kind of the same, but a different ID to return appropriate data in JSON format).
I have build out a Javascript activity that uses a loop to iterate through the result of a queryDef collection that has the IDs.
All is working fine except that I seem to be getting an error after around 1,017 requests, which makes the workflow error.
IOB-090008 Failure while resolving address 'www.example.com' (errno=-11)
I have checked and it is an Adobe Campaign rather than the API endpoint having a volume constraint of requests it can accept.
Below is a stripped back version of the code (where I have hardcoded the url of the request) just so I could test a loop and it errors at the same volume (which it does)
// Create variable to use in loop
var numberOfCalls = 5000;
//LOOP X TIMES
for (var i = 0; i < numberOfCalls; i++) {
var httpRequest = new HttpClientRequest("https://www.example.com/api/xyz");
httpRequest.header["User-Agent"] = "Adobe Campaign";
httpRequest.method = 'GET';
httpRequest.execute();
var response = httpRequest.response;
var responseBody = httpRequest.response.body;
var responseBodyString = httpRequest.response.body.toString();
var responseJSON = JSON.parse(responseBodyString);
logInfo("Counter: "+i)
logInfo(responseJSON.name)
}
Can anyone advise how we can remedy this?
Is it a known limitation?
I know you need a loop to handle processing in 10,000 chunks, but I wasn't aware of a limitation on HttpClientRequest
I've seen the "keep alive" example here: https://experienceleague.adobe.com/developer/campaign-api/api/c-HttpClientRequest.html , but in our real scenario we need to include a variable in the URL of the request, meaning each is different - so I don't think that's an option?
Any help gratefully received.
Thanks
David
Solved! Go to Solution.
Views
Replies
Total Likes
Hi @davidh2892249 - you might be running into a Javascript memory limitation with the amount of API calls you are making in a single Javascript activity. If you are an on-prem install, you can try bumping up the Javascript memory allocation : https://experienceleaguecommunities.adobe.com/t5/adobe-campaign-standard/javascript-query-limitation...
I have run into this a few times and have moved from a model where Campaign does all the API requests individually to offloading the requests in batch to a micro-services architecture and that performs the calls in an AWS Lambda instance. It seems to be much more reliable.
Cheers
Darren
Hi @davidh2892249 - you might be running into a Javascript memory limitation with the amount of API calls you are making in a single Javascript activity. If you are an on-prem install, you can try bumping up the Javascript memory allocation : https://experienceleaguecommunities.adobe.com/t5/adobe-campaign-standard/javascript-query-limitation...
I have run into this a few times and have moved from a model where Campaign does all the API requests individually to offloading the requests in batch to a micro-services architecture and that performs the calls in an AWS Lambda instance. It seems to be much more reliable.
Cheers
Darren
Hi @davidh2892249,
Was the given solution helpful to resolve your query or do you still need more help here? Do let us know.
Thanks!
Views
Replies
Total Likes
Views
Likes
Replies
Views
Likes
Replies
Views
Likes
Replies