Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
Bedrock Mission!

Learn more

View all

Sign in to view all badges

SOLVED

HttpClientRequest volume limitation?

Avatar

Level 5

Hi there,

 

I have a requirement where I need to be able to make around 5000 API calls out from a workflow.

It is a different URL for each request (kind of the same, but a different ID to return appropriate data in JSON format).

I have build out a Javascript activity that uses a loop to iterate through the result of a queryDef collection that has the IDs.

All is working fine except that I seem to be getting an error after around 1,017 requests, which makes the workflow error.

 

IOB-090008 Failure while resolving address 'www.example.com' (errno=-11)

 

I have checked and it is an Adobe Campaign rather than the API endpoint having a volume constraint of requests it can accept.

 

Below is a stripped back version of the code (where I have hardcoded the url of the request) just so I could test a loop and it errors at the same volume (which it does)

 

 

// Create variable to use in loop

var numberOfCalls = 5000;


//LOOP X TIMES
for (var i = 0; i < numberOfCalls; i++) {

var httpRequest = new HttpClientRequest("https://www.example.com/api/xyz");
httpRequest.header["User-Agent"] = "Adobe Campaign";
httpRequest.method = 'GET';
httpRequest.execute();
var response = httpRequest.response;
var responseBody = httpRequest.response.body;
var responseBodyString = httpRequest.response.body.toString();
var responseJSON = JSON.parse(responseBodyString);
logInfo("Counter: "+i)
logInfo(responseJSON.name)
}

Can anyone advise how we can remedy this?

Is it a known limitation?

I know you need a loop to handle processing in 10,000 chunks, but I wasn't aware of a limitation on HttpClientRequest

 

I've seen the "keep alive" example here: https://experienceleague.adobe.com/developer/campaign-api/api/c-HttpClientRequest.html , but in our real scenario we need to include a variable in the URL of the request, meaning each is different - so I don't think that's an option?

Any help gratefully received.

 

Thanks

 

David

 

 

 

 

1 Accepted Solution

Avatar

Correct answer by
Community Advisor

Hi @davidh2892249 - you might be running into a Javascript memory limitation with the amount of API calls you are making in a single Javascript activity. If you are an on-prem install, you can try bumping up the Javascript memory allocation : https://experienceleaguecommunities.adobe.com/t5/adobe-campaign-standard/javascript-query-limitation...

I have run into this a few times and have moved from a model where Campaign does all the API requests individually to offloading the requests in batch to a micro-services architecture and that performs the calls in an AWS Lambda instance. It seems to be much more reliable.

Cheers

Darren

View solution in original post

2 Replies

Avatar

Correct answer by
Community Advisor

Hi @davidh2892249 - you might be running into a Javascript memory limitation with the amount of API calls you are making in a single Javascript activity. If you are an on-prem install, you can try bumping up the Javascript memory allocation : https://experienceleaguecommunities.adobe.com/t5/adobe-campaign-standard/javascript-query-limitation...

I have run into this a few times and have moved from a model where Campaign does all the API requests individually to offloading the requests in batch to a micro-services architecture and that performs the calls in an AWS Lambda instance. It seems to be much more reliable.

Cheers

Darren

Avatar

Administrator

Hi @davidh2892249,

Was the given solution helpful to resolve your query or do you still need more help here? Do let us know.

Thanks!