Workflow automatically runs into error after few API calls

[AkshayAnand]

Hi Team

I am calling a Rebrandly API service to convert adobe generated WebApps link to short URLs. But after few calls (approx 1000 calls)/sometime (approx 12-15 mins) the workflow automatically runs into error as :-

"IOB-090008 Failure while resolving address 'api.rebrandly.com' (errno=-11)"

Find the code below used 

var startLine = 0;
var limit = 9999;
var loop= true;

while (loop)
{
//Fetching the customerIDs dynamically
var query = xtk.queryDef.create(
    <queryDef schema="temp:query" operation="select" startLine={startLine} limit={limit} >
    <select>
    <node expr="@SMS"/>
    </select>
    </queryDef>);

var res = query.ExecuteQuery();
if (Object.keys(res).length>0)
{
for each (var rcp in res.query)
{

//Fetching the customerIDs dynamically whose shortURL is EMPTY
    var query2 = xtk.queryDef.create(
    <queryDef schema="dbSchema" operation="select">
    <select>
    <node expr="@SMS"/>
    <node expr="@shortURL"/>    
    </select>
    <where>
    <condition expr={"@SMS ='"+ rcp.@SMS+"'"}/>
    </where>
    </queryDef>);
    
var res2 = query2.ExecuteQuery();

for each (var rcp2 in res2.dbSchema)
{
//Condition to check if shortURL is EMPTY
if (rcp2.@shortURL == '')
{

var ID = rcp2.@SMS;

//Script to get the hashed ID.
var encrypt= cryptString(ID);

var url= escapeUrl(encrypt);



//Script to get the Full URL.
var fullURL = 'ADOBEGENERATEDURL'+url;


var Data = "{\"domain\":{\"id\":\"XYZ\"},\"destination\":\""+ fullURL+"\"}";


//API Call to get short URL.
var http=new HttpClientRequest("https://rebrandly.com/");
http.header ["Content-Type"] = "application/json";
http.header ["apikey"] = "ABC";
http.header ["workspace"] = "DEF";
http.header ["Accept"] = "application/json";

http.method = "POST" ;
http.body = Data;
http.execute();

var resposne = http.response.body;

//Get the response in string format.
var result = http.response.body.toString();

//Parse the response in JSON format.
var result3 = JSON.parse(resposne);

//Pull ShortURL in string format from parsed response.
var result6 = result3.shortUrl ;
}
}

}
startLine = startLine + limit +1;  
}
else
loop=false;
}

Further, I have also tried to used "Keep Alive" mode to send requests but unfortunately it throws the below error :

"13/12/2022 13:08:32 js3 HTTP header 'WWW-Authenticate' used for proxy authentication not found."

Code used 

//API connection 
var http=new HttpClientRequest("https://rebrandly.com");
http.header ["Content-Type"] = "application/json";
http.header ["apikey"] = "ABC";
http.header ["workspace"] = "DEF";
http.header ["Accept"] = "application/json";

http.method = "POST" ;
http.connect();
// Connection kept alive

logInfo("Connection Established.");

var startLine = 0;
var limit = 9999;
var loop= true;

while (loop)
{
//Fetching the customerIDs dynamically
var query = xtk.queryDef.create(
    <queryDef schema="temp:query" operation="select" startLine={startLine} limit={limit} >
    <select>
    <node expr="@SMS"/>
    </select>
    </queryDef>);

var res = query.ExecuteQuery();
if (Object.keys(res).length>0)
{
for each (var rcp in res.query)
{

//Fetching the customerIDs dynamically whose shortURL is EMPTY
    var query2 = xtk.queryDef.create(
    <queryDef schema="dbSchema" operation="select">
    <select>
    <node expr="@SMS"/>
    <node expr="@shortURL"/>    
    </select>
    <where>
    <condition expr={"@SMS ='"+ rcp.@SMS+"'"}/>
    </where>
    </queryDef>);
    
var res2 = query2.ExecuteQuery();

for each (var rcp2 in res2.dbSchema)
{
//Condition to check if shortURL is EMPTY
if (rcp2.@shortURL == '')
{

var ID = rcp2.@SMS;

//Script to get the hashed ID.
var encrypt= cryptString(ID);

var url= escapeUrl(encrypt);



//Script to get the Full URL.
var fullURL = 'ADOBEGENERATEDURL'+url;


var Data = "{\"domain\":{\"id\":\"XYZ\"},\"destination\":\""+ fullURL+"\"}";


//API Call to get short URL.

http.body = Data;
http.execute();

var resposne = http.response.body;

//Get the response in string format.
var result = http.response.body.toString();

//Parse the response in JSON format.
var result3 = JSON.parse(resposne);

//Pull ShortURL in string format from parsed response.
var result6 = result3.shortUrl ;
}
}

}
startLine = startLine + limit +1;  
}
else
loop=false;
}

 Could you please give any suggestions or workaround for the same?

Regards

Akshay

[Denis_Bozonnet]

Hi @AkshayAnand 

If your isntance is hosted by Adobe on AWS, there is a AWS constraint when it comes to outbond API call and the trick is to sleep your API calls before the 1,000 is reached for a few seconds.... Let say every 500 calls, you sleep for 10s or 30s and carries on.... The limit can't be lifted as it is imposed by AWS...

Also, would not be better if the 3rd party provide a bulk API instead of a single API, it'd be certainly much more efficient for all system concerned.

Thanks

Denis.

[AkshayAnand]

Hi @Denis_Bozonnet 

Thanks for the reply and suggestion.

I have already used sleep in my code but still after sometime the flow runs into the same error which I mentioned.

Also, according to your suggestion a 3rd party system should be used to get the bulk API response from Rebrandly rather than adobe making the call to Rebrandly. (Just for my clarification  )

Regards

Akshay

[Denis_Bozonnet · Accepted solution]

HI @AkshayAnand 

Aie, haven't seen the error after a sleep and it seems to have worked for me (maybe the sleep was few minutes, I won't be able to know). As for the API, of course the bulk API would have to be designed to handle bulk injection and response with probably each individual record having its own response in the payload....Only the 3rd party system could tell you the feasibility (or availability)

Thanks

Denis.