using httpClient for real time data streaming

[kevinedmond]

Hello everyone.

we are currently faccing a dead end with some javscript call (in js  and Workflow).

our use case is the following :

we need to make an httpcall to a outside server to get some data.

the server is currently sending data on a stream  that is not closed as it will keep sending data if new events happen.

If we are making a call with curl (outside adobe) the flow will be the following

curl call

stream open

server send data

stream stay open

server send somr keep alive stream to be sure the stream stay open

(if new data , server send the new data)

We cannot make the call with the regular httpClientRequest as it generate an error as soon as it get the data.

we cannot make a curl call in the workflow eitheir as the task stay "stuck" on the curl call, so we got no way to read the stream. (and we have to kill the workflow => no data in that case)

Has anyone ever get that requirement and any idea on how to do it?

Thank you

[Marcel_Szimonisz]

Hello @kevinedmond,

The url has to be whitelisted in the urlPermission in the instance config. I guess this part is done.

HTTP calls are made by HttpClientRequest

Marcel

[kevinedmond]

Hello Marcel.

the url is already whitelisted (we are making  other calls for it). our issue is really on the fact that the httpclientRequest library used by Adobe doesn't seems to be able to handle streaming answer.

(unless i miss some parameter on the call)

[Marcel_Szimonisz]

Can you print whatever the reaponse body is? 

[kevinedmond]

sorry i did miss that answer.

there are no reponse from adobe httpclientreques as the  process just stay hanged

[LaurentLam]

According to me, there's no healthy way to keep an AC HTTP request kept alive too long (specially with SSL)
I would recommand to use a kafka topic between AC and the distant server: you'll workarround your problem and you'll gain in resilience

1- you'll httpClientRequest to post data to an endPoint that will (hi Adobe, we're still waiting for a kafka connector) integrate it in the kafka queue.
2- perhaps there's an existing kafka connector with your distant server to consume a kafka topic

[kevinedmond]

thanks @LaurentLam .

sadly there is no kafka connector on the distant server.

I don't need to keep the request alive, the requested behavior would be :

open connection

make the call

catch the answer

close the request

make another request

but as it is a streaming connection, Adobe doesn't want me to close the connection and analyze the answer

thanks