using httpClient for real time data streaming | Community
Skip to main content
kevinedmond
kevinedmond
Level 2
September 27, 2021
Question

using httpClient for real time data streaming

  • September 27, 2021
  • 2 replies
  • 2209 views

Hello everyone.

we are currently faccing a dead end with some javscript call (in js  and Workflow).

our use case is the following :

we need to make an httpcall to a outside server to get some data.

the server is currently sending data on a stream  that is not closed as it will keep sending data if new events happen.

 

If we are making a call with curl (outside adobe) the flow will be the following

curl call

stream open

server send data

stream stay open

server send somr keep alive stream to be sure the stream stay open

(if new data , server send the new data)

We cannot make the call with the regular httpClientRequest as it generate an error as soon as it get the data.

we cannot make a curl call in the workflow eitheir as the task stay "stuck" on the curl call, so we got no way to read the stream. (and we have to kill the workflow => no data in that case)

Has anyone ever get that requirement and any idea on how to do it?

 

Thank you

 

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

2 replies

Marcel_Szimonisz
Community Advisor
Marcel_SzimoniszCommunity Advisor
Community Advisor
September 27, 2021

Hello @kevinedmond,

The url has to be whitelisted in the urlPermission in the instance config. I guess this part is done.

HTTP calls are made by HttpClientRequest

 

Marcel

kevinedmond
kevinedmondAuthor
Level 2
September 28, 2021

Hello Marcel.

the url is already whitelisted (we are making  other calls for it). our issue is really on the fact that the httpclientRequest library used by Adobe doesn't seems to be able to handle streaming answer.

(unless i miss some parameter on the call)

Marcel_Szimonisz
Community Advisor
Marcel_SzimoniszCommunity Advisor
Community Advisor
September 28, 2021

Can you print whatever the reaponse body is? 

LaurentLam
LaurentLam
Level 5
September 27, 2021

According to me, there's no healthy way to keep an AC HTTP request kept alive too long (specially with SSL)
I would recommand to use a kafka topic between AC and the distant server: you'll workarround your problem and you'll gain in resilience

1- you'll httpClientRequest to post data to an endPoint that will (hi Adobe, we're still waiting for a kafka connector) integrate it in the kafka queue.
2- perhaps there's an existing kafka connector with your distant server to consume a kafka topic

kevinedmond
kevinedmondAuthor
Level 2
September 28, 2021

thanks @laurentlam .

sadly there is no kafka connector on the distant server.

I don't need to keep the request alive, the requested behavior would be :

open connection

make the call

catch the answer

close the request

make another request

but as it is a streaming connection, Adobe doesn't want me to close the connection and analyze the answer

thanks

 

Terms & ConditionsAccessibility statement

Loading footer...