Apache Kafka connector

[#1]

It's easy to create some libraries to communicate outside Adobe Campaign with the simplicity of Javascript and the HTTPClientRequest class, but as Kafka is not based on HTTP protocol, it's not possible to consume or produce over topics in Campaign. Apache Kafka needs specific libraries (Java, Python, NodeJS) to be used. It would be great to have a specific API into the Campaign javascript environment.

I'm sure that question is not new and the R&D team thought about it, and it's won't be technicaly easy, but this question is comming for several of my customers so I'm suggesting here.

Thanks

Cedric

[#2]

@CedricRey- interesting that you post this as we are looking at using Kafka to manage some enterprise-wide configuration changes.

Campaign Classic already uses Kafka for its Triggers / Pipelined implementation so you would expect it shouldn't be too difficult to expose those APIs internally for general Kafka use.

Cheers

Darren

[#3]

Hi @CedricRey,

As per teh product team's response - Campaign does use kafka topics internally. But what would really help is the business use case that you are trying to solve. This will help us drive and prioritze your request with the product team.

[#4]

Hello @Sukrity_Wadhwa ,

We would appreciate the opportunity to work directly on ACC with Kafka topics. Today we need a third party application to read / write datas for us and put it in Database.

But we are about to change the technical architecture and won't be able to expose database (soon hosted by Adobe).

I think this case is not particular to us and may be of  interess to many of other customers as Kafka is a common dataflow product, as you can see, @Darren_Bowers is also insterested.

Thank you.

Cedric

[#5]

Hi @CedricRey,

Internally we do use kafta containers. The product team is interested to know your specific business use case(s) for them to understand and eventually, better position and prioritize your request. 

Thanks!

[#6]

Hi @Sukrity_Wadhwa ,

Thank you for your response. At one of my clients, we've identified a crucial need for a Kafka connector since upgrading to Adobe Campaign V8. The hosted environment has posed challenges in maintaining real-time communication with our internal systems.

Our specific use case involves using Kafka to enhance data freshness and streamline the flow of information between Adobe Campaign and other internal IT systems. This would significantly improve the efficiency and relevance of our marketing campaigns.

I hope this helps in understanding our request. Any further assistance or guidance would be greatly appreciated.

Best regards,

Cedric

[#7]

hi @CedricRey @Sukrity_Wadhwa ,

Chiming in here with another use case that requires ACv7 (onprem) use case as a Kafka message producer.

Our client is depeding on AC to create/personalize message topics, and then distribute these to a Kafka endpoint using a stream connection - NOT HTTP.   Therefore we would expect to use something like kafkaJS library in AC-  But I understand that there would be architectural limitations for the AC module to create a socket connection operation eg as documented: https://kafka.apache.org/0100/protocol.html#:~:text=Kafka%20uses%20a%20binary%20protocol,as%20reques....

This leaves us with either uncovering a secret workaround with the AC API (per above), or building a seperate node.js application to read and update prepared broadlog tables for this custom channel.

Any feedback?  I can imagine this will become quite a topic given kafka's popularity and the need to diseminate AC-generated deliveries via this message channel.

S

[#8]

Hi @Sukrity_Wadhwa , any potential update here?

We would otherwise proceed to leverage an HTTP-Kafka proxy as an additional application required in the martech stack, but highlights an AC weakness for this integration method.  Would be great to have an engineering response to @Darren_Bowers  point about it already communicating kafka sockets internally with other adobe products.  Thanks!