org.apache.kafka.common.errors.TimeoutException: AEMaaCS with External Kafka

[HeenaMadan]

Hi Team,

I have requirement to integrate kafka with AEM. On my local, it is working fine.(Using SDK) but on AEM cloud, getting below error while pushing records to kafka topic:

org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-topic-0:120000 ms has passed since batch creation.

where:

my-topic - kafka topic name

Kafka is installed on Azure kubernetes.

All requests are allowed right now.

All advance networking configurations are fine.

Installed kafka version and jars have same version(3.0.0)

kafka cluster id is also printing in logs but issue while sending data to producer.

Followed links for adv networking:

https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/security/configurin...

Please advise what is the issue here?

[Jörg_Hoh]

it still sounds like a networking problem.  From the documentation, even "Flexible Port Egress" should be sufficient to allow you to any remote system. 

How have you configured advanced networking and are you sure that Kafka allows you to connect from AEM CS?

[HeenaMadan]

Advance networking is correct but still getting above timeout error. No Network issue as per Adobe Support team. Now i am using kafka rest api ( kafka on AKS using Strimzi bridge) and able to push data to Kafka topic.

[MateuszIw]

Hello @HeenaMadan  have you resolved your issue? - we are facing the same problem in our implementation. AEM cloud, VPN access to the Client, connection to all others system on Client side works, with Kafka we are connecting to the Topic but after few minutes we are getting: 

11.06.2024 10:19:47.937 [XXX] *INFO* [look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1] org.apache.kafka.clients.NetworkClient [Consumer clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1-consumer, groupId=look] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 16237 ms.
11.06.2024 10:19:47.937 [XXX] *WARN* [look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1] org.apache.kafka.clients.NetworkClient [Consumer clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1-consumer, groupId=look] Bootstrap broker 57.57.0.32:9092 (id: -1 rack: null) disconnected
11.06.2024 10:19:48.929 [XXX] *INFO* [kafka-admin-client-thread | look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] org.apache.kafka.clients.NetworkClient [AdminClient clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 16945 ms.
11.06.2024 10:19:52.334 [XXX] *INFO* [kafka-admin-client-thread | look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] org.apache.kafka.clients.admin.internals.AdminMetadataManager [AdminClient clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata

 Any idea?