org.apache.kafka.common.errors.TimeoutException: AEMaaCS with External Kafka | Community
Skip to main content
HeenaMadan
Community Advisor and Adobe Champion
HeenaMadanCommunity Advisor and Adobe Champion
Community Advisor and Adobe Champion
March 29, 2022

org.apache.kafka.common.errors.TimeoutException: AEMaaCS with External Kafka

  • March 29, 2022
  • 1 reply
  • 7903 views

Hi Team,

 

I have requirement to integrate kafka with AEM. On my local, it is working fine.(Using SDK) but on AEM cloud, getting below error while pushing records to kafka topic:

org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-topic-0:120000 ms has passed since batch creation.

where:

my-topic - kafka topic name

Kafka is installed on Azure kubernetes.

All requests are allowed right now.

All advance networking configurations are fine.

Installed kafka version and jars have same version(3.0.0)

kafka cluster id is also printing in logs but issue while sending data to producer.

 

Followed links for adv networking:

https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/security/configuring-advanced-networking.html#dedicated-egress-IP-address

 

Please advise what is the issue here?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

joerghoh
Adobe Employee
joerghohAdobe Employee
Adobe Employee
March 29, 2022

it still sounds like a networking problem.  From the documentation, even "Flexible Port Egress" should be sufficient to allow you to any remote system. 

How have you configured advanced networking and are you sure that Kafka allows you to connect from AEM CS?

    HeenaMadan
    Community Advisor and Adobe Champion
    HeenaMadanCommunity Advisor and Adobe ChampionAuthor
    Community Advisor and Adobe Champion
    June 13, 2022

    Advance networking is correct but still getting above timeout error. No Network issue as per Adobe Support team. Now i am using kafka rest api ( kafka on AKS using Strimzi bridge) and able to push data to Kafka topic.

     

      M
      MateuszIw
      June 11, 2024

      Hello @heenamadan  have you resolved your issue? - we are facing the same problem in our implementation. AEM cloud, VPN access to the Client, connection to all others system on Client side works, with Kafka we are connecting to the Topic but after few minutes we are getting: 

      11.06.2024 10:19:47.937 [XXX] *INFO* [look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1] org.apache.kafka.clients.NetworkClient [Consumer clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1-consumer, groupId=look] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 16237 ms. 11.06.2024 10:19:47.937 [XXX] *WARN* [look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1] org.apache.kafka.clients.NetworkClient [Consumer clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-StreamThread-1-consumer, groupId=look] Bootstrap broker 57.57.0.32:9092 (id: -1 rack: null) disconnected 11.06.2024 10:19:48.929 [XXX] *INFO* [kafka-admin-client-thread | look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] org.apache.kafka.clients.NetworkClient [AdminClient clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] Disconnecting from node -1 due to socket connection setup timeout. The timeout value is 16945 ms. 11.06.2024 10:19:52.334 [XXX] *INFO* [kafka-admin-client-thread | look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] org.apache.kafka.clients.admin.internals.AdminMetadataManager [AdminClient clientId=look-1485a3f3-6a2b-4350-a91c-02505cda9cdd-admin] Metadata update failed org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata

       Any idea?

      Terms & ConditionsAccessibility statement

      Loading footer...