Guaranteed Replication Status

[sreenivasula]

Hi All,

        We have a publishing system called Relay and a middle ware rabbit MQ and a staging system AEM author and many public facing AEM publishers (6 publishers).

Content is authored in Relay and sent to AEM author for staging. Once the authors verify in Relay that content is fine. Relay sent a message(rabbit MQ message) to AEM author to replicate content to AEM publishers. We have a work flow which triggers the replication.

We have to sent a reply message to Relay through rabbit MQ if the content is successfully replicated to any one of the AEM publisher.

Please suggest the best way to check whether the content is successfully published in any one of the publisher ?

Regards

Sreeni

[Paul_McMahon · Accepted solution]

There are probably a bunch of ways to handle it. 

The simplest way - assuming you have store all the information that is needed by Relay in the content you could write a replication event listener that runs on the publish instances and whenever it receives something it sends the response to Relay. That assumes that Relay will ignore all the duplicate responses since it will get 6 responses. You could set up the replication event listener to only run on one publish server - but then if that one was down you would never get any responses to Relay. 

You could have the replication event listener send a message back to author. You would have to write a servlet that ran on author and accepted the message from the publish servers. This servlet would then send the message to Relay. The servlet would have to keep track of which activation had been confirm with Relay and which not so that would prevent the duplicate messages to Relay. 

You could have a step in the workflow after publication that had some sort of wait mechanism in it and have that poll replication queue logs to determine if the activation was successful and have it send the message to relay. That's going be suboptimal because the polling mechanism will be completely custom - I'd tend to stay away from this option.

If really wanted to go all out you could write and extension of the default replication agent's  and build the logic to send the relay message after it removes something from the queue. You'd have to make it smart enough to track whether or not another agent had already sent it but this would work. I'd tend to stay away from this as well unless you are very comfortable with the APIs and interfaces around replication agents. 

There are probably a bunch of other variations on this, but they will all boil probably down to one of three concepts:

• A replication event listener on publish servers that somehow triggers the Relay message
• Some sort of polling mechanism that watches the replication queues
• An extension of the default replication agent. 

I'd tend to work with some based of off a replication event listener in publish but that's personal opinion. 

[sreenivasula]

Thanks a ton for your quick response orotas!!. We have the below constraints on the suggested approaches.

• A replication event listener on publish servers that somehow triggers the Relay message --- Due to security reasons, we have disabled all the replication agents on all publishers
• Some sort of polling mechanism that watches the replication queues --- its not an option, because of performance issues and huge number of concurrent publication requests.
• An extension of the default replication agent. --- Please elaborate on this option, considering the concurrent publication requests/messages from Relay to AEM author for replication to AEM publishers.

Thanks in advance.

Sreeni

[Paul_McMahon]

Your first constraint shouldn't impact the proposed concept. A replication event listener can listen for either sent or receive events. So the replication event listener would sit on the publish servers and listen for replication received events - which are happening in your design. A replication event listener does not require a replication agent. Now if you security restriction is that you can't have any outgoing network traffic from your publish servers than you would have an issue, but turning off the replication agents on publish wouldn't prevent you for following this approach. 

The option suffers from the same problem of of these options have - somewhere you have keep track of which activation you have confirmed to Relay already so you can ignore the 5 events after the first 1. So in the case of the replication agent design you have come up with a thread safe way to storing the status of each activation from Relay so that you only confirm to Relay once per event. So somewhere (either in the JCR or in an external data store) you keep track of every activation request from Relay with either confirmed or waiting status. Then you extend the default replication agent so that after a successful replication it checks this log and if the status of the corresponding Relay message is waiting it changes the status and then sends the message. When all the other replication agents finish with the request activation they check the status and see that is is confirmed and do nothing. There are two challenging aspects - the replication agent API and figuring out how to track the status per activation in a thread safe way. You also have to worry about upgrade/hotfix implications of extending the default replication agent. This is a core piece of the product and by extending if you put yourself in a position of potentially breaking the functionality during an upgrade. 

[sreenivasula]

Thanks Orotas !! Yes, our security constraint is no outgoing network traffic from publish servers.

For the replication agent customization, do you have some samples or high level steps for me to take it forward, as i am not very strong in these API.

Thanks in Advance

Regards

Sreeni