Suggest best practice to save 3rd party webhook requests timestamp details with in AEM 6.5 to avoid older request from processing.

[Rajumuddana]

Hi All,

I have an use case where a 3rd party system make a post call to AEM author and post a payload (this is webhook scenario).

Based on the timestamp of the request, I should ignore older requests.

For this I’m thinking to store under /content to verify timestamp like below,

/content/abc/en

                      /webhooks

                        /webhook-id

                         -timestamp - Date - <time stamp value>

So that, whenever requests comes in, i can fetch respective node and compare timestamps.

If that is latest one process the request & update the timestamp.

If that is older than existed timestamp will ignore.

Please suggest me the best practice in this case.

Thanks,

Raju.

[arunpatidar]

Hi @Rajumuddana 

It depends on what webhook/request payload used for

• Content-related payload: If the payload affects the creation or update of content, pages, or assets, it's appropriate to store the timestamp directly as a property(e.g. webhook-sync-timestamp) in the related content node (e.g., under /content/abc/en).
• Non-content-related payload: If the payload is not directly related to content, storing timestamps and other webhook-related data under /var is more appropriate. The /var directory is designed for storing system-related or process-related data that isn’t directly tied to the content hierarchy. Example

  /var/webhooks/abc/en/webhook-id
    - timestamp - Date - <time stamp value>
  ​

[Rajumuddana]

Hi @arunpatidar ,

Thanks for the response.

Yes, it is related to Content. I'm attaching sequence diagram for better understanding. Please take a look.

[arunpatidar]

Hi @Rajumuddana 
If you are not getting many events to Author within a time period then you can add timestamp in the Article page properties.

The only drawback I can see with storing this data in page properties if you are running too many/frequent queries for filtering out outdated pages.

However If you are already running a query to find out pages based on article id then adding timestamp in the query will optimize this.

[Rajumuddana]

Hi @arunpatidar 

Thanks for your reply.

Here, the catch is, webhook payload comprises of timestamp & list of Article IDs(payload).

I no need to validated the time stamp of each Article page rather timestamp of overall POST request.

So, I was thing to store the timestamp details independently from actual Article pages, based on Webhook ID (though the webhook requests are multiple, the IDs from the requests trigger are limited).

Here is the statement from my 3rd party system-

Check timestamps

A series of entity updates during a short time interval will result in the same Webhook firing multiple times. We do not guarantee that your external system will receive these Webhooks in the exact same order of those updates that were performed. The triggered_at timestamp in the payload's metadata indicates when the event was triggered:

{
    "metadata": {
        "triggered_at": "2023-07-19T09:05:13.000000Z",
        "webhook_id": "abc123"
    },
    "payload": {}
}

Your system should ignore any webhooks with a timestamp that is earlier than those already processed.

Thanks,

Raju.

[arunpatidar]

Hi @Rajumuddana 
I think you can store webhook metadata at page property level as well, this can be used for auditing as well.

And change your logic to retrieve article

1. SQL2 query to fetch article page, filter by page type and timestamp.

2. Filter the list based on matched payload article, apply changes and update timestamp

To handle the concurrency, you may need to use threads and process status etc.

If you don't want real time article update then you can temporary store the webhook payload in /tmp and and proceed as batch after every 30 min and so on.

[Rajumuddana]

Hi @arunpatidar ,

Thanks for the explanation.

But i still didn't get the point of storing webhook request timestamp in each article page instead in webhook id page (as i mentioned above).

• I have limited webhooks vs numerous article pages.
• And the timestamp is belongs to webhook but not page, technically.

Thanks,

Raju.

[arunpatidar · Accepted solution]

Hi @Rajumuddana 
Creating a page solely for storing data is not the optimal solution; instead, you can create nodes under `/var`.