CSRF Token | Intermittent Empty Response | Encapsulated Token Support

[Himanshu_Phulara]

Background
Customer is on AEM 6.5 on-prem and trying to enable the CSRF token and accessing the CSRF token via dispatcher. The token is sometimes generated correctly and sometimes gives a blank response. - The issue is only happening on UAT environment , where there are 2 publishers and 4 dispatchers attached. When we disabled other 3 dispatchers and publishers and checked the scenario with only 1 publisher and 1 dispatcher , then there is no issues and the token is generated successfully at all times. Only when all publishers/dispatchers are enabled the CSRF token is returned blank intermittently. 
We have verified, customer has put
1.  granite.csrf.standalone dependency in the required templates
2. Dispatcher configurations for CSRF support has been added for both filter and cache rules
3. HMAC Key has been replicated from author to the 2 publishers. 

To fix this issue we tried enabling the Encapsulated Token Support on UAT Publish 1. After that the CSRF token was generated successfully at all time while accessing via the dispatcher, but after sometime the Publish 1 instance got corrupted and below error was noted : 

java.io.IOException: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record f097b81997116c7596eb2b7d88448f515808e49db4082dd908886cb535ecefdb does not exist. 

Customer recreated Publish 1 instance and again configured the  Encapsulated Token Support. Again the instance got corrupted. 
Questions: 
1. Intermittently CSRF token is coming blank while accessing via dispatcher url on UAT environment , where there are multiple publishers and dispatchers. With single publish and dispatcher the CSRF token is generated successfully at all times. What could be the fix to this ?
2. The issue seemed to be fixed when we enabled Enapsulated token support on UAT publish1, but the instance got corrupted after sometime.(Error mentioned above) What could be done to fix this if this is the right approach?

[Shashi_Mulugu]

@Himanshu_Phulara have you checked the request flow when empty response is received? Is it getting served from dispatcher or publisher? Is it any consistent pattern of one publisher serving empty response or is it random? What if we hit same publisher directly during the same time.

[Lokesh_Vajrala]

@Himanshu_Phulara 

I guess one way to fix the issue would be to use the 1:1 dispatcher and publish mapping and with this approach you can configure the sticky sessions, which ensures that the token retry is always successful. 

With the existing setup (2 publish & 4 dispatchers), is there any user sync running on the publish instances? If yes, disable the sync and test the tokens.

Thanks

Lokesh 

[Himanshu_Phulara]

Hi Lokesh,
With sticky connections although a user would always be directed to the same publish instance, As a consequence, truly optimal load balancing is not possible. In case a publish instance becomes unavailable, all the users authenticated on that instance will lose their session.

[Tanika02 · Accepted solution]

Hello @Himanshu_Phulara - 

The issue you're facing with the CSRF token and the intermittent blank response could be related to the configuration and setup of your AEM environment. Here are some suggestions to address your questions:

1. Fixing Intermittent Blank CSRF Token:

• Double-check the Dispatcher configurations for all dispatchers to ensure that the CSRF support has been correctly added in both filter and cache rules.
• Validate that the Dispatcher caching is properly configured to avoid any caching conflicts that might affect the CSRF token generation.
• If you're using Akamai/Fastly, review its settings to ensure that it is correctly forwarding requests to the appropriate publishers and dispatchers. (double-check through logs once)
• Also could you please check/share the Dispatcher logs for any relevant errors/warnings that might give more highlights on the intermittent blank issue?

2. Encapsulated Token Support and Instance Corruption:

• Double-check the configuration of Encapsulated Token Support on UAT Publish 1 to ensure that it is properly set up according to Adobe's documentation & best practices.
• Analyze the logs of the corrupted instance to identify any specific errors or issues that occurred after enabling Encapsulated Token Support. Look for any potential conflicts or misconfigurations that might have led to the instance corruption.
• If enabling Encapsulated Token Support consistently leads to instance corruption. You may want to explore alternative approaches or configurations that provide a stable CSRF token generation.