AEM - Issues with Cache Flush in AEM As Cloud | AEM As Cloud - Content Not Refreshing After Publish

[VeenaVikraman]

Hi All,

We're facing a challenging issue with our AEM as a Cloud project. Our caching had been working correctly for a long time, but recently we've noticed that content isn't refreshing after publishing or even after the max-age is reached. Below are our findings from the debugging we've done so far.

Additionally, our dispatcher cache settings haven't been changed in a very long time.

1. As per the documentation https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/implementing/con... , our vhost configurations appear to be correct.

2. The CDN cache shows a MISS upon reaching max-age, but the content retrieved from the server remains outdated. This suggests potential caching within the Fastly CDN or a persistent Dispatcher cache.

3. Adding the Surrogate-Control header (as documented by Fastly  https://www.fastly.com/documentation/reference/http/http-headers/Surrogate-Control/ ) to bypass the Fastly cache yielded mixed results. While the initial requests retrieved the updated content, subsequent attempts did not.

4. We removed all caching headers from the Dispatcher files. As per the AEM documentation [https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/implementing/con.... ], this should trigger the default private cache behavior (with a max-age of 600 seconds). However, the browser cache still reflected a max-age of 300 seconds, and content updates weren't reflected after 300 seconds.

5. Bypassing the Dispatcher cache entirely via URL confirmed that the content is indeed published and available on the Publish server.

We're at a loss for the cause of this issue. Any guidance on further debugging or resolving this problem would be greatly appreciated.

P.S. Due to the client's strict access policies, I am unable to access Cloud Manager for further investigation.

Thanks in Advance 

Veena ✌

[aanchal-sikka]

@VeenaVikraman 

If possible enable debug logs on dispatcher. That would really help !

• Please assure s-max-age is not used in Cache-Control. Use only Surrogate-control. 
• Please check if the Dispatcher Flush is configured the pipeline after deployment: https://experienceleague.adobe.com/en/docs/experience-manager-cloud-manager/content/using/pipelines/...
  • If not configured, the headers from old cache might still be effective.
• If possible implement SCD to flush Dispatcher on Demand
• If you use Adobe provided Fastly CDN, then you can flush CDN on demand http://experience-aem.blogspot.com/2023/01/aem-cloud-service-invalidate-dispatcher-purge-fastly-cdn-...

For detailed analysis via headers I had authored a blog some time ago: https://techrevel.blog/2023/09/05/exploring-aem-request-and-response-headers-analysis-of-browser-cdn...

[VeenaVikraman]

Thanks @aanchal-sikka . We do not have access to Cloud Manager. But I will pass this info to the team who can check that. We do have s-max-age as we have another CDN infront of Fastly. Why do you think we should not use s-max ?

Also regarding 

• ---

  If possible enable debug logs on dispatcher. That would really help !

  • Please assure s-max-age is not used in Cache-Control. Use only Surrogate-control. 
  • Please check if the Dispatcher Flush is configured the pipeline after deployment: https://experienceleague.adobe.com/en/docs/experience-manager-cloud-manager/content/using/pipelines/...
    • If not configured, the headers from old cache might still be effective.
  • If possible implement SCD to flush Dispatcher on Demand
  • If you use Adobe provided Fastly CDN, then you can flush CDN on demand http://experience-aem.blogspot.com/2023/01/aem-cloud-service-invalidate-dispatcher-purge-fastly-cdn-...

  For detailed analysis via headers I had authored a blog some time ago: https://techrevel.blog/2023/09/05/exploring-aem-request-and-response-headers-analysis-of-browser-cdn...

  ---

  The content is properly flush when deployment happens. But we see issue in the flush only when the content activation is happening

[aanchal-sikka]

@VeenaVikraman 

As per the documentation, "Use the Surrogate-Control header to control CDN caching independent from browser caching. 

Since, you have custom & Fastly CDN, you might need both:

• s-max-age (as per the custom CDN)
• surrogate-control (Fastly will ignore s-max-age, when surrogate-control is present). 

Both, CDN need to be flushed to assure that latest content is fetched from dispatcher. 

Adobe can flush entire CDN on request. They can also share a curl command for resource-based flush.

[aanchal-sikka]

@VeenaVikraman 

Also, close the browser before checking the cache again. Even incognito returns from local cache.

[VeenaVikraman]

Saw this comment in my linkedIn post https://www.linkedin.com/feed/update/urn:li:activity:7191663266912206850?commentUrn=urn%3Ali%3Acomme... . If so then this seems like a dispatcher issue and we may need to raise an Adobe ticket. 

[aanchal-sikka]

Thanks for sharing @VeenaVikraman .

Please do share the update with us. We have a similar set-up in one of the projects and no issues have been reported. May be a specific configuration/use-case. Would be good to know.

[VeenaVikraman]

In our case , when the s-maxage is reacehd , it is MISS -ing the CDN cache and same with Fastly. Ideally then it is pulling content from Dispatcher . But I cannot access the dispatcher URL directly to verify if the cache is gettinf flushed there. 

[aanchal-sikka]

@VeenaVikraman 

As per my knowledge, we can access publish only via dispatchers. Direct access to publish is not available. Example: https://author-p46369-e128324.adobeaemcloud.com/ is via dispatcher. Do you have access to different URLs?

[Meenapl]

@VeenaVikraman In addition to enable Dispatcher log,  good to see if the cache folder in dispatcher is configured with necessary permissions even then flush can't happen appropriately.