Fastly CDN not caching the content properly

[Luca_Moreau]

Hi all,

We are using 6.5 on-prem with a Fastly CDN in front of our Apache Dispatcher. Despite setting proper cache headers in our dispatcher.any and Apache config, some pages are not being cached by Fastly (we verified using their cache inspection tools).

Here is a simplified example of what we send from Apache (verified with curl -I):

HTTP/1.1 200 OK
Content-Type: text/html
Cache-Control: max-age=300, public
X-Cache: MISS
X-Served-By: cache-bom4743-BOM

Apache configuration:
Header set Cache-Control "max-age=300, public"

Even with Cache-Control: max-age=300, public, Fastly often bypasses cache (X-Cache: MISS) or marks the response as unavailable for caching.

Are there any specific headers (like Surrogate-Control or Age) that I should explicitly set—alongside Cache-Control - to ensure Fastly caches AEM content properly, and could AEM Dispatcher’s /cache rules conflict with Fastly’s caching behavior?

[SantoshSai · Accepted solution]

Hi @Luca_Moreau,

I think you should explicitly set both Cache-Control and Surrogate-Control headers from Apache (Dispatcher layer). as per my understanding, Fastly relies primarily on Surrogate-Control for edge caching decisions.

Could you try setting the following headers?

Header set Cache-Control "max-age=300, public"
Header set Surrogate-Control "max-age=600"

• Cache-Control: tells the browser to cache for 5 minutes.

• Surrogate-Control: tells Fastly to cache at the edge for 10 minutes.

Without Surrogate-Control, Fastly may ignore Cache-Control or treat your response as uncacheable.

For more details: https://docs.fastly.com/en/guides/controlling-caching

Hope that helps!

[AmitVishwakarma]

Hi @Luca_Moreau ,

Fastly does not rely solely on Cache-Control to determine whether to cache. If Surrogate-Control is missing or not set correctly, Fastly may bypass caching even when Cache-Control: public is present.

Try below step's:

Step 1: Set the Right Headers in Apache (Dispatcher Layer)

Add these headers to Apache's config (dispatcher_vhost.conf or equivalent):

# For browsers
Header set Cache-Control "public, max-age=300, s-maxage=600"

# For Fastly
Header set Surrogate-Control "max-age=600"

# Optional: Explicitly mark as cacheable at all levels
Header unset Pragma
Header unset Expires

Notes:

  - s-maxage=600 instructs shared caches (like Fastly) to cache for 10 minutes.

  - Cache-Control with public ensures browser + intermediate cache eligibility.

  - Surrogate-Control is Fastly’s preferred caching header.

  - Pragma and Expires can override caching if set by AEM; remove them to avoid conflicts.

Step 2: Confirm Fastly is Not Ignoring the Response

Use curl -I or Fastly's Real-Time Log Tailing and check the headers:

curl -I https://www.example.com/page.html

You should see:

Cache-Control: public, max-age=300, s-maxage=600
Surrogate-Control: max-age=600
X-Cache: HIT

If you see X-Cache: MISS consistently, continue to Step 3.

Step 3: Check Your Fastly VCL or Configuration

Fastly by default respects Surrogate-Control, but your VCL or settings might include conditions that cause a bypass:

  - Login to Fastly UI => Go to your service.

  - In Caching rules, ensure:

        - The origin (your Apache Dispatcher) is not marked as "pass".

        - There's no custom condition that disables caching for HTML files or specific URL patterns.

Step 4: Check Dispatcher Cache Rules (dispatcher.any)

Make sure dispatcher.any allows caching for the pages in question:

/cache
{
    /rules
    {
        /0001 { /glob "*" /type "allow" }
    }
    /statfileslevel "2"
    /docroot "/path/to/publish/docroot"
    /allowAuthorized "0"
    /headers
    {
        "Cache-Control"
        "Surrogate-Control"
    }
}

Ensure your /rules are not accidentally blocking specific paths or file types.

Make sure dispatcherFlush is working correctly after content changes.

Step 5: Add Diagnostic Headers (Optional but Helpful)

Add these headers to trace Fastly behavior:

Header always set X-Debug-Fastly "Surrogate-Control seen"
Header always set X-Cache-Status "%{Fastly-Cache-Status}e"

You’ll be able to see in responses:

  - If Fastly saw the Surrogate-Control

  - Whether the cache status is HIT / MISS / PASS

Regards,
Amit