Product Functional Testing is failing in PROD deployment pipeline

[pnagy]

Hi,

there is a failing test in our product functional testing when trying to deploy to PROD env:

2024-10-08 08:23:38,714 [main] ERROR com.adobe.cq.cloud.testing.it.smoke.rules.ContentPublishRule - PAGE_AVAILABLE: Failed to check the page /content/test-site/testpage_af1724ac-ab7b-44f3-a1f2-224ebe190c83.html via the AEM publish ingress (expected status 404). Please ensure that the CDN and Dispatcher configurations allow fetching the page.
com.adobe.cq.cloud.testing.it.smoke.exception.PublishException: PAGE_AVAILABLE: Failed to check the page /content/test-site/testpage_af1724ac-ab7b-44f3-a1f2-224ebe190c83.html via the AEM publish ingress (expected status 404). Please ensure that the CDN and Dispatcher configurations allow fetching the page.
at com.adobe.cq.cloud.testing.it.smoke.rules.ContentPublishRule.getPublishException(ContentPublishRule.java:210)
at com.adobe.cq.cloud.testing.it.smoke.rules.ContentPublishRule.checkPage(ContentPublishRule.java:200)
at com.adobe.cq.cloud.testing.it.smoke.rules.ContentPublishRule.checkPage(ContentPublishRule.java:141)
at com.adobe.cq.cloud.testing.it.smoke.rules.ContentPublishRule.deactivateAssertPublish(ContentPublishRule.java:244)
at com.adobe.cq.cloud.testing.it.smoke.PublishEndToEndIT.activateAndDeactivate(PublishEndToEndIT.java:98)
...
Caused by: org.apache.sling.testing.clients.exceptions.TestingValidationException: Expected HTTP Status: 404 . Instead 200 was returned!

...
2024-10-08 08:23:44,822 [INFO]
2024-10-08 08:23:44,822 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0
2024-10-08 08:23:44,822 [INFO]
2024-10-08 08:23:44,836 [INFO] ------------------------------------------------------------------------
2024-10-08 08:23:44,836 [INFO] BUILD SUCCESS
2024-10-08 08:23:44,836 [INFO] ------------------------------------------------------------------------
2024-10-08 08:23:44,910 [INFO] Total time: 21:07 min

All the other tests are succeeding and the pipeline was succeeding previously.
The changes - that could be relevant - are:
- added the file opt-in/USE_SOURCES_DIRECTLY to move from legacy to flexible Dispatcher configuration

- added configuration to CDN with skipCache action.

Even the Product Functional Testing tests were running as

Average duration*

3 minutes 17 seconds

*Average calculated over past 90 days across 6 successful step executions.
, but the failing run last for 22+ mins.

Can you suggest how to troubleshoot the issue?

Thanks,
Peter

[h_kataria · Accepted solution]

Could be a temporary glitch since these are OOB tests which the pipeline runs. Looking at the error, it seems it was the deactivation test which didn't work probably. The test tried to deactivate the page /content/test-site/testpage_af1724ac-ab7b-44f3-a1f2-224ebe190c83.html and then tried to retrieve it and expected a 404 but instead received a 200. If possible, cross check once if this page got deactivate or still present on your publish. 
If it is present not present on publish, then try disabling the skipCache option once maybe to rule out some caching interference.
Hope this helps.

[narendiran_ravi]

We faced a similar issue due to load in the distribution queue. I recommend checking the author distribution queue to see if the deactivation request is still present. If the queue has more items, processing these requests can cause delays or failures.