[Cloud Manager - AEMAACS] deployment pipeline fails on deploy step (1 of 10 deployments runs through)

[mwahlmueller]

I am trying to deploy my AEM Cloud repo, but it always fails on the cloud pipeline while the exact same branch works in 1 of 10 cases on the exact same pipeline.

Here the Deploy step log of succesful deployment:

2023-06-15T16:30:14+0000 Summary of events during the deployment step:
2023-06-15T16:30:14+0000 Begin deployment in aemaacs-dev-backup [CorrelationId: 39444825]
2023-06-15T16:30:21+0000 Update author indexes job has started.
2023-06-15T16:33:03+0000 Update author indexes job has finished successfully.
2023-06-15T16:33:04+0000 Update publish indexes job has started.
2023-06-15T16:33:13+0000 Update publish indexes job has finished successfully.
2023-06-15T16:36:11+0000 Build Transform job has started.
2023-06-15T16:36:11+0000 Build Transform job has finished successfully.
2023-06-15T16:48:01+0000 Install mutable content job has started.
2023-06-15T17:34:34+0000 Install mutable content job has finished successfully.
2023-06-15T17:34:37+0000 Finished deployment in aemaacs-dev-backup

And here of the not successful deployment with exact same branch (fails 9/10 times currently)

2023-06-13T16:59:41+0000 Summary of events during the deployment step:
2023-06-13T16:59:41+0000 Begin deployment in aemaacs-dev-backup [CorrelationId: 39336215]
2023-06-13T16:59:48+0000 Update author indexes job has started.
2023-06-13T17:02:34+0000 Update author indexes job has finished successfully.
2023-06-13T17:02:35+0000 Update publish indexes job has started.
2023-06-13T17:02:45+0000 Update publish indexes job has finished successfully.
2023-06-13T17:05:41+0000 Build Transform job has started.
2023-06-13T17:05:41+0000 Build Transform job has finished successfully.
2023-06-13T18:06:00+0000 Failed deployment in aemaacs-dev-backup

It seems to fail installing the mutable content, the pipeline only says that we should get in touch with Adobe support (where we wait for answer currently):

I download the "Deploy to Dev" error log, but there is no issue that really that stands up to me.

The package builds fine, but just cannot deploy even after multiple tries, and the error log does not help. 

Any idea on what the issue might be, or how to troubleshoot/debug this would be much appreciated.

Anyone else also having this issue sinca about one week with Cloud Manager?

We face the issues since June 5th

Thanks for you help and Kind regards,

Martin Wahlmüller

[aanchal-sikka]

@mwahlmueller Same error for us as well.

Please add the resolution here, if Adobe suggests anything

[mwahlmueller]

@aanchal-sikka Adobe provided a temporary fix for it and increased the traversal limit of queries to not run into an exception thrown by Oak

Exception in your logs could look like this:

> org.apache.sling.jcr.resource.internal.helper.jcr.JcrNodeResourceIterator seek: Problem creating Resource for next node, skipping
> org.apache.jackrabbit.oak.query.RuntimeNodeTraversalException: The query read or traversed more than 100000 nodes. To avoid affecting other tasks, processing was stopped.

The pipeline startup failure was due to the traversal of a large number of nodes.

Permanent fix for the issue will be provided, but is not yet in place. There is no confirmed release date for the fix announced. 

Kind regards,

Martin

[A_H_M_Imrul]

Hello mwahlmueller,

I faced a similar issue once, though there was no clear root cause of it at that point of time. Later I figured out that, there was a maintenance window of Adobe during the deployment period. I would suggest to keep an eye here: https://status.adobe.com/, just to plan accordingly.

Thanks