Investigating

Cloud Manager : Add emergency production deployment pipeline

Seung-ju · 2024-06-14T00:01:37+00:00

Request for Feature Enhancement (RFE) Summary: Cloud Manager Deployment: Add emergency production deployment pipeline Use-case: LGE.com is trying to roll back the deployment after a successful production deployment. Currently, the only way to roll back is to re-deploy the code in Cloud Manager, which takes 2 to 3 hours. Given that they are supporting 24 websites (with more to come in the future), they need to reduce the build time for any rollback operations. The best way to achieve this is to enable re-deployment using an earlier stable built image. Current/Experienced Behavior: The production deployment is completed. After the deployment, the customer discovered blocker issues on one of the 24 websites. They decided to roll back the deployment. They reverted the change that caused the issue and pushed the revised changes to Cloud Manager. They restarted the Stage/Prod build, which took 2 to 3 hours. The customer is trying to minimize this 2 to 3-hour downtime as much as possible. The current production deployment pipeline is as follows: 1.Stage Deployment(77m) validation → build & unit testing(38m) → code scanning(14m) → build images(10m) → deploy to stage(15m) code scanning(14m) : code coverage(42%) 2.Stage Testing(5m) product functional testing(3m) → custom functional testing(2m) → custom ui testing → experience audit custom functional testing : Only library downloads required for testing, no actual testing 3.Production Deployment schedule production deployment → deploy to production Improved/Expected Behavior: The basic idea is an expedited deployment pipeline is a staged deployment after production deployment. Deployment using the emergency production deployment pipeline begins. first task : Production Deployment Only build (Excluding unit tests)→ build images → deploy to production second task : Stage Sync deploy to stage Since testing and code verification pipelines exist in QA or DEV pipelines and are verified by our deployment team, we believe that this task is unnecessary in production pipelines. Environment Details (AEM version/service pack, any other specifics if applicable): AEM as a Cloud (2024.4.16145.20240430T082417Z) Customer-name/Organization name: LG Electronics, Global HQ Korea 24 subsidiaries (UK,DE,PE,TR,JP,FR,SG,CO,SA,SA_EN,CA_FR,CA_EN,KZ,ID,IT,CL,TH,PT,IN,HK,HK_EN,BR ,MX ,PA) Screenshot (if applicable): Code package (if applicable):

Forum|Forum|1 year ago
June 14, 2024
10 replies
1666 views

Request for Feature Enhancement (RFE) Summary:	Cloud Manager Deployment: Add emergency production deployment pipeline
Use-case:	LGE.com is trying to roll back the deployment after a successful production deployment. Currently, the only way to roll back is to re-deploy the code in Cloud Manager, which takes 2 to 3 hours. Given that they are supporting 24 websites (with more to come in the future), they need to reduce the build time for any rollback operations. The best way to achieve this is to enable re-deployment using an earlier stable built image.
Current/Experienced Behavior:	The production deployment is completed. After the deployment, the customer discovered blocker issues on one of the 24 websites. They decided to roll back the deployment. They reverted the change that caused the issue and pushed the revised changes to Cloud Manager. They restarted the Stage/Prod build, which took 2 to 3 hours. The customer is trying to minimize this 2 to 3-hour downtime as much as possible. The current production deployment pipeline is as follows: 1.Stage Deployment(77m) validation → build & unit testing(38m) → code scanning(14m) → build images(10m) → deploy to stage(15m) code scanning(14m) : code coverage(42%) 2.Stage Testing(5m) product functional testing(3m) → custom functional testing(2m) → custom ui testing → experience audit custom functional testing : Only library downloads required for testing, no actual testing 3.Production Deployment schedule production deployment → deploy to production
Improved/Expected Behavior:	The basic idea is an expedited deployment pipeline is a staged deployment after production deployment. Deployment using the emergency production deployment pipeline begins. first task : Production Deployment Only build (Excluding unit tests)→ build images → deploy to production second task : Stage Sync deploy to stage Since testing and code verification pipelines exist in QA or DEV pipelines and are verified by our deployment team, we believe that this task is unnecessary in production pipelines.
Environment Details (AEM version/service pack, any other specifics if applicable):	AEM as a Cloud (2024.4.16145.20240430T082417Z)
Customer-name/Organization name:	LG Electronics, Global HQ Korea 24 subsidiaries (UK,DE,PE,TR,JP,FR,SG,CO,SA,SA_EN,CA_FR,CA_EN,KZ,ID,IT,CL,TH,PT,IN,HK,HK_EN,BR ,MX ,PA)
Screenshot (if applicable):
Code package (if applicable):

Seung-juAuthor

Related to the rollback idea below.

- https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager-ideas/cloud-manager-enabling-rollback-from-previous-build-snapshots/idi-p/675082

If you find it difficult to apply the rollback idea quickly, please consider this idea as well.

arunpatidar

Community Advisor

Hi @seung-ju
Adobe pipeline already has emergency mode, that skip all the functional & performance test.

Code build & code quality and deployment on stage(non-prod) is essential step, which can be done anytime without waiting for Go to PROD, once you have a go, you just deploy on PROD using emergency mode.

but based on pain points of various customer this can be improved further.

Arun Patidar

Seung-juAuthor

Hi @arunpatidar

Thanks for your reply.

Can you tell me how to set up pipeline for emergency mode?

It seems that stage deployment cannot be excluded in the production pipeline we use.

I look forward to your reply.

arunpatidar

Community Advisor

Hi @seung-ju

When starting a production pipeline execution, if the emergency pipeline execution mode feature has been activated(by Adobe CSE) for the program, you can start the execution in either normal or emergency mode from a dialog box.

https://experienceleague.adobe.com/en/docs/experience-manager-cloud-manager/content/using/code-deployment#emergency-pipeline

Arun Patidar

Seung-juAuthor

Hi @arunpatidar

I'll check. thank you very much.

cc @kwangmin11

K

kwangmin11

Adobe Employee

Hi @arunpatidar ,

The provided document related to Emergency prod pipeline is for AMS customer but LG electronics is using AEMCS.

Is it also applicable to AEMCS as well?

cc : @seung-ju

Regards,

Kwangmin

K

kwangmin11

Adobe Employee

Hi @arunpatidar

I also heard from Shankari that the prod-only and stage-only pipeline will be available for CS customers as a roadmap item in 2nd half possibly.

Are you able to check the status of this plan when LGE can expects this to be rolled out?

This will also be helpful to expedite LGE's rollback requirements

https://experienceleague.adobe.com/en/docs/experience-manager-cloud-manager/content/using/pipelines/stage-prod-only

cc : @seung-ju

Regards,

Kwangmin

arunpatidar

Community Advisor

Hi @kautuk_sahni
Could you please help checking the status of prod-only and stage-only pipeline features availability?

Arun Patidar

K

kwangmin11

Adobe Employee

Hi @kautuk_sahni

Please help check the status for the AEMCS customer

Regards,

Kwangmin

kautuk_sahni

Community Manager

@seung-ju

Thanks for proposing this idea.

This has been reported to the engineering under the internal reference SITES-23491. The product team will triage this request to verify feasibility based on the prioritization model. This post will be updated according to the Jira status.

Kautuk Sahni

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded