Expand my Community achievements bar.

Custom CI/CD Pipeline for Asset Compute Worker Using Bitbucket

Avatar

Level 3

Hi,  

We have written code for asset manipulation using Asset Compute Worker in App Builder and want to deploy the code to a higher environment using Bitbucket. Can you please guide me on how to achieve this using Bitbucket?

 

Request you to please help me with the deployment process.

Thanks,

@EstebanBustamante 
@arunpatidar 
@Harwinder-singh 
@sravs 
@abhishekanand_
@tmj 

34 Replies

Avatar

Community Advisor

Hi @Vishal_Jain03 ,

Adobe provided excellent documentation to setup CI-CD pipeline using github actions

https://developer.adobe.com/app-builder/docs/resources/ci-cd/requirements/

You can refer to my this reply. But this works only for Github.

 

We cant directly trigger Github actions from Bitbucket. Should try some workarounds

  1. Setup github repo. Trigger sync from Bitbucket into github. And then run the adobe provided github actions to deploy to AppBuilder.
  2. Install Bitbucket Post webhook plugin. The plugin will trigger github actions. But this once again requires sync of code from Bitbucket into Github
  3. Reinvent the wheel. Refer to adobe deploy script. And rewrite the yaml to work from Bitbucket pipelines. afaik no direct adobe documentation for Bitbucket pipeline. But steps involved would as below. 
    1. Checkout code
    2. Install node, npm i
    3. install adobe/aio-cli-setup-action,  adobe/aio-apps-action
    4. setup secrets as env variables
    5. Finally run `aio app deploy`. If required add --force-build, verbose, --no-action, whatever required flags.  

Sounds you ll need to reinvent with option3 to rewrite the github actions yaml into bitbucket pipeline. I have not done this. But I have written as Jenkins job to perform above steps in above order to checkout code from TFS and deploy onto AppBuilder. All the best!

Avatar

Level 3

Thank you for your reply @sarav_prakash 

Actually, we are exploring both options bitbucket pipelines and Jenkins job
I have one question related to Jenkins job

If the .env file contains (

AIO_RUNTIME_NAMESPACE,AIO_RUNTIME_AUTH) then what is the need of separately configuring the file for auth and then what is the actual content of that AIO_RUNTIME_AUTH file?


Also, why we need to set the AIO_RUNTIME_NAMESPACE

again, in the shell command section if it is already present in the .env file?

please find the attached screenshot

Picture1.png

Picture2.png

Request you to please help me with the deployment process.
Thanks & Regards,

Vishal Jain

Avatar

Level 3

Hi @sarav_prakash

we have followed the steps given in the video( App Builder Live - CI/CD) regarding Jenkins setup.
Below is our Bash File

#!/bin/bash
source ~/.bashrc
echo $ENV_FILE_CONTENTS  > .env
cp .aio-stage .aio
npm i
export AIO_runtime_namespace=13xxx-xxxxassetcompute-stage
aio app deploy

Error
/tmp/jenkins2065956486425470096.sh: line 9: aio: command not found

and when we change the bash file with

#!/bin/bash
source ~/.bashrc
echo $ENV_FILE_CONTENTS  > .env
cp .aio-stage .aio
npm install -g @adobe/aio-cli
npm i
export AIO_runtime_namespace=13xxx-xxxxassetcompute-stage
aio app deploy


Error
The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user

Also as you mentioned 
->But I have written as Jenkins job to perform above steps in above order to checkout code from TFS and deploy onto AppBuilder

can you please send us the detail steps you followed to write Jenkins job and also the sample bash file which you used. It will be really helpful for us.

Request you to please help me with the deployment process.

Thanks & Regards,

Vishal Jain

Avatar

Level 3

Hi @sarav_prakash 

Have you had the chance to go through the previous post? Please take a moment to go through it, and feel free to reach out to us if you have any questions.

Looking forward to hearing from you!"

Thanks & Regards,
Vishal Jain

Avatar

Community Advisor

Hi @Vishal_Jain03 ,

sorry for late reply. Here is a custom github action that runs on custom runner and not github runner 

name: AIO App CI

on:
  push:
    branches:
      - dev
jobs:
  deploy:
    name: Deploy to dev
    runs-on: "${{ vars.CICD_DEFAULT_RUNNER }}"
    strategy:
      max-parallel: 1
      matrix:
        node-version: ["20"]
        os: [ubuntu-latest]
     
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: get-secrets-action
        id: get-secrets-action
        with:
          vault: "***"
          secrets: "AIO-RUNTIME-NAMESPACE,AIO-RUNTIME-AUTH"
        uses: example/cicd-devops-actions/.github/actions/get-secrets@main
      - name: get-certificates-action
        id: get-certificates-action
        with:
          certificates: "ca-root-devops"
        uses: example/cicd-devops-actions/.github/actions/get-certificates@main
      - name: set-certificates-node
        id: set-certificates-node
        shell: bash
        run: |
          echo "[DEBUG] Configuring Cert with Node"
          export NODE_EXTRA_CA_CERTS="${CA_ROOT_DEVOPS}"
          env
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - name: npm install
        run: npm ci
      - id: install-aio
        name: install-aio
        shell: bash
        run: |
          echo "[DEBUG] Installing AIO";
          npm install -g @adobe/aio-cli;
          echo "[DEBUG] NPM version";
          npm -v;
          echo "[DEBUG] NODE.JS version";
          node -v;
          echo "[DEBUG] AIO version";
          aio --version;
      - name: Setup CLI
        uses: adobe/aio-cli-setup-action@1.3.0
        with:
          os: ${{ matrix.os }}
          version: 10.x.x
      - name: Build
        # env:
        #   AIO_RUNTIME_NAMESPACE: ${{ secrets.AIO-RUNTIME-NAMESPACE }}
        uses: adobe/aio-apps-action@3.4.0
        with:
          os: ${{ matrix.os }}
          command: build
      - name: Deploy
        # env:
        #   AIO_RUNTIME_NAMESPACE: ${{ secrets.AIO-RUNTIME-NAMESPACE }}
        #   AIO_RUNTIME_AUTH: ${{ secrets.AIO-RUNTIME-AUTH }}
        uses: adobe/aio-apps-action@3.4.0
        with:
          os: ${{ matrix.os }}
          command: deploy

 

Point is, before `npm i` and `npm i -g @adobe/aio-cli`, we added 

`export NODE_EXTRA_CA_CERTS="${CA_ROOT_DEVOPS}"`

We are loading our cert into runner, and then had to global install proceed. I ll check if we have bash script. It took a while to figure trial-n-error. We had to disable cert check, disable TLS, after few attempts only got it working. 

This is sure bumpy ride to figure non-adobe-way.

Avatar

Level 3

Hi @sarav_prakash 

Currently, our organization does not use GitHub so we can't use GitHub actions.

Bitbucket pipelines is also not an option it has limitation of no of minutes on pipeline executions. 

 

So, we decided to proceed with Jenkins Job Setup, and we are facing below issues

#!/bin/bash
source ~/.bashrc
echo $ENV_FILE_CONTENTS  > .env
cp .aio-stage .aio
npm install -g @adobe/aio-cli
npm i
export AIO_runtime_namespace=13xxx-xxxxassetcompute-stage
aio app deploy

Error
The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user

Request you to please help me with the deployment process.

Thanks & Regards,

Vishal Jain

Avatar

Employee
Employee

@Vishal_Jain03 this video is a few years old but I think it should still prove to be a useful reference: https://www.youtube.com/watch?v=lbB2jl2rQZM&ab_channel=AdobeDevelopers

 

Avatar

Level 3

Thank you for your reply @tmj 

Actually, we are exploring both options bitbucket pipelines and Jenkins job
I have one question related to Jenkins job

If the .env file contains (

AIO_RUNTIME_NAMESPACE,AIO_RUNTIME_AUTH) then what is the need of separately configuring the file for auth and then what is the actual content of that AIO_RUNTIME_AUTH file?


Also, why we need to set the AIO_RUNTIME_NAMESPACE

again, in the shell command section if it is already present in the .env file?

please find the attached screenshot

Picture1.png

 

Picture2.png

Request you to please help me with the deployment process.

Thanks & Regards,

Vishal Jain

Avatar

Employee
Employee

Hi @Vishal_Jain03 

 

>If the .env file contains (AIO_RUNTIME_NAMESPACE,AIO_RUNTIME_AUTH) then what is the need of separately configuring the file for auth and then what is the actual content of that AIO_RUNTIME_AUTH file?

> Also, why we need to set the AIO_RUNTIME_NAMESPACE

again, in the shell command section if it is already present in the .env file?


It's been a few years since the video was recorded so I cannot tell you off the top of my head. But did you try with/without it? Did you face any problems? Can you show us your code and tell us what problems you are facing? 

 

App Builder provides an out-of-the-box CI/CD pipeline with Github actions. If you want to implement it in Bitbucket or Jenkins that can be achieved but you have to write the code yourself. See docs for more help - https://developer.adobe.com/app-builder/docs/guides/deployment/ci_cd_for_firefly_apps/ 

 

Thanks
Manik

 

Avatar

Level 3

Hi @tmj 

Currently, our organization does not use Github so we cant use Github actions. 

Bitbucket pipelines is also not an option it has limitation of no of minutes on pipeline executions. 

 

So, we will mostly proceed with Jenkins. Before the setup we had few doubts so reached out. We will try to deploy the code using Jenkins. If we get stuck along the way, we will reach out to you.


Thanks & Regards
Vishal Jain

Avatar

Level 3

Hi @tmj 
we have followed the steps given in this video regarding Jenkins setup.
Below is our Bash File

#!/bin/bash
source ~/.bashrc
echo $ENV_FILE_CONTENTS  > .env
cp .aio-stage .aio
npm i
export AIO_runtime_namespace=13xxx-xxxxassetcompute-stage
aio app deploy

Error
/tmp/jenkins2065956486425470096.sh: line 9: aio: command not found

and when we change the bash file with

#!/bin/bash
source ~/.bashrc
echo $ENV_FILE_CONTENTS  > .env
cp .aio-stage .aio
npm install -g @adobe/aio-cli
npm i
export AIO_runtime_namespace=13xxx-xxxxassetcompute-stage
aio app deploy


Error
The operation was rejected by your operating system.
npm error It is likely you do not have the permissions to access this file as the current user

Request you to please help me with the deployment process.

Thanks & Regards,

Vishal Jain

Avatar

Level 3

Hi @tmj 

Have you had the chance to go through the previous post? Please take a moment to go through it, and feel free to reach out to us if you have any questions.

Looking forward to hearing from you!"

Thanks & Regards,
Vishal Jain

Avatar

Employee
Employee

Hi Vishal, it seems like AIO CLI is unavailable in your CI/CD pipeline.

 

Have you tried installing it through the steps mentioned in this article - https://support.atlassian.com/bitbucket-cloud/docs/specify-dependencies-in-your-pipelines-build/?

 

Edit: I see you are trying to use Jenkins now. In that case, you will have to install aio cli directly on your Jenkins server by running 

 

npm install -g @adobe/aio-cli

 

Thanks
Manik

Avatar

Level 3

Hi @tmj 


I have 2 follow up questions

 

1. Do you mean we need to remove it from pipeline specific shell command and directly install it in Jenkins server?
2. We had faced an error related to permission
"npm error Error: EACCES: permission denied, mkdir '/usr/lib/node_modules/@adobe'". Our Jenkins team was asking whether there is a way to install it in local directory instead of system directory?

Thanks & Regards,
Vishal Jain

 

 

Avatar

Employee
Employee

> Do you mean we need to remove it from pipeline specific shell command and directly install it in Jenkins server?

 

Yes

 

>Our Jenkins team was asking whether there is a way to install it in local directory instead of system directory?

You can try installing without the -g flag. However, please ensure that when the Jenkins pipeline job runs, it's bash can find and execute aio.

Avatar

Level 3

Hi @tmj 
Thanks for your reply

Our Jenkins team mentioned that they cannot install aio at server level so they kept the command in execute shell section itself and removed -g as you suggested.
Below is the configuration. 

#!/bin/bash

source ~/.bashrc

echo $ENV_FILE_CONTENTS  > .env
cp .aio-stage .aio
npm install @adobe/aio-cli
npm i
export AIO_runtime_namespace=13xxxx-xxxxassetcompute-stage
export PATH="$PATH:./node_modules/.bin"
aio app deploy --verbose

in PATH variable aio installation directory is mentioned.

After adding this config below is the error we are getting now. 
Error is as follows

+ aio app deploy --verbose ›   Error: IMSOAuthLibError: [IMSOAuthSDK:TIMEOUT] Timed out after 120 
›   seconds.
›       at new <anonymous> (/opt/var/lib/jenkins/workspace/AEM eDAM/App 
›   Builder/Wxxx Asset Compute CI-CD/node_modules/@adobe/aio-lib-core-errors/s
›   rc/AioCoreSDKErrorWrapper.js:33:9)
›       at Timeout._onTimeout (/opt/var/lib/jenkins/workspace/AEM eDAM/App 
›   Builder/Wxxx Asset Compute 
›   CI-CD/node_modules/@adobe/aio-lib-ims-oauth/src/login.js:72:14)
›       at listOnTimeout (node:internal/timers:581:17)
›       at process.processTimers (node:internal/timers:519:7)

Request you to please help me with the deployment process.

CC: @sarav_prakash 

Thanks & Regards,

Vishal Jain

Avatar

Community Advisor

this error means, your jenkins server running behind your firewall is unable to talk to aio cloud. hmm we too faced similar security issues and connection timeouts. After pointed above, after multiple trial-n-error, installing certs, disable cert checks, only then we got it. But still mine is in github action running from corp runner behind firewall. I fear this is specific to your firewall. Is your jenkins onprem? Can you try from external cloud https://www.jenkins.io/doc/tutorials/tutorials-for-installing-jenkins-on-Google-Cloud/ ?

 

Avatar

Employee
Employee

You may need to allow list the following domains in your Firewall 

 

*.adobeio-static.net
*.adobeioruntime.net
*.adobe.io 
addons.adobe.com
adobe-addons.com
adobe-runtime.com

 

Avatar

Level 2

Hey @tmj , 

Are these all domains use port 443? Infra team was asking which port should be added in firewall for these domains.

 

I am working with Vishal on solving this issue?

@Vishal_Jain03 

Avatar

Employee

@Amit_Zulphe - yes, aio communication is all over port 443.