Custom CI/CD Pipeline for Asset Compute Worker Using Bitbucket

[Vishal_Jain03]

Hi,  

We have written code for asset manipulation using Asset Compute Worker in App Builder and want to deploy the code to a higher environment using Bitbucket. Can you please guide me on how to achieve this using Bitbucket?

Request you to please help me with the deployment process.

Thanks,

@EstebanBustamante 
@arunpatidar 
@Harwinder-singh 
@sravs 
@abhishekanand_
@tmj 

[Amit_Zulphe]

Hey @tmj @parussel , 

I checked the Infra team. They mentioned there is nothing blocked from Jenkins firewall. The curl command to below servers and connection seem to work fine.

adobeioruntime.net

adobe.io

adobe-runtime.com

For below domains host could not be resolved.  Not sure if these domains exist.

$curl https://addons.adobe.com curl: (6) Could not resolve host: addons.adobe.com 

$curl https://adobe-addons.com curl: (6) Could not resolve host: adobe-addons.com

$ curl https://adobeio-static.net curl: (6) Could not resolve host: adobeio-static.net

So not sure if whitelisting is the problem. 

[parussel]

It is possible those are not currently utilized but reserved for future use. Here's the full list of our domains:

{wildcard}.adobeio-static.net
{wildcard}.adobeioruntime.net
{wildcard}.adp.adobe.io
addons.adobe.com
adobe-addons.com
adobe-runtime.com
adobeio-static.net
adobeioruntime.net
adp-installservice.adobe.io
adp.adobe.io
analyzer.adobe.io
api.adobe.io
app-builder-delete-proj.adobe.io
app-builder-installation.adobe.io
appregistry.adobe.io
ars.adobe.io
console.adobe.io
csm.adobe.io
developer.adobe.com
developers.adobe.io
eg-ingress.adobe.io
egcm.adobe.io
events.adobe.io
eventsingress.adobe.io
exchange.adobe.io
filterservice.adobe.io
firefly-tvm.adobe.io
jaeger.adobe.io
pia.adobe.io
rtalarmprovider.adobe.io
runtime.adobe.io
s3-journey.adobe.io
scia.adobe.io
skoll.adobe.io
tpia.adobe.io
tracemanager.adobe.io
webhooks.adobe.io
xchg.adobe.io

[Amit_Zulphe]

Hey @parussel , 

There is no specific configuration at our Jenkins server to block any domain. 

Should we manually check if connection to each of these domains you provided work fine?

[parussel]

No, I don't think that would be necessary if you're not blocking any traffic. I posted it as many customers have strict outbound rules and need to open each domain individually. 

[Amit_Zulphe]

Hey @parussel ,

Thanks for the response. I also had an support ticket open for this issue where engineer mentioned I can check with you here all further questions. 

Our use case is to generate image renditions by reading the configurations from processing profile on AEM Cloud environment. The code to deployed in app builder environments reads the image height/width parameters from processing profile and generates the renditions using JIMP library. I am not sure of this will need interactive login or not.

When I tried to use --no-publish parameter in aio app deploy step of Jenkins. It gave below error. 

+ aio app deploy --no-publish --verbose- Updating log forwarding configuration
ℹ Log forwarding is not updated: no configuration is provided
- Building actions for 'dx/asset-compute/worker/1'✔ Built 1 action(s) for 'dx/asset-compute/worker/1'
ℹ No frontend or a build already exists, skipping frontend build for 'dx/asset-compute/worker/1'
- Deploying actions for 'dx/asset-compute/worker/1'
ℹ lastDeployedActions not found, it will be created after first deployment
- Deploying actions for 'dx/asset-compute/worker/1'✖ Deploying actions for 'dx/asset-compute/worker/1'
 › Error: OpenWhiskError: GET https://adobeioruntime.net/api/v1/namespaces/13
 › 4913-wyndassetcompute-stage/packages Returned HTTP 401 (Unauthorized) --> 
 › "The supplied authentication is invalid"
 › at Client.handleErrors (/opt/var/lib/jenkins/workspace/AEM eDAM/App 
 › Builder/Wynd Asset Compute 
 › CI-CD/node_modules/openwhisk/lib/client.js:258:11)
 › at /opt/var/lib/jenkins/workspace/AEM eDAM/App Builder/Wynd Asset 
 › Compute CI-CD/node_modules/openwhisk/lib/client.js:181:26
 › at process.processTicksAndRejections 
 › (node:internal/process/task_queues:95:5)
 › at async findProjectHashOnServer (/opt/var/lib/jenkins/workspace/AEM 
 › eDAM/App Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plug
 › in-app/node_modules/@adobe/aio-lib-runtime/src/utils.js:1719:22)
 › at async Object.syncProject (/opt/var/lib/jenkins/workspace/AEM 
 › eDAM/App Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plug
 › in-app/node_modules/@adobe/aio-lib-runtime/src/utils.js:1575:29)
 › at async deployWsk (/opt/var/lib/jenkins/workspace/AEM eDAM/App 
 › Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plugin-app/no
 › de_modules/@adobe/aio-lib-runtime/src/deploy-actions.js:293:3)
 › at async Object.deployActions (/opt/var/lib/jenkins/workspace/AEM 
 › eDAM/App Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plug
 › in-app/node_modules/@adobe/aio-lib-runtime/src/deploy-actions.js:141:28)
 › at async Deploy.deploySingleConfig (/opt/var/lib/jenkins/workspace/AEM
 › eDAM/App Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plu
 › gin-app/src/commands/app/deploy.js:195:39)
 › at async Deploy.run (/opt/var/lib/jenkins/workspace/AEM eDAM/App 
 › Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plugin-app/sr
 › c/commands/app/deploy.js:106:9)
 › at async Deploy._run (/opt/var/lib/jenkins/workspace/AEM eDAM/App 
 › Builder/Wynd Asset Compute CI-CD/node_modules/@adobe/aio-cli-plugin-app/no
 › de_modules/@oclif/core/lib/command.js:117:22)

Does it have do with any incorrect configuration or does it mean that for our use case we cannot use --no-publish flag?

@tmj @Vishal_Jain03 

[parussel]

--no-publish is likely fine for your use case. You're sending the incorrect auth for that namespace as indicated in the error: 

401 (Unauthorized) --> 
 › "The supplied authentication is invalid"

I'd recommend checking that you're using the correct namespace auth token for staging. 

[Amit_Zulphe]

Hey @parussel , 

I cross checked the Auth value from the credentials file I downloaded from stage workspace. It seem to match correctly. 

But the way we have configured the secrets is as below. Can you please review if probably this needs an update?

.env-stage	AIO_runtime_auth=<auth-value> AIO_runtime_namespace=<stage-namespace> AIO_runtime_apihost=https://adobeioruntime.net SERVICE_API_KEY= <key>
auth-stage	<auth-value>

Should the auth-stage secret be configured as key value pair or it should be stored as just the value like we have done for bindings config to work.

@tmj @Vishal_Jain03 

[tmj]

Hi @Amit_Zulphe, we are unable to provide low-level guidance or debug the Jenkins setup for you. The information needed to make the CI/CD pipeline work is available in our documentation and on this thread. If you need community help to set up Jenkins, you can try StackOverflow or a Jenkins community.

[Amit_Zulphe]

Hey @tmj , 

My question was specific to format of the Auth value we are storing in Jenkins Secrets and not related to how I do set up Jenkins configuration related steps. 

I understand your point of view and thank you so much for the valuable inputs shared until now. I will double check the auth tokens and will also do trial and error from my end. 

[tmj]

I appreciate your understanding, @Amit_Zulphe ! 

[Amit_Zulphe]

Hey @tmj @parussel @sarav_prakash , 

We tried few trail and error options but still running into the same issue. 

It would be of great help if you could help us confirm if .env file should be configured as mentioned below with screenshots. Shared the details of which exact values are configured in .env file as compared to project configuration file which we downloaded from the workspace. 

The AIO_runtime_auth value in .env file we are configuring as value of project > workspace > details > runtime > namespaces > auth as shown in below screenshot. The value is configured along with colon. 

 The SERVICE_API_KEY value in .env file we are configuring as value of project > workspace > details > credentials > oauth_server_to_server > client_id. 

@Vishal_Jain03 

[tmj]

No that looks like the config.json file you downloaded from the Dev Console - Could you confirm? 

1. Go to your project directory

2. Copy the config.json file in that directory

3. Run aio app use config.json

4. This should generate the right .aio and .env files which you can use

[FeliciaKo]

Not sure how to.. can you help

[Amit_Zulphe]

Hey @tmj , 

Yes I downloaded the config JSON from developer console workspace specific configurations. Refer screenshot below. 

I followed the steps you suggested and it created the .aio and .env file. 

I noticed that below variables were not present in the manually created .env file as compared to generated file. Rest of the values are correct. 

I will try configuring this new file in Jenkins and see if that helps. 

Thanks,

Amit