Actions in Project Firefly returning a 503 status code with some weird AWS Lambda error sporadically

[rkishoreindraganti]

Hi,

I hope you are doing good!

We are facing a weird issue in our project firefly app development, in our API layer (using actions here; in node.js) we are trying to establish AEP datasets and issuing some query against the dataset using ODBC strategy (Not HTTP Query Service);

On invoking the REST endpoint that I stated above we are getting some weird error with a status code as 503 sporadically (Find the screenshot below).

This is being a show stopper to continue our development, we are feeling reluctant to proceed further considering this may be a limitation in the Adobe IO, your suggestions are much appreciated.

Please let us know the root cause and solution ASAP.

Thanks!

[duypnguyen]

Hi @rkishoreindraganti - I assume you are calling your action via "https://your-namespace.adobeio-static.net/...", is that correct?

Please try replacing "adobeio-static.net" by "adobeioruntime.net", and try again.

The reason why you got this error is that, your action ran longer than 30 seconds and caused a timeout at the CDN "adobeio-static.net".

The "adobeioruntime.net" domain has a timeout of 1 minute, which is default for all web actions running on I/O Runtime.

That being said, a web request taking more than 30 seconds is considered very slow. Please re-visit the implementation to optimize it if possible, for example by moving the time consuming part to an async (non-blocking) execution.

As a side note, please also update your AIO CLI with "npm install -g @adobe/aio-cli". All the URLs should be on "adobeioruntime.net" in a recent CLI release.

[kishoreindraganti]

@duypnguyen : Thanks for the quick response, it is much appreciated! 

I have tried your approach on updating the @adobe/aio-cli though it is referring to adobe-io.static.net. Please find the below snippet. 

[duypnguyen]

@kishoreindraganti - thanks for sharing the screenshot. I can see the same output from the `aio app deploy` command.

Regardless, it is only an output from the CLI command. You should still be able to call the same URL with adobeioruntime.net - could you please try in Postman? If you develop your app locally by `aio app run`, adobeioruntime.net is used.

I will follow up with the team on fixing the CLI output.

(Just to clarify: the different domains mentioned above are only applied for the action URLs. The frond-end is always served from https://your-namespace.adobeio-static.net/index.html)

[kishoreindraganti]

I am trying your other solutions, in the meantime, I just want to make things clear, it is not only on the CLI output, even the application also makes the call to adobeio-static.net only not to the adobeioruntime.net though I am using the latest CLI version.

[kishoreindraganti]

"aio app run"  uses "adobeioruntime.net" but "aio app deploy" using "adobe-io.static.net"

[duypnguyen]

You're right, sorry for the confusion. And yes, pursuing the other approach (async call for long requests) is strongly recommended.

[kishoreindraganti]

What do you mean by async call? can you point me to some resources, please?

[duypnguyen]

You can see how to call your action in async mode here: https://github.com/AdobeDocs/adobeio-runtime/blob/master/guides/asynchronous_calls.md.

I understand that your app has a UI as well. So the end user experience on the UI would be:

- you trigger the web action (A) to run dataset query

- (A) receives your request, calls another action to run the actual dataset query job (B) in an async mode

- (A) responds immediately to the UI (without waiting for (B) to complete), e.g. with status "Query is running"

- you have a button "check query status" to poll the progress of (B). When it's completed, you will receive "Query is complete. Here is the results..."

[kishoreindraganti]

Ok. For that, I want to create an action that has to reside out of the ProjectFirefly application? so we can't increase the timeout for the actions which is coming as part of ProjectFirefly? This is my understanding finally, is that correct?

[duypnguyen]

You can create any action within the project Firefly app, both web action (1 minute timeout) and async (max 30 minutes timeout). Simply run `aio app add action` to add a new one. The important thing is to remove "web: yes" for that action and increase the timeout as you wish. Your manifest.yml should look like:

packages:
  __APP_PACKAGE__:
    license: Apache-2.0
    actions:
      query-dataset:
        function: actions/client/index.js
        web: 'no'
        runtime: 'nodejs:12'
        inputs:
          LOG_LEVEL: debug
        limits:
          timeout: 600000 # 10 minutes
        annotations:
          final: true
      web-action:
        function: actions/caller/index.js
        web: 'yes'
        runtime: 'nodejs:12'
        inputs:
          LOG_LEVEL: debug
        annotations:
          require-adobe-auth: true
          final: true

Then in the "web-action", you call "query-dataset". You can use the openwhisk npm lib to do so: https://www.npmjs.com/package/openwhisk. Example:

const activationId = await ow.actions.invoke({
        name: `your-app-0.0.1/query-dataset`,
        blocking: false,
        result: false,
        params: { name: 'hello' } 
})

[kishoreindraganti]

@duypnguyen : thanks for all your help, it is much appreciated by our team! We will try to update the framework to accommodate the async flow and will reach you if any blockers that we hit.

[kishoreindraganti]

@duypnguyen seeing this weird issue "The action produced a response that exceeded the allowed length: 1117055 > 1048576 bytes" , so the openwhisk won't support the response size of greater than 1 mb?

[duypnguyen]

@kishoreindraganti - the payload limit in Runtime is 1MB, more details at: https://www.adobe.io/apis/experienceplatform/runtime/docs.html#!adobedocs/adobeio-runtime/master/gui....

To deliver larger payload you can leverage a cloud storage, for example upload your files to S3 and return a redirect response from your action to the file URL.