Write AEMasaCS logs directly to ElasticSearch

[RustamMussabay]

Hello,

Is it possible to configure AEMasaCS project to write logs directly to external ElasticSearch to monitor logs in Kibana?

I know that we can download/tail logs via aio and then use beats to submit data to ES, but I was told to investigate if there is a way to send data directly. I've checked ACS Commons and saw a SyslogAppender feature that is not compatible with AEMasaCS, and I believe the entire ch.qos.logback package is marked as Deprecated with a "This internal logback API is not supported by AEM as a Cloud Service." message. So, I'm wondering if this is even a valid approach. Also, I'm not sure how much performance overhead it will cost us to send logs via network.

[Peter_Puzanovs]

Hi Rustam,

Sending logs directly through the jvm process to Elastic is definitely not good, it may indeed impact many parts of the Adobe application.

Adobe IO gives you the opportunity to download logs via [1] download logs API,

In ELK stack we can utilize logstack cron based events system to import data into your index[2]. Once it's in index we can do all of the monitoring needed. This way you will utilise aio events that live outside of your jvm process.

[1] https://developer.adobe.com/experience-cloud/cloud-manager/reference/api/#tag/Environments/operation...

[2] https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html

Regards,

Peter

[RustamMussabay]

Hi Peter,

Thanks for your response!

Yes, I'm aware of the aio utility to download/tail logs and my suggestion was to use it to gather the logs and then feed it to ES. But I also want to understand whether it's possible to send logs from AEM application or not, and if not, why. If it was a regular Java app, I would add logback-elasticsearch-appender dependency, but it depends on ch.qos.logback packages that are not compatible with AEM as a CS.

Nevertheless, AEM as a CS uses something to store logs in a file system, so may be there is a documentation I couldn't find or something undocumented that sheds a light on this topic. 

[Peter_Puzanovs · Accepted solution]

Hi Rustam,

We have very limited direct access to FS in CS version of AEM.

There's no such functionality in AEM as a CS to share files directly with Elastic.

There's an option for Splunk(Not Elastic) to setup forwarder which will get you a bit closer to your instance, but then you still may get blocked depending on your Adobe Azure security settings when connection to your organisation.

AFAIK cron job in logstack and aio API is the best option I found so far.

May be someone else in community found a better approach.

Regards,

Peter

[RustamMussabay]

Thanks, Peter.

I'm going to mark this reply as correct, because I don't believe we can get any closer to the answer without an insight from Adobe. 

Have a nice weekend!

Rustam

[Jörg_Hoh]

I am aware of some thoughts to support log-forwarding to other systems than Splunk. You might want to contact customer support which can probably get you in contact with the right Adobe folks.

[RustamMussabay]

Hello Jorg,

Apologies for the delay in responding.

I reached out to our contact at Adobe via email about the issue and quoted your previous response.

Your help is really appreciated.