On using AWSSDK some dependencies are not resolved

[#1]

Hi Experts,

I have to use Amazon SQS to read product information from queue in my AEM project and for that I am adding below dependencies to my parent pom.xml

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.15.14</version>
<type>pom</type>
<scope>import</scope>
</dependency>

and in core/pom.xml as 

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
</dependency>

I have added my package where I am using AWS APIs under <Export-Package> and under groupId of <Embed-Dependency> as well.

I have put my custom logic which is leveraging AWS SQS inside the core bundle. On deployment, AWS packages are not getting resolved at Felix Console.

How to resolve these? I am trying the plugin approach but seems every time I am adding new packages to the manifest export list I am seeing new packages which start showing not resolved issues.

like below list:

io.netty.bootstrap,version=[4.1,5) -- Cannot be resolved
io.netty.buffer,version=[4.1,5) -- Cannot be resolved
io.netty.channel,version=[4.1,5) -- Cannot be resolved
io.netty.channel.epoll,version=[4.1,5) -- Cannot be resolved
io.netty.channel.nio,version=[4.1,5) -- Cannot be resolved
io.netty.channel.pool,version=[4.1,5) -- Cannot be resolved
io.netty.channel.socket.nio,version=[4.1,5) -- Cannot be resolved
io.netty.handler.codec -- Cannot be resolved
io.netty.handler.codec.http,version=[4.1,5) -- Cannot be resolved
io.netty.handler.codec.http2,version=[4.1,5) -- Cannot be resolved
io.netty.handler.logging,version=[4.1,5) -- Cannot be resolved
io.netty.handler.ssl,version=[4.1,5) -- Cannot be resolved
io.netty.handler.ssl.util,version=[4.1,5) -- Cannot be resolved
io.netty.handler.timeout,version=[4.1,5) -- Cannot be resolved
io.netty.util,version=[4.1,5) -- Cannot be resolved
io.netty.util.concurrent,version=[4.1,5) -- Cannot be resolved
io.netty.util.internal,version=[4.1,5) -- Cannot be resolved

Thanks

Regards

Shy

[shya · Accepted solution]

Hi @shya,

Add below as part of  <instructions> in maven-bundle-plugin in addition to <Embed-Depedendency>

<Embed-Transitive>true</Embed-Transitive>  

This is responsible for including the dependencies which AWS SQS is dependent on. (AKA Transitive dependencies)

Also, it is suffice to use SQS dependency alone in both main and core/pom.xml (In core/pom.xml, we need not specify the version)

<!-- https://mvnrepository.com/artifact/software.amazon.awssdk/sqs -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
<version>2.16.25</version>
</dependency>

If you are using bnd-maven-plugin to build core module, above entry is not applicable. In that case, create OSGI bundle out of SQS dependency separately outside your project using maven-bundle-plugin.

Recently, had a chance to try AWS SNS in AEM and details about it is available in https://myaemlearnings.blogspot.com/2021/03/aws-service-in-aem.html

Let know if you are facing any issues still.

[shya · Accepted solution]

Hi @Vijayalakshmi_S,

parent pom.xml is having below entry

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
<version>2.16.25</version>
</dependency>

here is my core/pom.xml

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
</dependency>

maven-bundle-plugin is 

<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Sling-Model-Packages>
com.shya.www.core.models,com.aem.community.core,com.shya.www.core.cache.model
</Sling-Model-Packages>
<Import-Package>
javax.inject;version=0.0.0,com.day.cq.replication;version="[6.0.0,7)",
com.day.cq.workflow;version="[1.0,2)",
com.day.cq.workflow.exec;version="[1.0,2)",
com.day.cq.workflow.metadata;version="[1.0,2)",
org.apache.sling.models.annotations;version="[1.1,2)",
com.day.cq.commons.jcr;version="[5.7.0,7.0.0)",
*
</Import-Package>
<Private-Package>org.apache.commons.lang.*</Private-Package>
<Export-Package>
com.shya.aem.*,
com.shya.www.core.service.*,
com.shya.www.core.services.*,
com.shya.www.core.*,
</Export-Package>
<Embed-Dependency>
json,jxl,commons-collections4,*;scope=compile|runtime,jsoup;inline=true
</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>
</instructions>
</configuration>
</plugin>

here is what I am seeing in the felix console

~Shya

[shya · Accepted solution]

@Vijayalakshmi_S,

To avoid above aws package resolution issues,

I have added auth, regions dependency and added those artifacts in <Embeded-Dependency> and have denied remaining packages from import packages list, which were not resolved.

as

<Import-Package>
javax.inject;version=0.0.0,com.day.cq.replication;version="[6.0.0,7)",
com.day.cq.workflow;version="[1.0,2)",
com.day.cq.workflow.exec;version="[1.0,2)",
com.day.cq.workflow.metadata;version="[1.0,2)",
org.apache.sling.models.annotations;version="[1.1,2)",
com.day.cq.commons.jcr;version="[5.7.0,7.0.0)",
!software.amazon.awssdk.awscore.*,
!software.amazon.awssdk.core.*,
!software.amazon.awssdk.http.*,
!software.amazon.awssdk.metrics.*,
!software.amazon.awssdk.profiles.*,
!software.amazon.awssdk.protocols.*,
!software.amazon.awssdk.utils.*,
!software.amazon.eventstream,
!org.reactivestreams,
*
</Import-Package>

<Embed-Dependency>
auth,regions,sqs,json,jxl,commons-collections4,*;scope=compile|runtime,jsoup;inline=true
</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>

<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sqs</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>auth</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>regions</artifactId>
</dependency>

I am seeing all the bundles are active but my OSGi component which listens to AWS SQS is in satisfied state,  error.log shows below exception at the time of deployment.

(158)[org.apache.sling.commons.scheduler.impl.WhiteboardHandler(322)] : DependencyManager : invokeBindMethod : Service not available from service registry for ServiceReference [java.lang.Runnable] for reference registerRunnable
25.03.2021 20:53:43.222 *ERROR* [FelixDispatchQueue] org.apache.sling.commons.scheduler FrameworkEvent ERROR (org.osgi.framework.ServiceException: Service factory returned null. (Component: com.shya.www.core.impl.schedulers.AWSSWSQueueReadTask (3643)))
org.osgi.framework.ServiceException: Service factory returned null. (Component: com.shya.www.core.impl.schedulers.AWSSWSQueueReadTask (3643))
at org.apache.felix.framework.ServiceRegistrationImpl.getFactoryUnchecked(ServiceRegistrationImpl.java:381)
at org.apache.felix.framework.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:248)
at org.apache.felix.framework.ServiceRegistry.getService(ServiceRegistry.java:350)

Here is the scheduler code 

import java.util.List;
import java.util.Map;

import org.apache.felix.scr.annotations.Activate;
import org.apache.felix.scr.annotations.Component;
import org.apache.felix.scr.annotations.Properties;
import org.apache.felix.scr.annotations.Property;
import org.apache.felix.scr.annotations.Service;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sqs.SqsClient;
import software.amazon.awssdk.services.sqs.model.GetQueueUrlRequest;
import software.amazon.awssdk.services.sqs.model.Message;
import software.amazon.awssdk.services.sqs.model.ReceiveMessageRequest;
import software.amazon.awssdk.services.sqs.model.SqsException;

@Component
@Service(value = Runnable.class)
@Properties({ @Property(name = "scheduler.expression", value = "*/30 * * * * ?", description = "Cron-job expression"),
@Property(name = "scheduler.concurrent", boolValue = false, description = "Whether or not to schedule this task concurrently") })
public class AWSSWSQueueReadTask implements Runnable {

static final Logger LOG = LoggerFactory.getLogger(AWSSWSQueueReadTask.class);

@Override
public void run() {

AwsBasicCredentials awsCreds = AwsBasicCredentials.create("my_key", "my_secret");

SqsClient sqsClient = SqsClient.builder().credentialsProvider(StaticCredentialsProvider.create(awsCreds))
.region(Region.US_EAST_1).build();

GetQueueUrlRequest getQueueRequest = GetQueueUrlRequest.builder().queueName("my_queue").build();

String queueUrl = sqsClient.getQueueUrl(getQueueRequest).queueUrl();

try {
ReceiveMessageRequest receiveMessageRequest = ReceiveMessageRequest.builder().queueUrl(queueUrl)
.maxNumberOfMessages(1).build();
List<Message> messages = sqsClient.receiveMessage(receiveMessageRequest).messages();

LOG.info("Messages are {}", messages);

} catch (SqsException e) {
LOG.error("Exception while sending the message {}", e.awsErrorDetails().errorMessage());
}

}

@Activate
protected void activate(final Map<String, Object> config) {
LOG.info("scheduler activated....");
configure(config);
}

private void configure(final Map<String, Object> config) {
LOG.info("AWS SQS Scheduler configure");
}

Regards

Shya

[Vijayalakshmi_S · Accepted solution]

Hi @shya, Is this a typo in this line ? after scope, you have mentioned jsoup

<Embed-Dependency>
json,jxl,commons-collections4,*;scope=compile|runtime,jsoup;inline=true
</Embed-Dependency>

[Vijayalakshmi_S · Accepted solution]

Can you do the below

• Revoke the changes done within maven-bundle-plugin entry
• Retain the dependency in main and core/pom.xml alone.
• Build the AWS SQS dependency as a separate OSGI bundle and install directly. (Link to my previous post for the details of building and installing OSGi bundle is mentioned in the link shared)

This will make your bundle active and functionality should work. 

Later, if you don't want it to be separate bundle and it has to be as part of your project code base, share the core/pom.xml (In particular, maven-bundle-plugin entry as it was before making any changes related to this)

[shya · Accepted solution]

Hi @Vijayalakshmi_S

I am able to resolve all the dependency issues and component statisfied state issue as well with the external OSGI bundle (for AWS SQS Jar) approach as you suggested, Thnak you for your help.

I want to bring this bundle as part of my project, sharing my core/pom.xml as you asked,

<plugin>
<groupId>org.apache.felix</groupId>
<artifactId>maven-bundle-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<instructions>
<Sling-Model-Packages>
com.shya.www.core.models,com.aem.community.core,com.shya.www.core.cache.model
</Sling-Model-Packages>
<Import-Package>
javax.inject;version=0.0.0,com.day.cq.replication;version="[6.0.0,7)",
com.day.cq.workflow;version="[1.0,2)",
com.day.cq.workflow.exec;version="[1.0,2)",
com.day.cq.workflow.metadata;version="[1.0,2)",
org.apache.sling.models.annotations;version="[1.1,2)",
com.day.cq.commons.jcr;version="[5.7.0,7.0.0)",
*
</Import-Package>
<Private-Package>org.apache.commons.lang.*</Private-Package>
<Export-Package>
com.shya.aem.*,
com.shya.www.core.service.*
</Export-Package>
<Embed-Dependency>
jsoup,json,jxl,commons-collections4,*;scope=compile|runtime
</Embed-Dependency>
<Embed-Transitive>true</Embed-Transitive>
</instructions>
</configuration>
</plugin>

Regards,

Shya

[Vijayalakshmi_S · Accepted solution]

Hi @shya

Have just amended  <Embed-Dependency> entry to include dependencies inline (inline=true)

Have added my explanation as XML comments below to better relate. 

In your local, 

• Uninstall the AWS SQS bundle. 
• In project code base, check if main pom has shared SQS dependency entry + core/pom, without version. 
• Add inline=true in Embed-Dependency (If jsoup, json and other explicit entries is available in core/pom, you need not explicitly mention. Using *  alone will include all dependency entries )
• Build and Test

<plugin>
    <groupId>org.apache.felix</groupId>
    <artifactId>maven-bundle-plugin</artifactId>
    <extensions>true</extensions>
    <configuration>
        <instructions>
            <Sling-Model-Packages>
                com.shya.www.core.models,com.aem.community.core,com.shya.www.core.cache.model
            </Sling-Model-Packages>
            <Import-Package>
                javax.inject;version=0.0.0,com.day.cq.replication;version="[6.0.0,7)",
                com.day.cq.workflow;version="[1.0,2)",
                com.day.cq.workflow.exec;version="[1.0,2)",
                com.day.cq.workflow.metadata;version="[1.0,2)",
                org.apache.sling.models.annotations;version="[1.1,2)",
                com.day.cq.commons.jcr;version="[5.7.0,7.0.0)",
                *
            </Import-Package>
            <Private-Package>org.apache.commons.lang.*</Private-Package>
            <!--Add SQS related Package entry here only if you need that to be exported/available for use by any other bundle 
            If multiple bundles need to make use of SQS and/or if you foresee use of other AWS service in future, then it is good to keep it as separate bundle 
            If it is solely for this project bundle and not for anything else, no need of entry in Export-Package -->
            <Export-Package>
                com.shya.aem.*,
                com.shya.www.core.service.*
            </Export-Package>
            <!-- Dependency entries part of core/pom file that are of compile and runtime included inline 
            Use of "*" in addition to explicit dependency artifact will include our SQS dependency too if it is available in core/pom.xml -->
            <Embed-Dependency>
                jsoup,json,jxl,commons-collections4,*;scope=compile|runtime;inline=true
            </Embed-Dependency>
            <Embed-Transitive>true</Embed-Transitive>
            </instructions>
    </configuration>
</plugin>

[shya · Accepted solution]

Hi @Vijayalakshmi_S   It seems inline true is not working at all, still at the initial stage where amazon's packages were not getting resolved in my core bundle.

and these packages can't be denied as I am using these packages in core.

Regards,
Shya

[Vijayalakshmi_S · Accepted solution]

Hi @shya,

Can you try the following in your local. 

<Embed-Dependency>*</Embed-Dependency> => This implies inline=false, includes dependencies of all scope and all artifacts that is part of core pom (It should include jsoup and other explicit mentions too if those are available as dependency entries)

[#11]

Hey @shya try replacing :

<scope>import</scope>

with

<scope>provided</scope>

Thanks,

Bilal.