HttpClient is not passing through proxy

[chandran8596539]

Am trying to consume a web service which is running on different sub-domain but on the same domain. I am getting java.net.ConnectException: Connection timed out. Below is code which I am using to call the webservice from OSGI Component class.

       String  url = "https://banking.pnc.com";
        PostMethod post = null;
        BufferedReader br = null;
        try {
            HttpClient httpClient = new HttpClient();
            post = new PostMethod(url);
            post.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false));

            String json_Params = "{\"token\":\"" + token \"}";
            StringRequestEntity requestEntity = new StringRequestEntity(json_Params, "application/json", "UTF-8");
            post.setRequestEntity(requestEntity);

            int statusCode = httpClient.executeMethod(post);
            logger.debug("statusCode :::"+statusCode);
            response.setStatus(statusCode);
           

        } catch (Exception e) {
            logger.error("Error while ", e);
        } finally {

        }

Configured the proxy settings in OSGI Config "Day Commons HTTP Client 3.1", This request is not passing through the proxy , because of this am getting Connection timed out. Added the following proxy configuration to the HttpClient in the code and this is working good, it is passing thru proxy server.

//        HostConfiguration config = httpClient.getHostConfiguration();
//        config.setProxy(HTTP_Proxy_URL,Http_Proxy_port);

//      int statusCode = httpClient.executeMethod(config, post);

if we configure the proxy settings in OSGI Config "Day Commons HTTP Client 3.1", will HttpClient instance automatically picks proxy settings? or we need to pass the proxy settings in the code as shown above?

[acroabhi]

I'm using AEM 6.3 and have the same issue.

I'm making a request to external server using apache http-client.

AEM is behind a proxy. Proxy is set in Apache HTTP components proxy configuration. I expect my http client to use the same proxy by default.

Any idea what I am doing wrong

[Gaurav-Behl]

Is that external service hosted on HTTPS? If yes, did you configure TLS settings in your code?

could you share the error from server logs and request/response including headers?

[acroabhi]

I have access to both HTTP and HTTPS versions of the service.

Simply the connection times out as the http-client cannot access external world without a proxy

If I add this:

httpClientBuilder.setProxy(new HttpHost(uri.getHost(), uri.getPort(), uri.getScheme()));

it just works fine.

But I want it to use default proxy set in AEM so I don't need to provide any field for setting proxy.

I tried : HttpClientBuilder.create().useSystemProperties() but that doesn't work.

Thanks,
Abhijeet

[Gaurav-Behl]

Are you talking about Apache HTTP Components Proxy Configuration? If yes, then this has to be ready via OSGI Config Admin.

Could you share more details about the default proxy in AEM?

[acroabhi]

Yes.
I set the proxy here(as shown in screenshot). My expectation is my http-client should use this proxy?

Am I missing something

[Gaurav-Behl]

Use  org.osgi.service.cm.Configuration and org.osgi.service.cm.ConfigurationAdmin to read the values.

It must have got created under /apps/system/config/org.apache.http.proxyconfigurator-xxx.config

Example -

Adobe Experience Manager Help | Reading Adobe Experience Manager OSGi Configuration Values

Try this first rather than going the above mentioned route.

//Add the HTTP Client Builder Factory reference like below in your OSGI component.

@Reference

private HttpClientBuilderFactory httpFactory; //in your method then you can get the HTTP Client object.

HTTPClient client =  httpFactory.newBuilder().build();

[acroabhi]

I tried using HttpFactory as you mentioned but it is throwing exception:

Code I had previously:

private static CloseableHttpClient createHttpClient()

    {

        final PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();

        final HttpClientBuilder httpClientBuilder = HttpClientBuilder.create().setConnectionManager(cm);

        httpClientBuilder.addInterceptorFirst(new ContentLengthHeaderRemover());

        return httpClientBuilder.build();

    }

After change:

private CloseableHttpClient createHttpClient()

    {

        final PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();

        final HttpClientBuilder httpClientBuilder = httpFactory.newBuilder();

        httpClientBuilder.setConnectionManager(cm);

        httpClientBuilder.addInterceptorFirst(new ContentLengthHeaderRemover());

        return httpClientBuilder.build();

    }

Exception:

0 TIMER_START{Request Processing}

      0 COMMENT timer_end format is {<elapsed microseconds>,<timer name>} <optional message>

      3 LOG Method=GET, PathInfo=null

      5 TIMER_START{handleSecurity}

    649 TIMER_END{642,handleSecurity} authenticator org.apache.sling.auth.core.impl.SlingAuthenticator@5436aff3 returns true

    805 TIMER_START{ResourceResolution}

    890 TIMER_END{84,ResourceResolution} URI=/apps/acrolinx/proxy/sidebar/v14/index.html resolves to Resource=NonExistingResource, path=/apps/acrolinx/proxy/sidebar/v14/index.html

    895 LOG Resource Path Info: SlingRequestPathInfo: path='/apps/acrolinx/proxy/sidebar/v14/index.html', selectorString='null', extension='html', suffix='null'

    895 TIMER_START{ServletResolution}

    896 TIMER_START{resolveServlet(/apps/acrolinx/proxy/sidebar/v14/index.html)}

   1377 LOG {0}: no servlet found

   1451 TIMER_END{554,resolveServlet(/apps/acrolinx/proxy/sidebar/v14/index.html)} Using servlet org.apache.sling.servlets.get.DefaultGetServlet

   1454 TIMER_END{558,ServletResolution} URI=/apps/acrolinx/proxy/sidebar/v14/index.html handled by Servlet=org.apache.sling.servlets.get.DefaultGetServlet

   1456 LOG Applying Requestfilters

   1458 LOG Calling filter: com.adobe.granite.resourceresolverhelper.impl.ResourceResolverHelperImpl

   1463 LOG Calling filter: org.apache.sling.i18n.impl.I18NFilter

   1465 LOG Calling filter: com.adobe.granite.httpcache.impl.InnerCacheFilter

   1471 LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter

   1473 LOG Calling filter: com.adobe.cq.mcm.campaign.servlets.CampaignCopyTracker

   1475 LOG Calling filter: com.adobe.cq.history.impl.HistoryRequestFilter

   1594 LOG Calling filter: com.day.cq.wcm.core.impl.WCMRequestFilter

   1602 LOG Calling filter: com.adobe.cq.wcm.core.components.internal.servlets.CoreFormHandlingServlet

   1605 LOG Calling filter: com.adobe.granite.optout.impl.OptOutFilter

   1609 LOG Calling filter: com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet

   1611 LOG Calling filter: com.adobe.cq.social.commons.cors.CORSAuthenticationFilter

   1613 LOG Calling filter: org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter

   1615 LOG Calling filter: com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter

   1617 LOG Calling filter: com.day.cq.wcm.core.impl.AuthoringUIModeServiceImpl

   1733 LOG Calling filter: org.apache.sling.security.impl.ContentDispositionFilter

   1735 LOG Calling filter: com.adobe.granite.csrf.impl.CSRFFilter

   1742 LOG Calling filter: com.adobe.granite.rest.assets.impl.AssetContentDispositionFilter

   1748 LOG Calling filter: com.adobe.granite.requests.logging.impl.RequestLoggerImpl

   1755 LOG Calling filter: com.adobe.granite.rest.impl.servlet.ApiResourceFilter

   1758 LOG Calling filter: com.day.cq.dam.core.impl.servlet.ActivityRecordHandler

   1769 LOG Calling filter: com.day.cq.dam.core.impl.assetlinkshare.AdhocAssetShareAuthHandler

   1771 LOG Calling filter: com.adobe.cq.social.ugcbase.security.impl.SaferSlingPostServlet

   1773 LOG Calling filter: com.day.cq.wcm.core.impl.warp.TimeWarpFilter

   1778 LOG Applying Componentfilters

   1779 LOG Calling filter: com.day.cq.personalization.impl.TargetComponentFilter

   1781 LOG Calling filter: com.day.cq.wcm.core.impl.WCMComponentFilter

   1791 LOG Calling filter: com.day.cq.wcm.core.impl.WCMDebugFilter

   1799 TIMER_START{org.apache.sling.servlets.get.DefaultGetServlet#0}

   1919 TIMER_END{119,org.apache.sling.servlets.get.DefaultGetServlet#0}

   1957 LOG Filter timing: filter=com.day.cq.wcm.core.impl.WCMDebugFilter, inner=0, total=0, outer=0

   1959 LOG Filter timing: filter=com.day.cq.wcm.core.impl.WCMComponentFilter, inner=0, total=0, outer=0

   2008 LOG Filter timing: filter=com.adobe.cq.dam.webdav.impl.io.DamWebdavRequestFilter, inner=0, total=0, outer=0

   2010 LOG Filter timing: filter=com.day.cq.wcm.core.impl.warp.TimeWarpFilter, inner=0, total=0, outer=0

   2012 LOG Filter timing: filter=com.adobe.cq.social.ugcbase.security.impl.SaferSlingPostServlet, inner=0, total=0, outer=0

   2014 LOG Filter timing: filter=com.day.cq.dam.core.impl.assetlinkshare.AdhocAssetShareAuthHandler, inner=0, total=0, outer=0

   2015 LOG Filter timing: filter=com.day.cq.dam.core.impl.servlet.ActivityRecordHandler, inner=0, total=0, outer=0

   2017 LOG Filter timing: filter=com.day.cq.dam.core.impl.servlet.DamContentDispositionFilter, inner=0, total=0, outer=0

   2018 LOG Filter timing: filter=com.adobe.granite.rest.impl.servlet.ApiResourceFilter, inner=0, total=0, outer=0

   2019 LOG Filter timing: filter=com.adobe.granite.requests.logging.impl.RequestLoggerImpl, inner=0, total=0, outer=0

   2021 LOG Filter timing: filter=com.adobe.granite.rest.assets.impl.AssetContentDispositionFilter, inner=0, total=0, outer=0

   2022 LOG Filter timing: filter=com.adobe.granite.csrf.impl.CSRFFilter, inner=0, total=0, outer=0

   2023 LOG Filter timing: filter=org.apache.sling.security.impl.ContentDispositionFilter, inner=0, total=0, outer=0

   2024 LOG Filter timing: filter=com.day.cq.wcm.core.impl.AuthoringUIModeServiceImpl, inner=0, total=0, outer=0

   2025 LOG Filter timing: filter=com.day.cq.wcm.mobile.core.impl.redirect.RedirectFilter, inner=0, total=0, outer=0

   2027 LOG Filter timing: filter=org.apache.sling.engine.impl.debug.RequestProgressTrackerLogFilter, inner=0, total=0, outer=0

   2028 LOG Filter timing: filter=com.adobe.cq.social.commons.cors.CORSAuthenticationFilter, inner=0, total=0, outer=0

   2029 LOG Filter timing: filter=com.day.cq.wcm.foundation.forms.impl.FormsHandlingServlet, inner=0, total=0, outer=0

   2030 LOG Filter timing: filter=com.adobe.granite.optout.impl.OptOutFilter, inner=0, total=0, outer=0

   2031 LOG Filter timing: filter=com.adobe.cq.wcm.core.components.internal.servlets.CoreFormHandlingServlet, inner=0, total=0, outer=0

   2032 LOG Filter timing: filter=com.day.cq.wcm.core.impl.WCMRequestFilter, inner=0, total=0, outer=0

   2034 LOG Filter timing: filter=com.adobe.cq.history.impl.HistoryRequestFilter, inner=0, total=1, outer=1

   2035 LOG Filter timing: filter=com.adobe.cq.mcm.campaign.servlets.CampaignCopyTracker, inner=1, total=1, outer=0

   2037 LOG Filter timing: filter=org.apache.sling.rewriter.impl.RewriterFilter, inner=1, total=1, outer=0

   2039 LOG Filter timing: filter=com.adobe.granite.httpcache.impl.InnerCacheFilter, inner=1, total=1, outer=0

   2041 LOG Filter timing: filter=org.apache.sling.i18n.impl.I18NFilter, inner=1, total=1, outer=0

   2042 LOG Filter timing: filter=org.apache.sling.distribution.servlet.DistributionAgentCreationFilter, inner=1, total=1, outer=0

   2113 LOG Applying Error filters

   2115 LOG Calling filter: org.apache.sling.i18n.impl.I18NFilter

   2117 LOG Calling filter: org.apache.sling.rewriter.impl.RewriterFilter

   2123 TIMER_START{handleError:status=404}

   2138 TIMER_END{14,handleError:status=404} Using handler /libs/sling/servlet/errorhandler/404.jsp

   3817 LOG Found processor for post processing ProcessorConfiguration: {contentTypes=[text/html], order=-1, active=true, valid=true, processErrorResponse=true, pipeline=(generator=Config(type=htmlparser, config={}), transformers=(Config(type=linkchecker, config={}), Config(type=mobile, config=JcrPropertyMap [node=Node[NodeDelegate{tree=/libs/cq/config/rewriter/default/transformer-mobile: { jcr:primaryType = nt:unstructured, component-optional = true}}], values={jcr:primaryType=nt:unstructured, component-optional=true}]), Config(type=mobiledebug, config=JcrPropertyMap [node=Node[NodeDelegate{tree=/libs/cq/config/rewriter/default/transformer-mobiledebug: { jcr:primaryType = nt:unstructured, component-optional = true}}], values={jcr:primaryType=nt:unstructured, component-optional=true}]), Config(type=contentsync, config=JcrPropertyMap [node=Node[NodeDelegate{tree=/libs/cq/config/rewriter/default/transformer-contentsync: { jcr:primaryType = nt:unstructured, component-optional = true}}], values={jcr:primaryType=nt:unstructured, component-optional=true}]), serializer=Config(type=htmlwriter, config={}))}

   3924 TIMER_END{3923,Request Processing} Dumping SlingRequestProgressTracker Entries

Thanks,
Abhijeet

[Peter_Puzanovs]

Hey,

So far have not heard that machine where AEM is installed can contact needed server.

First make sure you are able to connect from point A via Proxy to Point B without any Java code.

E.g. use good old terminal and execute curl with --proxy parameter + your destination, make sure resource is accessible and no timeouts are experienced.

If it works well, only then go into Java code and implement exactly same call via java call via either day http client or your preferred http client library of choice.

Regards,

Peter

[acroabhi]

Hi Peter,

The server on which AEM is installed can talk to the external world via proxy

AEM -----> Forward Proxy---------------Internet---------------->Reverse Proxy(Loadbalancer)--->Server(REST API)

The curl call goes through.

In my java code if I set httpproxy that works too.

{

httpClientBuilder.setProxy(new HttpHost(uri.getHost(), uri.getPort(), uri.getScheme()));  //This Works

}

What I want to achieve is, I don't want to set proxy in Java code explicitly, it should just use the proxy set in AEM(Apache HTTP Components Proxy Configuration) and go through.

Is this even possible? Or setting proxy in my Java code is the only way?

Thanks,
Abhijeet

[smacdonald2008]

When a 3rd party restful service needs to be invoked - you can build a custom Service (Java code) and invoke that Service to get restful data, For example - assume you want to display 3rd party restful data in a component. This is best way.

[Gaurav-Behl]

The idea here is to use a @Reference of HTTPClient from Felix so that it could provide the proxy configurations automatically (that's my understanding) rather than creating a new connection/pool object that you have done in the code - new PoolingHttpClientConnectionManager();

If you want to go with new object approach then you would need to read the OSGI properties via ConfigurationAdmin and pass it to the HTTPClient in your source code and you need to write code for that.

Hope that helps

[acroabhi]

Hi Gaurav,

Which class should I be extending from?

When I just add

    @Reference

    private HttpClientBuilderFactory httpFactory;

it throws runtime exceptions

currently i use:

public class AEMClient extends SlingAllMethodsServlet implements OptingServlet

Thanks,
Abhijeet

[Gaurav-Behl]

This is R6+ example. Just borrow the dependencies and relevant pom.xml entries

acs-aem-commons/HttpClientFactoryImpl.java at master · Adobe-Consulting-Services/acs-aem-commons · G...