AEM 5.6.1 is crashing some time

[Chandra_gupta]

Spring OSGI services make Web services call to web sphere based external  application and lot of 500 and 503 error happens in OSGI Java call with whatever reason from back end web services.. Spring classes from OSGI to Web services is based on https protocal (Secure)..

Manually testing on CQ5 server works very well CQ5 pull information from Web services all works well.

Automatic load testing on CQ like 200 concurrent users for an hours also looks good but when same testing carry our few times in week ...Very Randomly any one of CQ5 publish get crashed and doesn't show any evidence like PermGen, Out of memeory or Diskspace. Log also doesn't have any evident why it crashes ... I see only three kind of error ...500,503 and this SSL exception

0.09.2013 15:31:23.582 *WARN* [172.22.20.105 [1378845083565] <parse>] servletengine I/O error in execution: {} javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 88
    at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1476)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:92)
    at java.io.InputStream.read(InputStream.java:101)
    at com.day.j2ee.servletengine.ServletHandlerImpl.parseRequest(ServletHandlerImpl.java:450)
    at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:301)
    at com.day.j2ee.servletengine.HttpListener$Worker.run(HttpListener.java:644)
    at java.lang.Thread.run(Thread.java:722)
Caused by: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 88
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:974)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
    at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2173)
    at com.day.j2ee.servletengine.ServletHandlerImpl.getCipherSuite(ServletHandlerImpl.java:978)
    at com.day.j2ee.servletengine.RequestImpl.recycle(RequestImpl.java:269)
    at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:289)
    ... 2 more
Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 88
    at sun.security.ssl.CipherBox.removePadding(CipherBox.java:684)
    at sun.security.ssl.CipherBox.decrypt(CipherBox.java:423)
    at sun.security.ssl.InputRecord.decrypt(InputRecord.java:154)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:969)
    ... 8 more
10.09.2013 15:31:23.692 *WARN* [172.22.20.105 [1378845083692] GET /servic except 500, 503 and this SSL exception.

My doubt is on this SSL exception is crashing CQ5 JVM..  Please share your experience.

Thanks Chandra

[smacdonald2008]

Can you please provide more information on what you are trying to perform here. 

[AshokPelluru]

Getting same error

<parse>] servletengine I/O error in execution: {} javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 194

                at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source)

                at sun.security.ssl.AppInputStream.read(Unknown Source)

                at java.io.InputStream.read(Unknown Source)

                at com.day.j2ee.servletengine.ServletHandlerImpl.parseRequest(ServletHandlerImpl.java:466)

                at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:317)

                at com.day.j2ee.servletengine.HttpListener$Worker.run(HttpListener.java:625)

                at java.lang.Thread.run(Unknown Source)

Caused by: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 194

                at sun.security.ssl.Alerts.getSSLException(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)

                at com.day.j2ee.servletengine.ServletHandlerImpl.getCipherSuite(ServletHandlerImpl.java:947)

                at com.day.j2ee.servletengine.RequestImpl.recycle(RequestImpl.java:272)

                at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:309)

                ... 2 more

Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 194

                at sun.security.ssl.CipherBox.removePadding(Unknown Source)

                at sun.security.ssl.CipherBox.decrypt(Unknown Source)

                at sun.security.ssl.InputRecord.decrypt(Unknown Source)

any help?

[AshokPelluru]

Getting same Error

<parse>] servletengine I/O error in execution: {} javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 194

                at sun.security.ssl.SSLSocketImpl.checkEOF(Unknown Source)

                at sun.security.ssl.AppInputStream.read(Unknown Source)

                at java.io.InputStream.read(Unknown Source)

                at com.day.j2ee.servletengine.ServletHandlerImpl.parseRequest(ServletHandlerImpl.java:466)

                at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:317)

                at com.day.j2ee.servletengine.HttpListener$Worker.run(HttpListener.java:625)

                at java.lang.Thread.run(Unknown Source)

Caused by: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 194

                at sun.security.ssl.Alerts.getSSLException(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)

                at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)

                at com.day.j2ee.servletengine.ServletHandlerImpl.getCipherSuite(ServletHandlerImpl.java:947)

                at com.day.j2ee.servletengine.RequestImpl.recycle(RequestImpl.java:272)

                at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:309)

                ... 2 more

Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 194

                at sun.security.ssl.CipherBox.removePadding(Unknown Source)

                at sun.security.ssl.CipherBox.decrypt(Unknown Source)

                at sun.security.ssl.InputRecord.decrypt(Unknown Source)

[Chandra_gupta]

Its Spring OSGI bundle that has certain spring based custom java services  makes https call to extrenal price related web services ..consumes JSON outpout.

[Peter_Puzanovs · Accepted solution]

Try AppDynamics, it allows you to monitor the transaction path in details, when fails you can dig deep and figure out the exact method that fails and exact path it fails on.

[Chandra_gupta]

I have almost identified issue.

Oracle JDk 7.21 for linux has bug that carch JVM when java class establishes https connection to any external services. this does not happen over http.

Oracle JDK 7.25 has fixed this issue... we are in process of applying that patch and update this forum.

Thanks Chandra
 

[Chandra_gupta]

JDK update has not helped..

When I get CQ running on http and external web services since CQ is sending non-encrypted request packet  to extreanl web services and that has no issue  but as soon i get working CQ on HTTPS... after every few hundred call to web services CQ logs shows this exception

------------------------------------------------------------------------------------------------

8.09.2013 16:12:17.985 *INFO* [10.100.183.102 [1379538737984] G 18.09.2013 16:12:18.481 *WARN* [10.100.183.102 [1379538738465] <parse>] servletengine I/O error in execution: {} javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 156 at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1476) at sun.security.ssl.AppInputStream.read(AppInputStream.java:92) at java.io.InputStream.read(InputStream.java:101) at com.day.j2ee.servletengine.ServletHandlerImpl.parseRequest(ServletHandlerImpl.java:450) at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:301) at com.day.j2ee.servletengine.HttpListener$Worker.run(HttpListener.java:644) at java.lang.Thread.run(Thread.java:724) Caused by: javax.net.ssl.SSLHandshakeException: Invalid Padding length: 156 at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:974) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) at sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:2173) at com.day.j2ee.servletengine.ServletHandlerImpl.getCipherSuite(ServletHandlerImpl.java:978) at com.day.j2ee.servletengine.RequestImpl.recycle(RequestImpl.java:269) at com.day.j2ee.servletengine.ServletHandlerImpl.process(ServletHandlerImpl.java:289) ... 2 more Caused by: javax.crypto.BadPaddingException: Invalid Padding length: 156 at sun.security.ssl.CipherBox.removePadding(CipherBox.java:684) at sun.security.ssl.CipherBox.decrypt(CipherBox.java:423) at sun.security.ssl.InputRecord.decrypt(InputRecord.java:154) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:969) ... 8 more 18.09.2013 16:12:18.577 *WARN* [10.100.183.102 [1379538738576] GET /services

------------------------------------------------------------------------------------------------------

Defect has been reported oracle bug

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=8013059

I am thinking to apply RSA cipher in CQ... but not sure how to do that?