AEM 6.3 Package Installation issue

[#1]

Hi Team,

While installing our Project apps package into AEM 6.3.2.1, getting below error.  Not Sure how to resolve this. This is happening only from last week. This issue was not there earlier. Also, This is happening only in Dev environment. In QA and local environments this issue not there. Any pointers to resolve this issue is really appreciated.

*ERROR* [qtp546633110-13119] org.apache.jackrabbit.vault.fs.io.Importer Error while committing /apps/testproject/clientlibs/clientlib-site/resources/fonts/akzidenzgrotesk_bq_regular.woff: javax.jcr.RepositoryException: Inaccessible value

######################################################################################

10.09.2018 13:08:45.513 *ERROR* [qtp546633110-12911] org.apache.jackrabbit.vault.packaging.impl.ZipVaultPackage Error during install.

javax.jcr.RepositoryException: Inaccessible value

        at org.apache.jackrabbit.oak.plugins.value.ErrorValue.createException(ErrorValue.java:94)

        at org.apache.jackrabbit.oak.plugins.value.ErrorValue.getBinary(ErrorValue.java:60)

        at org.apache.jackrabbit.oak.plugins.value.ValueImpl.getBlob(ValueImpl.java:54)

        at org.apache.jackrabbit.oak.plugins.memory.BinaryPropertyState.binaryProperty(BinaryPropertyState.java:83)

        at org.apache.jackrabbit.oak.plugins.memory.PropertyStates.createProperty(PropertyStates.java:62)

        at org.apache.jackrabbit.oak.jcr.session.ItemImpl.createSingleState(ItemImpl.java:366)

        at org.apache.jackrabbit.oak.jcr.session.NodeImpl.internalSetProperty(NodeImpl.java:1362)

        at org.apache.jackrabbit.oak.jcr.session.NodeImpl.setProperty(NodeImpl.java:352)

        at org.apache.jackrabbit.vault.fs.impl.io.FileArtifactHandler.importNtResource(FileArtifactHandler.java:362)

        at org.apache.jackrabbit.vault.fs.impl.io.FileArtifactHandler.importFile(FileArtifactHandler.java:315)

        at org.apache.jackrabbit.vault.fs.impl.io.FileArtifactHandler.accept(FileArtifactHandler.java:176)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:927)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:784)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.commit(Importer.java:824)

        at org.apache.jackrabbit.vault.fs.io.Importer.run(Importer.java:425)

        at org.apache.jackrabbit.vault.packaging.impl.ZipVaultPackage.extract(ZipVaultPackage.java:233)

        at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:398)

        at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.extract(JcrPackageImpl.java:357)

        at org.apache.jackrabbit.vault.packaging.impl.JcrPackageImpl.install(JcrPackageImpl.java:351)

        at com.day.crx.packaging.impl.J2EEPackageManager.consoleInstall(J2EEPackageManager.java:353)

        at com.day.crx.packaging.impl.J2EEPackageManager.doPost(J2EEPackageManager.java:191)

        at com.day.crx.packaging.impl.PackageManagerServlet.doPost(PackageManagerServlet.java:128)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:644)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)

        at org.apache.felix.http.base.internal.handler.ServletHandler.handle(ServletHandler.java:85)

        at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:79)

        at com.adobe.granite.license.impl.LicenseCheckFilter.doFilter(LicenseCheckFilter.java:308)

        at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)

        at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)

        at org.apache.felix.http.sslfilter.internal.SslFilter.doFilter(SslFilter.java:96)

        at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)

        at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)

        at org.apache.sling.security.impl.ReferrerFilter.doFilter(ReferrerFilter.java:295)

        at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)

        at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)

        at org.apache.sling.i18n.impl.I18NFilter.doFilter(I18NFilter.java:138)

        at org.apache.felix.http.base.internal.handler.FilterHandler.handle(FilterHandler.java:135)

        at org.apache.felix.http.base.internal.dispatch.InvocationChain.doFilter(InvocationChain.java:74)

        at org.apache.felix.http.base.internal.dispatch.Dispatcher.dispatch(Dispatcher.java:127)

        at org.apache.felix.http.base.internal.dispatch.DispatcherServlet.service(DispatcherServlet.java:49)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

        at org.eclipse.jetty.server.Server.handle(Server.java:499)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)

        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Could not add record

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.writeBlob(DataStoreBlobStore.java:239)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.writeBlob(DataStoreBlobStore.java:215)

        at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.internalWriteStream(SegmentWriter.java:827)

        at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeStream(SegmentWriter.java:798)

        at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$500(SegmentWriter.java:365)

        at org.apache.jackrabbit.oak.segment.SegmentWriter$6.execute(SegmentWriter.java:278)

        at org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:106)

        at org.apache.jackrabbit.oak.segment.SegmentWriter.writeStream(SegmentWriter.java:274)

        at org.apache.jackrabbit.oak.segment.SegmentNodeStore.createBlob(SegmentNodeStore.java:388)

        at org.apache.jackrabbit.oak.core.MutableRoot.createBlob(MutableRoot.java:322)

        at org.apache.jackrabbit.oak.plugins.value.ValueFactoryImpl.createBinaryValue(ValueFactoryImpl.java:297)

        at org.apache.jackrabbit.oak.plugins.value.ValueFactoryImpl.createValue(ValueFactoryImpl.java:151)

        at org.apache.jackrabbit.vault.fs.impl.io.FileArtifactHandler.importNtResource(FileArtifactHandler.java:359)

        ... 56 common frames omitted

Caused by: org.apache.jackrabbit.core.data.DataStoreException: Could not add record

        at org.apache.jackrabbit.core.data.FileDataStore.addRecord(FileDataStore.java:230)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.addRecord(DataStoreBlobStore.java:538)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.writeStream(DataStoreBlobStore.java:637)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.writeBlob(DataStoreBlobStore.java:224)

        ... 68 common frames omitted

Thanks

k Chaitanya

[#2 · Accepted solution]

After talking to internal ppl - they said:

"This is a permission issue where process can't write binaries to the datastore locations. Check the owner and permissions for /crx-quickstart/repository/datastore and java /tmp location to see if something changed lately."

[rajramo]

Hi @krishna_chaita2 could you please elaborate how did you able to fix this issue ? it would be helpful 

[rajramo]

I am adding comment base on my experience with this issue.
JDK: Java-11
AEM: Cloud SDK (latest from April-22)
I attached the debugger like I used to attach it in Java-8.
I noticed following error when mavn build was trying to install the package.

Request failed: javax.jcr.RepositoryException: Inaccessible value (500)
I tried to install the package using curl and manually but I noticed the same error.

I stopped AEM and tried to run AEM without debugger and it worked fine. Then I searched and found that attaching debugger to jvm has changed as from version 9 and above. I followed the following link to attach the debugger correctly with Java 11 and things started working fine.

Run AEM in debug mode: Java-8
java -debug -XX:MaxMetaspaceSize=512M -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=3030 -jar aem-author-p4502.jar -nofork

Run AEM in debug mode: Java-11 (Java 9 and above, you need to specify IP along with port number)
java -debug -XX:MaxMetaspaceSize=512M -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=127.0.0.1:3030 -jar aem-author-p4502.jar

Hope this helps.

[#5]

You will also get this issue when the disk is full.

[#6]

I am glad its working.

[#7]

Hi Aneet,

One of the internal folder (/6f/fe/da/) under datastore lost permission. I granted the required permissions. Then it started working. Thanks for pointing me in the right direction

Thanks

K Chaitanya

[#8]

Hi Samcdonald,

One of the internal folder (/6f/fe/da/) under datastore lost permission. I granted the required permissions. Then it started working. Thanks for pointing me in the right direction

Thanks

K Chaitanya

[#9]

There can be many reasons why something on the AEM DEV instance broke or changed. I am going to talk to internal ppl about this.

[#10]

Hi , It is working in Other Environments QA and local. It is not working Only in Dev. I agree  issue is with AEM Dev instance. But what is the resolution?

[#11]

Did you try to install this package on a fresh AEM instance as I suggested. If it works on a fresh instance - issue is your DEV AEM instance.

[#12]

Hi Aneet,

I verified Permissions are Proper. Still It is not working

Thanks

K Chaitanya

[#13]

Hello Krishna,

The error "Could not add record" typically indicates a permissions issue with the datastore. You may have started AEM with a user which doesn't have permissions to write to the datastore directory. Can you please check?

Thanks,

Aneet

[#14]

Can you try this package installation on a fresh AEM DEV instance. This will tell us if the issue is with package or the AEM instance.