Expand my Community achievements bar.

Content migration using Crx2oak ( AEM 6.2 to 6.3)

Avatar

Level 2

Hello,

I'm trying to migration content from a 6.2 instance to a 6.3 instance using crx2oak.

I use following cmd:

./java8 -jar crx2oak-1.6.8-all-in-one.jar --cache=2048 --copy-orphaned-versions=false --copy-versions=false \

--include-paths=/content/dam/www-mysite-be,\

/content/www-mysite-be \

segment-old:/sft/aem62/author/crx-quickstart/repository/ \

/sft/aem6.3/author/crx-quickstart/repository/

But I'm bumping into below issue. It's quit strange as our 6.2 environment is using the OOTB FileDataStore (TarMK).

We have never configured any external data store nor blob store.

I've also seen that the latest crx2oak version is dated March 2017, so before AEM 6.3 was available. Could it be that there are still some bugs in that crx2oak version for AEM 6.3?

05.10.2017 21:17:06.345 ERROR  c.a.g.c.e.MigrationEngine: The internal error occurred during migration. The migration process has been aborted and it is not complete!

java.lang.RuntimeException: javax.jcr.RepositoryException: Failed to copy content

        at com.google.common.io.Closer.rethrow(Closer.java:149)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)

        at com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:67)

        at com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:92)

        at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)

        at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)

        at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)

Caused by: javax.jcr.RepositoryException: Failed to copy content

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:292)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:248)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)

        ... 5 common frames omitted

Caused by: java.lang.IllegalStateException: Attempt to read external blob with blobId [a96161363d0d21841c29643c71154d8aa9083497#140385] without specifying BlobStore

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getBlob(SegmentBlob.java:247)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getLength(SegmentBlob.java:256)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.length(SegmentBlob.java:112)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:52)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:228)

        at com.google.common.base.Objects.equal(Objects.java:60)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:243)

Thanks,

Johnny

25 Replies

Avatar

Level 10

Internal Adobe ppl suggested:

They have to verify that file a96161363d0d21841c29643c71154d8aa9083497 exists in datastore

Caused by: java.lang.IllegalStateException: Attempt to read external blob with blobId [a96161363d0d21841c29643c71154d8aa9083497#140385] without specifying BlobStore

A good idea for the customer is to run Datastore GC. If anything is missing copy from backup

Avatar

Level 2

I already tried to perform an offline compaction first on the source repository. This one will perform a TarMK revision cleanup.

This compaction ran fine, but the migration with crx2oak afterwards still gives the same exception.

Johnny

Avatar

Level 1

Did you ever find a way to resolve this?

I'm facing same issue and running out of ideas.

Thanks,
Alex

Avatar

Level 10

If the docs do not work and this suggestion - i recommend opening a support ticket so our Support Engs can investigate.

Avatar

Level 2

Hi,

Thank you for feedback. Yes, I followed all steps.

I have been using crx2oak for quit a while between many different versions of CQ/AEM.

Only in this specific case (6.2 --> 6.3), I stumble on this issue, but only in the author instance.

I also don't know if the exception is related to the source or destination repository.

Johnny

Avatar

Level 9

Hi,

Sounds SegmentNodeStoreService.(cfg|config) not picked up .

use fileinstaller tool to fix the corruption of RegisteredResourceList.ser

Get the tool from support.

Thanks,

Avatar

Level 1

MC Stuff,

That's exactly what happened in my case. I "resolved" it by deleting org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore from OSGI console and then it was able to pickup a new configuration.

Now, “resolved” is under quotes because it seemed like it’s fine, I was able to see packages under packMgr etc but now, when I try to open anything on the publisher I get:

500 Unresolved compilation problems: Only a type can be imported. com.day.cq.wcm.foundation.ELEvaluator resolves to a package ELEvaluator cannot be resolved

Cannot serve request to /content/admission/statistics.html on this server

This is very bizarre because I did the exactly same thing with the author and author is working just fine, did not have any issues with it.

When it comes to the problem above, I tried recompiling JSP, removing /var/classes but no luck.

Avatar

Level 9

Symptoms indicates the current instance state have few bundle in two different store & hence all class info not available. Take a backup . Stop instance & delete launchpad . Start instance & verify.

Avatar

Level 2

Hi all,

FYI, I did log a ticket on daycare and after 3 weeks I got this last note:

Hello Johnny,

Hope you are doing well.

I researched more on this error and could relate to one of OAK issue that could cause such behavior hence lets give it a try to use below steps.

1./ Referencing https://issues.apache.org/jira/browse/OAK-5255 for more flags , Can you try to run crx2oak explicitly passing the "--copy-binaries " to false? and see how it goes.

2./ If above does not help can you specify the datastore of the source instance, and it would be better if the destination also has the datastore, an example command would be:

java -Xmx12G -jar crx2oak-1.4.6-standalone.jar crx-quickstart/repository/ C:/Dev/CQ/Instances/AEM62/crx-quickstart/repository --src-datastore=crx-quickstart/repository/repository/datastore/ --datastore=C:/Dev/CQ/Instances/AEM62/crx-quickstart/repository/datastore/ --include-path=/content --copy-versions=false --merge-path=/content

Just to confirm, make sure that the repositories (source 6.2 and destination 6.3) should be shut while running the crx2oak migration.

If nothing helps share below:

Oak core on source

Oak core on destination

Package your content from "/content/dam/www-mysite-be" and give to me so that I can run the migration locally.

regards,

This didn't result in a solution. We did already try to upgrade the target to the latest oak version (1.4.9), we did try with --copy-binaries=false, we did find the comment about --src-datasource strange as we don't have a separate datastore in 6.2, ....

So unfortunately not very helpful.

From above replies I can try following, if my understanding is correct:

1) SegmentNodeStoreService.(cfg|config) not picked up --> not sure if this is on source or target and I never heard about a fileinstaller tool or RegisteredResourceList.ser.

2) delete org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore osgi config --> I assume on the source repository?

Avatar

Community Advisor

Hi,

I have recently successfully performed content migration from 6.1 to 6.3.

Just want you to try this (might be silly answer),Repository path without ending slash and target repository absolute path (i.e D:/sft/aem6.3/author/crx-quickstart/repository)

java8 -jar crx2oak-1.6.8-all-in-one.jar --cache=2048 --copy-orphaned-versions=false --copy-versions=false --include-paths=/content/dam/www-mysite-be,/content/www-mysite-be segment-old:/sft/aem62/author/crx-quickstart/repository /sft/aem6.3/author/crx-quickstart/repository

Avatar

Level 2

Just tried without trailing slash on the source and both locations were already absolute. The result is the same:

14.12.2017 15:40:24.370 ERROR  c.a.g.c.e.MigrationEngine: The internal error occurred during migration. The migration process has been aborted and it is not complete!

java.lang.RuntimeException: javax.jcr.RepositoryException: Failed to copy content

        at com.google.common.io.Closer.rethrow(Closer.java:149)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)

        at com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:67)

        at com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:92)

        at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)

        at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)

        at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)

Caused by: javax.jcr.RepositoryException: Failed to copy content

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:292)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:248)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)

        ... 5 common frames omitted

Caused by: java.lang.IllegalStateException: Attempt to read external blob with blobId [5508b7b91e8a316f7da92eb43c658631ba324cb347a933e39c1e4f4432643a4a#31424] without specifying BlobStore

Avatar

Level 2

In your migration from 6.1 to 6.3 did you also see these crx2oak statements about DummyBlobStore on source and target?

14.12.2017 15:40:21.648 INFO   o.a.j.o.u.c.p.DatastoreArguments: Source blob store: DummyBlobStore

14.12.2017 15:40:22.235 INFO   o.a.j.o.p.s.f.FileStore: TarMK ReadOnly opened: /sft/aem62/author/crx-quickstart/repository/segmentstore (mmap=true)

14.12.2017 15:40:22.263 INFO   o.a.j.o.p.s.SegmentNodeStore: Initializing SegmentNodeStore with the commitFairLock option enabled.

14.12.2017 15:40:22.460 INFO   o.a.j.o.u.c.p.DatastoreArguments: Destination blob store: DummyBlobStore

Avatar

Community Advisor

No I haven't seen DummyBlonStore error.

Do you have separate datastore for your repository?
try to add --datastore datastorePath()/crx-quickstart/repository/datastore)

Avatar

Level 2

No, that's the strange thing, I don't have a separate datastore or blobstore, not in source and not in target.

The statement to define the blobstore inline also seems correct:

14.12.2017 16:07:53.009 INFO   o.a.j.o.s.SegmentNodeStore$SegmentNodeStoreBuilder: Creating segment node store SegmentNodeStoreBuilder{blobStore=inline}

Avatar

Level 2

On slide 15 I also read something about a BlobId:

https://www.slideshare.net/TomaszRkawek/crx2oak-all-the-secrets-of-repository-migration/18

They advise to configure BlobStore properly, but as I don't have any external datastore, I'm not sure how to do this.

Avatar

Level 2

FYI, if I specify an external file datastore on the target repo, then it works!

--datastore=/sft/aem6.3/author/crx-quickstart/repository/datastore/

But this means I also have to create:

crx-quickstart/install/org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config

with content:

repository.home="crx-quickstart/repository"

tarmk.size=I"512"

customBlobStore=B"true"

What do you guys think? Is this a good approach to migrate from a 6.2 node store that also contains the binaries to a 6.3 node store with separate data store for binaries?

Avatar

Level 9

Hi Brijesh,

i am upgrading 6.2 repository(File Datastore) to 6.4.How do i confirm my upgrading is successful?

Is there any checklist to be verified?

Thanks,

GK

Avatar

Level 9

Thanks Brijesh.

I am facing an issue when doing migration from 6.2 to 6.4.i don't have any external repo but still getting below error.

Any idea, how to fix this issue?

24.04.2018 20:06:18.484 ERROR  c.a.g.c.e.MigrationEngine: The internal error occurred during migration. The migration process has been aborted and it is not complete!

java.lang.RuntimeException: javax.jcr.RepositoryException: Failed to copy content

        at com.google.common.io.Closer.rethrow(Closer.java:149)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)

        at com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:67)

        at com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:92)

        at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)

        at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)

        at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)

Caused by: javax.jcr.RepositoryException: Failed to copy content

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:285)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)

        ... 5 common frames omitted

Caused by: java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [66f1d39a64998bf2e641765057509c086f2474202d9fd24fa654dabc01aba988#95174]

        at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:248)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:85)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44)

        at com.google.common.io.ByteSource.contentEquals(ByteSource.java:345)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:227)

        at com.google.common.base.Objects.equal(Objects.java:60)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:242)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyProperties(NodeStateCopier.java:144)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:212)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:160)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.access$200(NodeStateCopier.java:75)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier$Builder.copy(NodeStateCopier.java:465)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copyWorkspace(RepositorySidegrade.java:468)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.migrateWithoutCheckpoints(RepositorySidegrade.java:417)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copyState(RepositorySidegrade.java:317)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:278)

        ... 7 common frames omitted

Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 66f1d39a64998bf2e641765057509c086f2474202d9fd24fa654dabc01aba988 does not exist

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:589)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:342)

        at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)

        ... 30 common frames omitted

Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record 66f1d39a64998bf2e641765057509c086f2474202d9fd24fa654dabc01aba988 does not exist

        at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:58)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:598)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:583)

Thanks,

Kishore