Content migration using Crx2oak ( AEM 6.2 to 6.3)

[johnnyd26246076]

Hello,

I'm trying to migration content from a 6.2 instance to a 6.3 instance using crx2oak.

I use following cmd:

./java8 -jar crx2oak-1.6.8-all-in-one.jar --cache=2048 --copy-orphaned-versions=false --copy-versions=false \

--include-paths=/content/dam/www-mysite-be,\

/content/www-mysite-be \

segment-old:/sft/aem62/author/crx-quickstart/repository/ \

/sft/aem6.3/author/crx-quickstart/repository/

But I'm bumping into below issue. It's quit strange as our 6.2 environment is using the OOTB FileDataStore (TarMK).

We have never configured any external data store nor blob store.

I've also seen that the latest crx2oak version is dated March 2017, so before AEM 6.3 was available. Could it be that there are still some bugs in that crx2oak version for AEM 6.3?

05.10.2017 21:17:06.345 ERROR  c.a.g.c.e.MigrationEngine: The internal error occurred during migration. The migration process has been aborted and it is not complete!

java.lang.RuntimeException: javax.jcr.RepositoryException: Failed to copy content

        at com.google.common.io.Closer.rethrow(Closer.java:149)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)

        at com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:67)

        at com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:92)

        at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)

        at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)

        at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)

Caused by: javax.jcr.RepositoryException: Failed to copy content

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:292)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:248)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)

        ... 5 common frames omitted

Caused by: java.lang.IllegalStateException: Attempt to read external blob with blobId [a96161363d0d21841c29643c71154d8aa9083497#140385] without specifying BlobStore

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getBlob(SegmentBlob.java:247)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getLength(SegmentBlob.java:256)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.length(SegmentBlob.java:112)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:52)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:228)

        at com.google.common.base.Objects.equal(Objects.java:60)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:243)

Thanks,

Johnny

[smacdonald2008]

Did you follow all steps listed here?

Using the CRX2Oak Migration Tool

[smacdonald2008]

Internal Adobe ppl suggested:

They have to verify that file a96161363d0d21841c29643c71154d8aa9083497 exists in datastore

Caused by: java.lang.IllegalStateException: Attempt to read external blob with blobId [a96161363d0d21841c29643c71154d8aa9083497#140385] without specifying BlobStore

A good idea for the customer is to run Datastore GC. If anything is missing copy from backup

[johnnyd26246076]

I already tried to perform an offline compaction first on the source repository. This one will perform a TarMK revision cleanup.

This compaction ran fine, but the migration with crx2oak afterwards still gives the same exception.

Johnny

[alexp17]

Did you ever find a way to resolve this?

I'm facing same issue and running out of ideas.

Thanks,
Alex

[smacdonald2008]

If the docs do not work and this suggestion - i recommend opening a support ticket so our Support Engs can investigate.

[johnnyd26246076]

Hi,

Thank you for feedback. Yes, I followed all steps.

I have been using crx2oak for quit a while between many different versions of CQ/AEM.

Only in this specific case (6.2 --> 6.3), I stumble on this issue, but only in the author instance.

I also don't know if the exception is related to the source or destination repository.

Johnny

[MC_Stuff]

Hi,

Sounds SegmentNodeStoreService.(cfg|config) not picked up .

use fileinstaller tool to fix the corruption of RegisteredResourceList.ser

Get the tool from support.

Thanks,

[alexp17]

MC Stuff,

That's exactly what happened in my case. I "resolved" it by deleting org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore from OSGI console and then it was able to pickup a new configuration.

Now, “resolved” is under quotes because it seemed like it’s fine, I was able to see packages under packMgr etc but now, when I try to open anything on the publisher I get:

500 Unresolved compilation problems: Only a type can be imported. com.day.cq.wcm.foundation.ELEvaluator resolves to a package ELEvaluator cannot be resolved

Cannot serve request to /content/admission/statistics.html on this server

This is very bizarre because I did the exactly same thing with the author and author is working just fine, did not have any issues with it.

When it comes to the problem above, I tried recompiling JSP, removing /var/classes but no luck.

[MC_Stuff]

Symptoms indicates the current instance state have few bundle in two different store & hence all class info not available. Take a backup . Stop instance & delete launchpad . Start instance & verify.

[johnny_degreef]

Hi all,

FYI, I did log a ticket on daycare and after 3 weeks I got this last note:

Hello Johnny,

Hope you are doing well.

I researched more on this error and could relate to one of OAK issue that could cause such behavior hence lets give it a try to use below steps.

1./ Referencing https://issues.apache.org/jira/browse/OAK-5255 for more flags , Can you try to run crx2oak explicitly passing the "--copy-binaries " to false? and see how it goes.

2./ If above does not help can you specify the datastore of the source instance, and it would be better if the destination also has the datastore, an example command would be:

java -Xmx12G -jar crx2oak-1.4.6-standalone.jar crx-quickstart/repository/ C:/Dev/CQ/Instances/AEM62/crx-quickstart/repository --src-datastore=crx-quickstart/repository/repository/datastore/ --datastore=C:/Dev/CQ/Instances/AEM62/crx-quickstart/repository/datastore/ --include-path=/content --copy-versions=false --merge-path=/content

Just to confirm, make sure that the repositories (source 6.2 and destination 6.3) should be shut while running the crx2oak migration.

If nothing helps share below:

Oak core on source

Oak core on destination

Package your content from "/content/dam/www-mysite-be" and give to me so that I can run the migration locally.

regards,

This didn't result in a solution. We did already try to upgrade the target to the latest oak version (1.4.9), we did try with --copy-binaries=false, we did find the comment about --src-datasource strange as we don't have a separate datastore in 6.2, ....

So unfortunately not very helpful.

From above replies I can try following, if my understanding is correct:

1) SegmentNodeStoreService.(cfg|config) not picked up --> not sure if this is on source or target and I never heard about a fileinstaller tool or RegisteredResourceList.ser.

2) delete org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore osgi config --> I assume on the source repository?

[BrijeshYadav]

Hi,

I have recently successfully performed content migration from 6.1 to 6.3.

Just want you to try this (might be silly answer),Repository path without ending slash and target repository absolute path (i.e D:/sft/aem6.3/author/crx-quickstart/repository)

java8 -jar crx2oak-1.6.8-all-in-one.jar --cache=2048 --copy-orphaned-versions=false --copy-versions=false --include-paths=/content/dam/www-mysite-be,/content/www-mysite-be segment-old:/sft/aem62/author/crx-quickstart/repository /sft/aem6.3/author/crx-quickstart/repository

[johnny_degreef]

Just tried without trailing slash on the source and both locations were already absolute. The result is the same:

14.12.2017 15:40:24.370 ERROR  c.a.g.c.e.MigrationEngine: The internal error occurred during migration. The migration process has been aborted and it is not complete!

java.lang.RuntimeException: javax.jcr.RepositoryException: Failed to copy content

        at com.google.common.io.Closer.rethrow(Closer.java:149)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)

        at com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:67)

        at com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:92)

        at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)

        at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)

        at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)

Caused by: javax.jcr.RepositoryException: Failed to copy content

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:292)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:248)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)

        ... 5 common frames omitted

Caused by: java.lang.IllegalStateException: Attempt to read external blob with blobId [5508b7b91e8a316f7da92eb43c658631ba324cb347a933e39c1e4f4432643a4a#31424] without specifying BlobStore

[johnny_degreef]

In your migration from 6.1 to 6.3 did you also see these crx2oak statements about DummyBlobStore on source and target?

14.12.2017 15:40:21.648 INFO   o.a.j.o.u.c.p.DatastoreArguments: Source blob store: DummyBlobStore

14.12.2017 15:40:22.235 INFO   o.a.j.o.p.s.f.FileStore: TarMK ReadOnly opened: /sft/aem62/author/crx-quickstart/repository/segmentstore (mmap=true)

14.12.2017 15:40:22.263 INFO   o.a.j.o.p.s.SegmentNodeStore: Initializing SegmentNodeStore with the commitFairLock option enabled.

14.12.2017 15:40:22.460 INFO   o.a.j.o.u.c.p.DatastoreArguments: Destination blob store: DummyBlobStore

[BrijeshYadav]

No I haven't seen DummyBlonStore error.

Do you have separate datastore for your repository?
try to add --datastore datastorePath()/crx-quickstart/repository/datastore)

[johnny_degreef]

No, that's the strange thing, I don't have a separate datastore or blobstore, not in source and not in target.

The statement to define the blobstore inline also seems correct:

14.12.2017 16:07:53.009 INFO   o.a.j.o.s.SegmentNodeStore$SegmentNodeStoreBuilder: Creating segment node store SegmentNodeStoreBuilder{blobStore=inline}

[johnny_degreef]

On slide 15 I also read something about a BlobId:

https://www.slideshare.net/TomaszRkawek/crx2oak-all-the-secrets-of-repository-migration/18​

They advise to configure BlobStore properly, but as I don't have any external datastore, I'm not sure how to do this.

[johnny_degreef]

FYI, if I specify an external file datastore on the target repo, then it works!

--datastore=/sft/aem6.3/author/crx-quickstart/repository/datastore/

But this means I also have to create:

crx-quickstart/install/org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config

with content:

repository.home="crx-quickstart/repository"

tarmk.size=I"512"

customBlobStore=B"true"

What do you guys think? Is this a good approach to migrate from a 6.2 node store that also contains the binaries to a 6.3 node store with separate data store for binaries?

[GK-AEM]

Hi Brijesh,

i am upgrading 6.2 repository(File Datastore) to 6.4.How do i confirm my upgrading is successful?

Is there any checklist to be verified?

Thanks,

GK

[BrijeshYadav]

Here is post upgrade checklist.
Post Upgrade Checks and Troubleshooting

/Brijesh

[GK-AEM]

Thanks Brijesh.

I am facing an issue when doing migration from 6.2 to 6.4.i don't have any external repo but still getting below error.

Any idea, how to fix this issue?

24.04.2018 20:06:18.484 ERROR  c.a.g.c.e.MigrationEngine: The internal error occurred during migration. The migration process has been aborted and it is not complete!

java.lang.RuntimeException: javax.jcr.RepositoryException: Failed to copy content

        at com.google.common.io.Closer.rethrow(Closer.java:149)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:81)

        at com.adobe.granite.crx2oak.engine.MigrationEngine$2.doMigration(MigrationEngine.java:67)

        at com.adobe.granite.crx2oak.engine.MigrationEngine.process(MigrationEngine.java:92)

        at com.adobe.granite.crx2oak.pipeline.Pipeline.run(Pipeline.java:103)

        at com.adobe.granite.crx2oak.CRX2Oak.run(CRX2Oak.java:66)

        at com.adobe.granite.crx2oak.CRX2Oak.main(CRX2Oak.java:51)

Caused by: javax.jcr.RepositoryException: Failed to copy content

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:285)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.sidegrade(OakUpgrade.java:92)

        at org.apache.jackrabbit.oak.upgrade.cli.OakUpgrade.migrate(OakUpgrade.java:78)

        ... 5 common frames omitted

Caused by: java.lang.RuntimeException: Error occurred while obtaining InputStream for blobId [66f1d39a64998bf2e641765057509c086f2474202d9fd24fa654dabc01aba988#95174]

        at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:49)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:248)

        at org.apache.jackrabbit.oak.segment.SegmentBlob.getNewStream(SegmentBlob.java:85)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob$1.openStream(AbstractBlob.java:44)

        at com.google.common.io.ByteSource.contentEquals(ByteSource.java:345)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractBlob.equal(AbstractBlob.java:67)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentBlob.equals(SegmentBlob.java:227)

        at com.google.common.base.Objects.equal(Objects.java:60)

        at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:59)

        at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.equals(SegmentPropertyState.java:242)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyProperties(NodeStateCopier.java:144)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:212)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:208)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.copyNodeState(NodeStateCopier.java:160)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier.access$200(NodeStateCopier.java:75)

        at org.apache.jackrabbit.oak.plugins.migration.NodeStateCopier$Builder.copy(NodeStateCopier.java:465)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copyWorkspace(RepositorySidegrade.java:468)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.migrateWithoutCheckpoints(RepositorySidegrade.java:417)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copyState(RepositorySidegrade.java:317)

        at org.apache.jackrabbit.oak.upgrade.RepositorySidegrade.copy(RepositorySidegrade.java:278)

        ... 7 common frames omitted

Caused by: java.io.IOException: org.apache.jackrabbit.core.data.DataStoreException: Record 66f1d39a64998bf2e641765057509c086f2474202d9fd24fa654dabc01aba988 does not exist

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:589)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getInputStream(DataStoreBlobStore.java:342)

        at org.apache.jackrabbit.oak.plugins.blob.BlobStoreBlob.getNewStream(BlobStoreBlob.java:47)

        ... 30 common frames omitted

Caused by: org.apache.jackrabbit.core.data.DataStoreException: Record 66f1d39a64998bf2e641765057509c086f2474202d9fd24fa654dabc01aba988 does not exist

        at org.apache.jackrabbit.core.data.AbstractDataStore.getRecord(AbstractDataStore.java:58)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getDataRecord(DataStoreBlobStore.java:598)

        at org.apache.jackrabbit.oak.plugins.blob.datastore.DataStoreBlobStore.getStream(DataStoreBlobStore.java:583)

Thanks,

Kishore