Crx2oak migration from AEM 6.1 (all in segment store) to AEM 6.3 (separate DataStore & Segment Store) and push to AWS S3 | Community
Skip to main content
M
Mishelini
April 28, 2020
Solved

Crx2oak migration from AEM 6.1 (all in segment store) to AEM 6.3 (separate DataStore & Segment Store) and push to AWS S3

  • April 28, 2020
  • 2 replies
  • 6097 views

Hi Folks, 

 

I have to do a PoC for using a AWS S3 bucket as external datastore. The Instance I have is with AEM 6.1 using default CRX2 TarMK (all in segment store). 

 

From what I have read and understood, the S3  works only for data store, so to accomplish my task I have to:

  1. Use the crx2oak migration tool to migrate/upgrade the AEM 6.1 repo to newer AEM6.3 with separate Segment Node store and Data Node Store
  2. Second run of crx2oak tool to "push" the DataStore to S3 bucket (assuming the S3 connector is already configured.)  

I have challenges with step 1. When I start the migration tool  with:

 

 

/opt/jdk1.8.0_191/bin/java -jar crx2oak.jar --copy-binaries --src-datastore=/home/me/snap/author/crx-quickstart/repository/segmentstore --datastore=/home/me/new_repo/datastore segment-old:/home/me/snap/author/crx-quickstart/repository /home/me/new_repo

 

 

where /home/me/snap/author/crx-quickstart/repository/segmentstore is the place of current all in one TarMK segmentstore;

           /home/me/new_repo/datastore is the place where I want new Data Store to go;

           /home/me/new_repo is the dir for the new repo 

the migration process somehow freezes shortly after start:

 

# /opt/jdk1.8.0_191/bin/java -jar crx2oak.jar --copy-binaries --src-datastore=/home/me/snap/author/crx-quickstart/repository/segmentstore --datastore=/home/me/new_repo/datastore segment-old:/home/me/snap/author/crx-quickstart/repository /home/me/new_repo 09:23:46.636 INFO c.a.g.c.CRX2Oak: started with args: [--copy-binaries, --src-datastore=/home/me/snap/author/crx-quickstart/repository/segmentstore, --datastore=/home/me/new_repo/datastore, segment-old:/home/me/snap/author/crx-quickstart/repository, /home/me/new_repo] 09:23:46.715 INFO c.a.g.c.c.VersionPrinter: CRX2Oak version: 1.10.0 (STANDALONE mode) 09:23:47.004 INFO c.a.g.c.c.VersionPrinter: crx2oak.jar (version: 1.10, checksum: 8582adb2b4f999866d4b24a48364295c4665e1ab9ea164460e122f555dd3e6421a81f55e74b45d72b86777079b7e48a29e2b8e6703c2a31d3b772e115743bfa5) 09:23:47.014 INFO c.a.g.c.p.ProfileHandler: Applying partly the command line (before loading a profile): [--copy-binaries, --src-datastore=/home/me/snap/author/crx-quickstart/repository/segmentstore, --datastore=/home/me/new_repo/datastore, segment-old:/home/me/snap/author/crx-quickstart/repository, /home/me/new_repo] 09:23:47.015 INFO c.a.g.c.p.ProfileHandler: The following template tags has been defined: {} 09:23:47.016 INFO c.a.g.c.p.ProfileHandler: The command line (after loading a profile): [--copy-binaries, --src-datastore, /home/me/snap/author/crx-quickstart/repository/segmentstore, --datastore, /home/me/new_repo/datastore, segment-old:/home/me/snap/author/crx-quickstart/repository, /home/me/new_repo] 09:23:47.018 INFO c.a.g.c.c.MigrationSpecGenerator: The effective command line for migration: [--copy-binaries, --src-datastore, /home/me/snap/author/crx-quickstart/repository/segmentstore, --datastore, /home/me/new_repo/datastore, segment-old:/home/me/snap/author/crx-quickstart/repository, /home/me/new_repo] 09:23:50.256 INFO o.a.j.o.p.s.f.FileStore: TarMK ReadOnly opened: /home/me/snap/author/crx-quickstart/repository/segmentstore (mmap=false)

 

 

And stays like this forever... I ran this in background and waited for it for a day - no change. 

I ran the migration on a mounted LVM snapshot made from the AEM data dir when instance was stopped. 
I can't find any other info in any logs. Nothing to point me to a problem. 

Any ideas what may be the problem? 

I also appreciate guidelines for the migration approach to S3. 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by joerghoh

Hi,

 

your 6.1 instance doesn't have a datastore, so you must not provide the "src-datastore" parameter. You just have a segmentstore, so the final command should look like this:

crx2oak.jar --copy-binaries --s3datastore=/home/me/new_repo/datastore --s3config=.... segment-old:/home/me/snap/author/crx-quickstart/repository /home/me/new_rep

(you need to adapt the parameter "--s3config" according to https://jackrabbit.apache.org/oak/docs/migration.html)

 

You can either add the parameter "--trace" to dump all log statements to the console, or use "--log-level TRACE" or "--log-level DEBUG" to cause crx2oak to add more information to a file upgrade.log

 

 

2 replies

akhoury
Adobe Employee
akhouryAdobe Employee
Adobe Employee
April 30, 2020

This issues sounds similar to a known issue with crx2oak on AEM6.3.

 

Please try running the same command using the latest oak-upgrade 1.6.20 jar here instead of crx2oak jar:

https://repo1.maven.org/maven2/org/apache/jackrabbit/oak-upgrade/1.6.20/

 

To migrate to S3 from file datastore, the easiest way is to do the following:

1. Run your oak-upgrade or crx2oak migration with --datastore, as you have done.

2. On the destination repository instance, remove all FileDataStore related config files - both from crx-quickstart/install (and from crx-quickstart/launchpad/config folders to be safe) and make sure your SegmentNodeStoreService OSGi config still has customBlobStore set to true.
3. Add your S3 related jars and configs per the official documentation: https://helpx.adobe.com/experience-manager/6-3/sites/deploying/using/data-store-config.html#AmazonS3DataStore

4. Start the destination instance and it would automatically migrate to S3 in the background.

    M
    MisheliniAuthor
    May 5, 2020

    Hi Akhoury,
    unfortunately I'm getting same behavior. I've tried with different versions and variations of migration tool: crx2oak-1.10.0-all-in-one.jar, crx2oak.jar, oak-upgrade-1.18.0.jar. All is same:

    /opt/jdk1.8.0_191/bin/java -jar oak-upgrade-1.6.20.jar segment-old:/home/me/snap/author/crx-quickstart/repository /home/me/new_repo 05.05.2020 07:51:42.544 [main] *INFO* org.apache.jackrabbit.oak.upgrade.cli.parser.MigrationOptions - copyVersions parameter set to 1969-12-31 05.05.2020 07:51:42.546 [main] *INFO* org.apache.jackrabbit.oak.upgrade.cli.parser.MigrationOptions - copyOrphanedVersions parameter set to 1969-12-31 05.05.2020 07:51:42.546 [main] *INFO* org.apache.jackrabbit.oak.upgrade.cli.parser.MigrationOptions - Cache size: 256 MB 05.05.2020 07:51:42.549 [main] *INFO* org.apache.jackrabbit.oak.upgrade.cli.parser.StoreArguments - Source: SEGMENT[segment-old:/home/me/snap/author/crx-quickstart/repository] 05.05.2020 07:51:42.551 [main] *INFO* org.apache.jackrabbit.oak.upgrade.cli.parser.StoreArguments - Destination: SEGMENT_TAR[/home/me/new_repo] 05.05.2020 07:51:46.019 [main] *INFO* org.apache.jackrabbit.oak.plugins.segment.file.FileStore - TarMK ReadOnly opened: /home/me/snap/author/crx-quickstart/repository/segmentstore (mmap=false)
    joerghoh
    Adobe Employee
    joerghohAdobe EmployeeAccepted solution
    Adobe Employee
    April 30, 2020

    Hi,

     

    your 6.1 instance doesn't have a datastore, so you must not provide the "src-datastore" parameter. You just have a segmentstore, so the final command should look like this:

    crx2oak.jar --copy-binaries --s3datastore=/home/me/new_repo/datastore --s3config=.... segment-old:/home/me/snap/author/crx-quickstart/repository /home/me/new_rep

    (you need to adapt the parameter "--s3config" according to https://jackrabbit.apache.org/oak/docs/migration.html)

     

    You can either add the parameter "--trace" to dump all log statements to the console, or use "--log-level TRACE" or "--log-level DEBUG" to cause crx2oak to add more information to a file upgrade.log

     

     

      M
      MisheliniAuthor
      May 5, 2020
      Hi Jörg,
      Terms & ConditionsAccessibility statement

      Loading footer...