crx2oak for Blue–Green deployment

[another-dave]

1. I've seen some threads discussing the use of crx2oak for Blue–Green deployment, which sounds quite interesting, as one of the challenges of applying this deployment pattern with AEM is the question of how to handle syncing repositories easily.
2. That said, other people recommend only running the repo migration offline (slide 9).
3. The official documentation (Using the CRX2Oak Migration Tool) isn't conclusive either way — not even mentioning offline or online migrations, let alone that one way or the other is recommended (or prohibited).

So, is the use of crx2oak for a migration against a running instance of AEM a supported use-case? If so,

• are there any limitations against running it in this way (e.g. speed)?
• If there are limiting factors, does it have an impact if both instances are offline (versus one online and one offline).

Any info would be appreciated, and links to canonical documentation would be great if they exist.

[smacdonald2008]

The Eng teams talks about the crx2oak tool in this GEMS session -- Deep dive into AEM upgrade process

[another-dave]

Thanks, though from the slides & description (haven't had a chance to check video yet), it seems to be related to upgrades, rather than deployments, is that right? — Is deployment a supported use-case for crx2oak, or is it only for upgrades?

[Jörg_Hoh]

Hi,

The blue-green deployment pattern post by Martin Fowler simply forgot a single item: What happens when blue is under constant change by its users while you prepare green?

This is the problem with AEM, as your blue publishs are under constant change by authoring users. Oh, and don't forget that you have your single point of truth as well, the authoring instance. There you cannot apply this pattern at all, if you don't want to have a planned downtime for the time of the deployment.

My conclusion: You cannot apply the classical blue-green approach.

I normally do deployments in a way, that I use planned service downtimes on authoring, but none on publish.

1. initiate service downtime on author
2. deploy author
3. remove 1st half of the publishing instances from loadbalancer, so that the 2nd half is serving all the requests.
4. deploy 1st half of the publish instances
5. switch loadbalancer, so the 1st half now servces all the requests
6. deploy 2nd half
7. bring all back online

This is a modified version of the blue-green approach: I don't have a standby instance which just changes roles with the production instance. But I have enough redundancy in the frontend that I am able to perform the deployment without downtime.

Jörg

[another-dave]

Thanks Jörg,

what I meant by Blue-Green in this case was to have a 'Blue' and a 'Green' author instance, one which is the live Production instance, and one acting as Pre-Prod. For example:

1. Deploy new release to non-live servers (green), both author and publish.
2. Enforce a content freeze on live author (blue).
3. Run a synchronisation of content between live & non-live servers.
4. Put green servers live, and set blue to non-live
5. Lift the content freeze for authors.

In theory, this set-up is quite feasible, even with a changing author, if we can quickly sync the latest changes from Blue back to Green — a minimal content freeze can be tolerable, especially if outside of normal business hours.

I'm still not clear as to whether this is a supported use-case of the tooling though?

[Jörg_Hoh]

Hi,

the approach sounds good, but the problem is indeed step 3. And unless you know a method, which can achieve this really quickly (that means 2 min at max) in 99% of all cases, I would doubt that this is doable.

Fast synchronization between AEM instances (especially if 1 instance is weeks behind) is hard; especially problematic is the versioning stuff, because the API prevents it to create the versioning nodes directly via JCR APi (you have to use the versioning API for it).

So yes, in theory it's possible. But I haven't seen it implemented yet :-)

Jörg

[another-dave]

Yeah, that's the trouble alright

We could keep them closer than weeks apart by pulling content back from Production (e.g. by using nightly disk-level back-ups of Prod to do a restore over Pre-Prod), but agree that the trouble is getting over the last hurdle alright!

Thanks!

Dave

[another-dave]

kautuksahni, you've marked this answer as 'resolved', so could you clarify if the use of crx2oak against a running instance is a supported use of the tool, or if it can only be run against offline instances?

[smacdonald2008]

Hey - Kautuk and I looked at the AEM Docs here:

Upgrading to AEM 6.3

It clearly states to stop the instance of AEM while using this tool.

I hope this helps.

[another-dave]

Hi Scott,

Still find these docs confusing to be honest… So think you're talking about under the heading "Content repository migration and upgrade" where step #1 says "First, stop the instance if it is running."?

These seems to be explicitly steps for upgrading though, rather than just content migraton, e.g. step #6 says "Start AEM to bring up the instance for the inplace upgrade."

On the child page in the left-hand nav, "Using the CRX2Oak Migration Tool" it calls out three use-cases:

I'm just looking at the second use-case — Copying data between multiple Oak repositories — rather than a full upgrade between versions.

Also, it sounds like these two use cases even use different modes within the tool; from further down the page:

So it's the "standalone" mode that I'm interested in rather than the "quickstart" mode.

Finally on that page, under "Parameters: Migration options", it mentions:

Which would seem to suggest that it is being run against an online instance (or at least, the instance is brought online by the tool as part of the migration?)

Also oak-upgrade (which I presume shares a lot of internals, as the imagery on the documentation is identical), mentions at least an in-part online upgrade for Blob storage.

---

Apologies if I didn't have enough detail in the first post & my question wasn't clear, but (as often seems the case with internal tools of AEM), that the docs are a little incomplete and at times contradictory.

Looking into oak-upgrade more, I may just use this tool directly, as it does explicitly mention that it can be used for incremental upgrade:

Incremental migration

I'll experiment with it, but this coupled with a Shared blob storage (e.g. S3) may provide a quick enough synchronisation during the deploy itself for what I'm after.

Thanks

Dave

[kautuk_sahni]

Hi another-dave

Accedently it got marked as correct. Thanks for the corrective notification. Apologies for this.

~kautuk