Bulk replication of Pages, Tags, Product Nodes | Community
Skip to main content
B
bhanuprakashdod
Level 2
May 24, 2020
Solved

Bulk replication of Pages, Tags, Product Nodes

  • May 24, 2020
  • 7 replies
  • 9434 views

Hi Everyone,

 

I have a scenario to automate the replication of huge number of pages, tags, nodes  (overall 30k will be there).
which is the best process to achieve this?

 

Thanks in Advance

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Theo_Pendle

Hi @bhanuprakashdod,

What is your concern exactly? I can see two possibilities:

  1. You are concerned about performance. You a huge replication could take several hours and you don't want your site to be slow during that time. In this case the simple answer is just to perform the replication in a lower environment to determine how long it would take (is it 30min or 6hours?) then identify the slot in the day when you have the least amount of traffic and do it then. If you have a X publishers and a load balancer to balance the traffic, you can do it even more seamlessly by doing the replication one publisher at a time, making sure to balance traffic to other publisher instances during that period.

  2. The content is not hierarchically related, ie: you want to publish /content/site/pageA and /content/site/pageB/1 but not /content/site/pageA/1 or something like that, so it's not as simple as publishing the root of a single website. Doing the publication via the TouchUI might require hundreds or thousands of clicks and there is a large risk of human error. In this case you will need to write some backend logic to replicate your content programatically. You can do this by using the com.day.cq.replication.Replicator service to trigger resource replication. See more information on how to use the API here.

7 replies

Ankur_Khare
Community Advisor
Ankur_KhareCommunity Advisor
Community Advisor
May 24, 2020

Try doing it programatically -

https://dev.to/jerinraj55551/replication-api-in-aem-adobe-experience-manager-405n

 

Also look at acs automatic package replication as well-

https://adobe-consulting-services.github.io/acs-aem-commons/features/workflow-processes/content-traversing-replicate-with-options/index.html

 

 

    B
    bhanuprakashdodAuthor
    Level 2
    May 25, 2020
    I have tried with code and it tooks around 6 hours. But if i try it by creating a package and replicate manually it took couple of minutes only. But I have a scenario to automate this and should be fast
    Theo_Pendle
    Theo_PendleAccepted solution
    Level 8
    May 24, 2020

    Hi @bhanuprakashdod,

    What is your concern exactly? I can see two possibilities:

    1. You are concerned about performance. You a huge replication could take several hours and you don't want your site to be slow during that time. In this case the simple answer is just to perform the replication in a lower environment to determine how long it would take (is it 30min or 6hours?) then identify the slot in the day when you have the least amount of traffic and do it then. If you have a X publishers and a load balancer to balance the traffic, you can do it even more seamlessly by doing the replication one publisher at a time, making sure to balance traffic to other publisher instances during that period.

    2. The content is not hierarchically related, ie: you want to publish /content/site/pageA and /content/site/pageB/1 but not /content/site/pageA/1 or something like that, so it's not as simple as publishing the root of a single website. Doing the publication via the TouchUI might require hundreds or thousands of clicks and there is a large risk of human error. In this case you will need to write some backend logic to replicate your content programatically. You can do this by using the com.day.cq.replication.Replicator service to trigger resource replication. See more information on how to use the API here.
      B
      bhanuprakashdodAuthor
      Level 2
      May 25, 2020
      I have tried with code and it tooks around 6 hours. But if i try it by creating a package and replicate manually it took couple of minutes only. But I have a scenario to automate this and should be fast
      VeenaVikraman
      Community Advisor
      VeenaVikramanCommunity Advisor
      Community Advisor
      May 25, 2020

      @bhanuprakashdod Agree with @theo_pendle , the only issue you might have here is the performance and he has suggested the best way to figure that out.

      Adding to that , in my experience the best way to do it is to write custom back-end process , may be a JOB which will run at a particular time window ( so that you don't disturb the business time) , if not you can write a servlet and hit the servlet to run the process . I would say a JOB will be the best approach in this case as it will ensure that the job is completed (If it fails , it will retry for the number of times you have mentioned ) . It will let you know if the Job is completed or failed ( If still it fails after retries ) by which you can be assured that all the content replication has been successfully completed or not. I have tried to explain the benefits of Job here 😊 Just see if this helps. ✌

        raghavc
        raghavc
        Level 4
        May 26, 2020

        Adding to the suggestions shared by others, if you are unable to perform the operation outside business hours , do consider creating a new replication agent and use this replication agent to replicate the content using your custom code, this will ensure any normal content activity is not impacted. You could possibly reduce the load by replicating content in batches.

          Theo_Pendle
          Theo_Pendle
          Level 8
          May 26, 2020
          If your biggest concern is performance then using a package could indeed be better (no HTTPS connection and data transfer to slow you down). However, keep in mind that your content would not be considered to be "replicated" by AEM. You would have no publication status, last publication dates, etc. So before you do this, just make sure you don't have other systems (services, workflows, components) that rely on the publication status of content to function correctly or you might find a problem later down the line 😕😕
          raghavc
          raghavc
          Level 4
          May 26, 2020
          You can use https://adobe-consulting-services.github.io/acs-aem-commons/features/package-replication-status-updater/index.html to ensure the replication status is updated when you install the package. Did you try to replicate the all the pages together or in batches ?
          Terms & ConditionsAccessibility statement

          Loading footer...