Suggestions on publishing a large set of Content Fragments updated through overnight job | Community
Skip to main content
eagleinhills
eagleinhills
Level 2
March 26, 2024
Solved

Suggestions on publishing a large set of Content Fragments updated through overnight job

  • March 26, 2024
  • 2 replies
  • 1260 views

We have an overnight job that takes data from 3rd party source and updates tens of thousands of Content Fragments. 

 

Our job then runs OOTB Publish Content Tree Workflow to publish all the content fragments. 

 

These publishing workflow jobs take long time to run and many time die running for more than 24 hours.

 

The other issue is OOTB Publish Content Tree Workflow updates all the Content Fragments which are almost over 100,000. 

 

Is there any efficient way to publish the Updated Content Fragments (only update the changed Content Fragments)?

 

Thank you!

 

 

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Kamal_Kishor

@eagleinhills : How are you updating the CFs?

Based on data from third-party, if only selected CFs need to be updated, we can trigger the "Publish Content Tree Workflow" only for content which is updated.
There are parameters available in this workflow, please see if you are already following these steps:-https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/replication#publish-content-tree-workflow

Parameters

  • includeChildren (boolean value, default: false). The value false means that only the path is published; true means that children are published too.

  • replicateAsParticipant (boolean value, default: false). If configured as true, the replication is using the userid of the principal which performed the participant step.

  • enableVersion (boolean value, default: false). This parameter determines if a new version is created upon replication.

  • agentId (string value, default means only agents for publish are used). It is recommended to be explicit about the agentId; for example, setting it the value: publish. Setting the agent to preview publishes to the preview service.

  • filters (string value, default means that all paths are activated). Available values are:

    • onlyActivated - only activate pages that have (already) been activated. Acts as a form of reactivation.
    • onlyModified - activate only paths which are already activated and have a modification date later than the activation date.
    • The above can be ORed with a pipe “|”. For example, onlyActivated|onlyModified.

2 replies

krati_garg
Adobe Employee
krati_gargAdobe Employee
Adobe Employee
March 27, 2024

Hi @eagleinhills 

 

For your replication issue, you might have to check the error logs, if your queues get blocked over a specific period of time. Based upon which we would have to figure out the way out.

 

For the Published Content Fragments, these must be updated by their replication date which is OOTB behavior. Do you have any business requirement which is being interfered with this behavior.

    Kamal_Kishor
    Community Advisor
    Kamal_KishorCommunity AdvisorAccepted solution
    Community Advisor
    March 27, 2024

    @eagleinhills : How are you updating the CFs?

    Based on data from third-party, if only selected CFs need to be updated, we can trigger the "Publish Content Tree Workflow" only for content which is updated.
    There are parameters available in this workflow, please see if you are already following these steps:-https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/replication#publish-content-tree-workflow

    Parameters

    • includeChildren (boolean value, default: false). The value false means that only the path is published; true means that children are published too.

    • replicateAsParticipant (boolean value, default: false). If configured as true, the replication is using the userid of the principal which performed the participant step.

    • enableVersion (boolean value, default: false). This parameter determines if a new version is created upon replication.

    • agentId (string value, default means only agents for publish are used). It is recommended to be explicit about the agentId; for example, setting it the value: publish. Setting the agent to preview publishes to the preview service.

    • filters (string value, default means that all paths are activated). Available values are:

      • onlyActivated - only activate pages that have (already) been activated. Acts as a form of reactivation.
      • onlyModified - activate only paths which are already activated and have a modification date later than the activation date.
      • The above can be ORed with a pipe “|”. For example, onlyActivated|onlyModified.
      eagleinhills
      eagleinhillsAuthor
      Level 2
      March 28, 2024

      Thank you @kamal_kishor. I was asking this question for another team. Let me run it by them if they are following the parameters/filters properly

       

       

        eagleinhills
        eagleinhillsAuthor
        Level 2
        March 28, 2024

        One more thing I would like to mention is the performance of Publish Content Tree in publishing 100 thousand Content Fragments if sometimes we have to update all the Content Fragments is very poor. It seems to fail even if we divide into sub categories with range from 10 thousand to 20 thousand. 

         

        From smaller updates this solution will work but any suggestions on when we need to publish all of them once in a while with new data from the external source?

          Terms & ConditionsAccessibility statement

          Loading footer...