Continuously synchronise between contents in dev and prod environments

[Anas_R]

I need to move all contents from AES development to production environments. To do this, I used package manager. The problem was that some folders have large sizes, so I had to build each subfolder and upload them to the development branch. Additionally, I need to copy all changes made in the future from development to production. It is not possible to accomplish this with package manager. Please suggest some appropriate mechanism for achieving this.

[markus_bulla_adobe · Accepted solution]

Hi @Anas_R!

There are multiple tools for content transfer between AEM environments available. Have you already seen the following community article?

Automated Content Sync

Depending on your exact requirements, you might also want to look at the following list of tools for content transfers:

• vlt Tool – Jackrabbit FileVault
• VLT-RCP Tool (ACS AEM Tools)
• CRX2Oak Migration Tool – the only content migration tool that is able to transfer versions
• Content Transfer Tool – official tool for migrations to AEM as a Cloud Service
• repo – FTP like tool for JCR content
• recap – rsync for CRX
• AEM Easy Content Upgrade (AECU) – see also this adapt.to session
• Grabbit (probably outdated)

Please note that not all tools will work with AEM Cloud Services and some might be a bit outdated.

Hope this helps!

[aanchal-sikka]

@Anas_R . In addition to @markus_bulla_adobe suggestions, please refer to https://adapt.to/2023/schedule/lightning-talks/acs-content-sync 

https://adobe-consulting-services.github.io/acs-aem-commons/features/contentsync/index.html

It incrementally synchronizes content between AEM Author instances

The comparison between various tools is available on: https://adapt.to/2020/schedule/content-sync-automation

[Anas_R]

@aanchal-sikka Tried ACS Content Sync tool as mentioned in the article. Configured the source environment with the url, user name and password, gave the path to synchronise and then on clicking 'Submit' button, I got an error

building catalog from https://author-p42404-e166196.adobeaemcloud.com/apps/acs-commons/content/contentsync/jcr:content/sync.catalog.json?root=%2Fcontent%2Fportfolio&strategy=com.adobe.acs.commons.contentsync.impl.LastModifiedStrategy
java.io.IOException: Failed to fetch data from https://author-p42404-e166196.adobeaemcloud.com/apps/acs-commons/content/contentsync/jcr:content/sync.catalog.json?root=%2Fcontent%2Fportfolio&strategy=com.adobe.acs.commons.contentsync.impl.LastModifiedStrategy, HTTP [401]

Could you please tell me what is the issue?

[markus_bulla_adobe]

Hi @Anas_R!

The error message clearly mentions an HTTP 401 Unauthorized status code when the tool is trying to access your Cloud Service environment.

There are different things to try to get to the root cause:

• Please double check on the credentials provided as well as permissions granted for the account you are using for the content sync. Have you tried a manual login with these credentials? Have you tried accessing the path mentioned in the error message via browser?
• Have you checked the log files on the instance where the sync tool is running for additional information? Are there other errors, stack traces of relevant log messages to be found that can help to guide you to the root cause?
• Have you also checked the log files (including access.log and request.log) on the target Cloud instance?

Hope this helps!

[Anas_R]

@markus_bulla_adobe @aanchal-sikka 

• Tried a manual login with these credentials? Yes, that was fine
• Tried accessing the path mentioned in the error message via browser? Yes, I was getting the required json response from this.
• Checked the log files on the instance where the sync tool is running for additional information? I couldn't find any log file from this tool. Only log I got is from the activity log area. That was a stack trace which I mentioned in the question.

• What is access.log and request.log ? Where can I find it? Do we need access to system/console to get these log files ?

[markus_bulla_adobe]

---

@Anas_R  schrieb:

• Tried a manual login with these credentials? Yes, that was fine
• Tried accessing the path mentioned in the error message via browser? Yes, I was getting the required json response from this.

---

One more check would be to try a call from the source AEM host, e. g. logging in to the command line and using curl, wget or similar. That way you could verify if there are any components on network level that might be blocking the call (e. g. because the target is only accepting requests from a dedicated list/range of IP addresses or similar restrictions). There could also be a WAF in place that might block the request based on certain criteria (including IP ranges, user agents and others).

@Anas_R  schrieb:

• Checked the log files on the instance where the sync tool is running for additional information? I couldn't find any log file from this tool. Only log I got is from the activity log area. That was a stack trace which I mentioned in the question.

You could try and setup a dedicated logger (via OSGI web console as explained here), ideally with a DEBUG or even TRACE log level for the relevant java classes of the tool. Apart from that, I recommend to check the error.log of the instance to see if there is anything interesting.

---

@Anas_R  schrieb:

• What is access.log and request.log ? Where can I find it? Do we need access to system/console to get these log files ?

---

Your target environment does seem to be a Cloud Service setup. In that case you won't have direct access to neither system console nor the file system where log files are stored. You need to go through Cloud Manager and Developer Console or use the Adobe I/O CLI Cloud Manager plugin as explained in this article. This documentation provides detailed information on available log files and log levels for AEM Cloud Services.

Just to double check: What AEM versions/flavors are you working with?

The instance running the sync tool is AEM 6.5? Is that self hosted or AMS based?

The instance that you are trying to reach is AEM Cloud Service?

[Anas_R]

@markus_bulla_adobe I was not able to get enough details from the logs. I am using AEM cloud service and my target environment also. Version is 2023.11.14227.

[markus_bulla_adobe]

Have you been able to add a logger for the sync tool and increase the log level to DEBUG?

[Anas_R]

No @markus_bulla_adobe . Will check that and get back to you. Thanks for your responses.

[vineetham123]

@markus_bulla_adobe @aanchal-sikka  Could you please elaborate on what to do to add a Custom log file for this end?

[MarkusBullaAdobe]

In AEM CS it is not possible to add custom log files but you can add a specific logger for the relevant classed/packages that will the log into the standard log files.

The following documentation outlines the process:

https://experienceleague.adobe.com/docs/experience-manager-cloud-service/content/implementing/develo...

You will need to add a logger OSGI configuration and deploy it to your instance.

[vineetham123]

If I want to add the dependency acs-aem-commons-6.3.2 how would the log file look like?

[vineetham123]

@aanchal-sikka  @markus_bulla_adobe  Do you know what to do to contact them for technical support?

[somen-sarkar]

Hi @Anas_R ,

We are also facing similar issue. Was you able to resolve the same and what was the issue ?

Thanks,

Somen