Not able to upload packages and edit pages after moving AEM instance from centOS to Rocky Linux

[Sebastian_Stei3]

We need to migrate an AEM 6.5.9 instance from a centOS7 to a Rocky Linux 9 server. Our initial idea was to just copy over the whole instance via rsync from the centOS to the Rocky Linux server using the steps described here: https://experienceleague.adobe.com/en/docs/experience-cloud-kcs/kbarticles/ka-16875.
The process itself is working fine, all permissions, file owners and timestamps seem to be unchanged, as far as we can tell. Also, the instance is starting up fine. However, there are multiple issues with the new instance:
* When we are trying to upload a package via crx/packmgr, we either get an "package file parameter missing" or an "No such file or directory" error. We've tested this with multiple packages, all of them are showing the same errors. Uploading the same packages on the centOS AEM instance is working fine.
* When opening an existing page in either Classic or Touch UI for editing, the page itself is rendering fine. But we can not add / edit / delete any components, as no elements for editing are rendering.
* No messages are shown in error log when performing such actions.
* Creating new pages and editing components on these new pages is working fine.
* Editing existing OSGi configurations via /system/console/configMgr is also working fine.

We are using the same Java Version 1.8.0_161 on both servers. SELinux is also deactivated on both servers.

We've copied the instance multiple times to make sure the issue is not caused by anything happening during the process. But we've seen the same results every time.
Consistency check on the repo via oak-run did not find any issues.

Does anyone have any hints on what could be wrong or what we should check to get the instance running on Rocky Linux?

[EstebanBustamante]

Hi, 

Can you try to upload the package via the command line? Perhaps this can give you more idea of what's going on. Try something like below, and check the "upload.txt" file.

curl -u username:password -F file=@"./YourPackage.zip" -F name=“Package” -F force=true -F install=true http://localhost:4502/crx/packmgr/service.jsp --progress-bar -o upload.txt

Hope this helps

[Raja_Reddy]

Hi @Sebastian_Stei3 

Based on the symptoms you described, it's possible that the issue is related to file permissions or ownership.

1. Check the file permissions and ownership of the AEM installation directory and its subdirectories. Make sure that the files and directories are owned by the same user and group as on the CentOS server. You can use the `ls -l` command to check the ownership and permissions.

2. Check the file permissions and ownership of the Java installation directory and its subdirectories. Make sure that the files and directories are owned by the same user and group as on the CentOS server. You can use the `ls -l` command to check the ownership and permissions.

3. Check the file permissions and ownership of the AEM run mode directory (`crx-quickstart/install` or `crx-quickstart/launchpad`) and its subdirectories. Make sure that the files and directories are owned by the same user and group as on the CentOS server. You can use the `ls -l` command to check the ownership and permissions.

4. Check the file permissions and ownership of the Apache Felix OSGi cache directory (`crx-quickstart/launchpad/felix`) and its subdirectories. Make sure that the files and directories are owned by the same user and group as on the CentOS server. You can use the `ls -l` command to check the ownership and permissions.

5. Check the file permissions and ownership of the Apache Sling logs directory (`crx-quickstart/logs`) and its subdirectories. Make sure that the files and directories are owned by the same user and group as on the CentOS server. You can use the `ls -l` command to check the ownership and permissions.

If the file permissions and ownership are correct, you can try the following:

1. Clear the AEM cache by deleting the `crx-quickstart/repository/segmentstore` directory and restarting AEM.

2. Check the AEM error.log file for any error messages related to the package manager or component editing.

3. Check the AEM access.log file for any HTTP 500 errors related to the package manager or component editing.

4. Try uploading a package using the `curl` command instead of the CRX Package Manager UI. Here's an example command:
curl -u admin:admin -F file=@/path/to/package.zip http://localhost:4502/crx/packmgr/service.jsp

Replace `admin:admin` with your AEM admin credentials and `/path/to/package.zip` with the path to your package file.

[Shashi_Mulugu]

@Sebastian_Stei3 , Did you find the suggestions from users helpful? Please let us know if more information is required. Otherwise, please mark the answer as correct for posterity. If you have found out solution yourself, please share it with the community

[Sebastian_Stei3]

Hi @EstebanBustamante , @Raja_Reddy , @Shashi_Mulugu ,

thanks you for your input. Unfortunately, my DevOps colleague with whom I'm working on this issue is not available this week. I'll let you know of our findings next week.

BR

Sebastian

[Sebastian_Stei3]

So we've done the checks supposed here.

• File permissions and ownerships are exactly the same on both instances
• Nothing suspicious to be found in the log files
• Uploading a package via curl gives the following response:

<crx version="1.22.7" user="admin" workspace="crx.default">
  <request>
  </request>
  <response>
    <data>
      +------------+-----------------------------------------+
      |  Arguments | Comment                                 |
      +------------+-----------------------------------------+
      |  cmd=help  | print this help                         |
      +------------+-----------------------------------------+
      |  cmd=ls    | print a list of all packages            |
      +------------+-----------------------------------------+
      |  POST      | remove a  package                       |
      |  cmd=rm    |                                         |
      |  name      | package name                            |
      |  [group]   | group name (optional)                   |
      +------------+-----------------------------------------+
      |  POST      | build a  package                        |
      |  cmd=build |                                         |
      |  name      | package name                            |
      |  [group]   | group name (optional)                   |
      +------------+-----------------------------------------+
      |  POST      | install a package                       |
      |  cmd=inst  |                                         |
      |  name      | package name                            |
      |  [strict]  | true to fail on error                   |
      |  [group]   | group name (optional)                   |
      +------------+-----------------------------------------+
      |  POST      | uninstall a package                     |
      |  cmd=uninst|                                         |
      |  name      | package name                            |
      |  [group]   | group name (optional)                   |
      +------------+-----------------------------------------+
      |  GET       | download a package                      |
      |            | (content-disposition header contains    |
      |            | the correct filename)                   |
      |  [cmd=get] | optional                                |
      |  name      | package name                            |
      |  [group]   | group name (optional)                   |
      +------------+-----------------------------------------+
      |  POST      | upload a new package                    |
      |  file      | package to upload                       |
      |  [name]    | optional name                           |
      |  [strict]  | true to fail on install error           |
      |  [install] | automatically install package if 'true' |
      +------------+-----------------------------------------+
    </data>
    <status code="200">ok</status>
  </response>
</crx>

When doing the curl upload, we don't see any entries in the logs at all.

Bundles are all up and running.

Any more ideas on what we could check?

[chaackchaack]

just as a wild guess - have you checked the max files settings?

regards

christian

[Sebastian_Stei3 · Accepted solution]

Hi christian,

we've checked that as well, but it did exactly match with the settings on the old servers.

In the end, we decided to set up the environment from scratch, as we were not able to make any progress on the issue and the deadline came closer....

[kautuk_sahni]

@Sebastian_Stei3 Did you find the suggestions from users helpful? Please let us know if more information is required. Otherwise, please mark the answer as correct for posterity. If you have found out solution yourself, please share it with the community.