About to go live with a multi-million pound website on AEM Cloud, but cant create or edit pages on production environment. We only see "CSV Report"

[TB3dock]

We raised a ticket with adobe enterprise support more than 2 months ago - but no answer yet.

We have several AEM Cloud environments, including the "stage" and "production" envs (and several separate dev and test ones).

We control access via the cloud admin console (https://adminconsole.adobe.com/). In the cloud manger, we assign product profiles to users (not via groups as this has bugs - another outstanding multi-month ticket with adobe support).  We use adobe logins (not AD).

See screen shots

We setup the groups and assigned the product profiles as instructed here: https://helpx.adobe.com/enterprise/using/user-groups.html 

When we assign the two product profiles, which look like "AEM Adminstrators-xxx" and "AEM Users-xxx" we can then login to the author instances as expected. But, on some envs we can create/edit pages, and some we cant. In addition, some users can, and some users cant, even though they have exactly the same setup in Admin Control panel.  This means our teams have to share the logins for the one or two users who can access the environemnts.

We have 7 envs, including dev, test, prod, stage etc. 

Each env has 3 product profiles in AEM:

1. Author: AEM Users-xxx
2. author: AEM Administrators-xxx
3. publish: AEM Users-xxx

If we assign all 27 profiles to a user, that user may then be able to login to say 5 of the 7 envs, and on those 5, may have page creation/admin rights on 3. Every few days this changes.

Note: we dont use the user admin settings on the environments themselves - these have not been touched - we only use the central admin tool to manage permissions (which would seem to be the only way to manage some many environments).

The show stopper is that now we have one env, production, which noone can edit or create pages on.  When it the "create" button in the top right, it only gives "CSV Report".  Things like "Create Page" are missing (see screen shot)

Adobe have no solution or workaround.  We have done several screen shares with their support teams, Adobe have had it investigated with their engineers 3 times, we have escalated to the account manger, no luck.

Anyone else have this issue? 

========== UPDATE 1 =========

We notice that when we assign both "AEM Administrators-xxx" and "AEM Users-xxx" to a user, either directly or via a group, some days they can login, some days they cant. Some days when they login they are administrators, and can create pages, some days they cant create pages.  AEM flips and flops the permissions at random, for no reason.  Noone is editing any permissions except me, and I only do it through the admin console in a controlled manner.

The interesting thing is this:

AEM is assigning the WRONG PROFILES, and changing them every few days. Its a sync issue.

If I look at our dev author envs, login and open the groups page, I see the following:

If you look, it shows my user has product profile ending in 9a17 (admin and user), and also a third one a194 (user).

the product 9a17 is for a completely different environment, which is why this is not working.

If I wait a few days, do nothing, it will sync the right profile, and I can edit pages again. Next day not.  

I think there is a major bug where its only able to sync the first admin product profile, and which it syncs is random, and it changes depending on the order.

If we look in the admin console, I have assigned to my user ALL product profiles.  So I should see the complete list.  But I only ever see one (admin one), and it changes every few days for no reason.

If I get someone who has (today) admin access on this env, they see all the envs product profile on this env.  If they then add me via the local AEM instance (not admin control panel) to the Admin-XXX group for this env, Magically I then see everything I should see, but later it gets overwritten with the wrong one.

Right now, there is no way to make me admin on any author instance.  Assigning the AEM Administrators product profile for that instance doesnt work (or rather, it randomly works some days, but not others). Right now, its not working, and there is nothing we can think of to get ourselves admin access to our own AEM instances. It cant be done via the admin control panel currently due to these bugs.  In the admin control panel, I have every permission possible, including administrative rights as system administrator, product administrator (all products), profile administrator, support admin etc.

=== update 3 ===

In desperation, we tried adding a new custom group in Admin console, and assigned all product profiles (hence all envs) to this custom group.  This group gets synced to each env (surprisingly, regardless of if there is a product profile for that env or not attached to that group).  Then we added this group to each AEM instance built in "administrators" group.  Bingo, this worked, all our administrators who, despite having the admin product profile who were not admins, were now admins.  BUT! the next day, the group assignment was reverted by AEM, so we are back to square one. No way to give users access to our AEM environments without the pot luck lottery we have now due to the major bug described above.  We guessed that AEM is wiping the local groups/permissions and resetting them back to factory each deploy, so we made a deployment, and it didnt reset it.  AEM cloud does deploy a mysterious config package, but trying to download this package to see whats in it just gives an error message in the AEM package manager.    So AEM is resetting the local groups on each cloud AEM env randomly (not on deploy).  No solution yet.

[Vijayalakshmi_S]

Hi @TB3dock,

Could you please confirm if you have configured AEM specific Groups (with desired permissions in Tools -> Security -> Groups) in Cloud service instance and have the synced IMS groups or product profile groups as members of those AEM groups.

[TB3dock]

Sorry, i have no idea what you mean.  In Admin console, we simple assign the product profiles to users.  We can do this with or without groups. If we do it without groups. 

We have 7 envs, including dev, test, prod, stage etc. 

Each env has 3 product profiles in AEM:

1. Author: AEM Users-xxx
2. author: AEM Administrators-xxx
3. publish: AEM Users-xxx

If we assign all 27 profiles to a user, that user may then be able to login to say 5 of the 7 envs, and on those 5, may have page creation/admin rights on 3. Every few days this changes.

We dont do anything with ACLs, we just use the built in starter product profiles.

[TB3dock]

As there is no way to assign admin permissions for an instance,  if I login to the author instances as me, I can only "see" what groups I am in, not edit anything.    I am in the wrong groups, as synced by the admin console.  e.g. I can see I am in a single AEM Administrators-XXX profile/group for an environment which is not this one, although in the adobe console I am assigned AEM Administrators profile for all environments.

[arunpatidar]

Hi,

The reason why user can't edit page is because of missing permission for IAM groups in AEM. Could you please check if those groups has proper rights?

I remeber when I was working on same, we configured content-admin AEM user group to add for all users as a default AEM group(there is a osgi configuration in AEM to do this) when user logged in. For other IAM groups in AEM, we had to set the required permisison.

For more info - https://experienceleague.adobe.com/docs/experience-manager-64/administering/security/ims-config-and-...

[TB3dock]

Hi, I don't really understand what you mean.  We are assigning product profiles to users via the admin control panel, we are not using groups at all.  This works perfectly for some users, for some of the time, as explained, but means today have the right permissions on env a, tomorrow env b, the next day neither, the next day both - AEM IMS is really broken.