Skip to main content
Level 1
Jul 9, 2026 12:38 PM UTC
Question

SSO breaks after upgrading to AEM 6.5.25 on-prem: BadPaddingException on all decrypts

SSO (SAML) breaks after an in-place upgrade to AEM 6.5.25 (CFP-25)

 

Environment: on-premise AEM Author, upgraded from 6.5.10 to 6.5.25.

Immediately after applying CFP-25, every SSO-related decrypt operation fails with:

javax.crypto.BadPaddingException: pad block corrupted

This appears to break decryption of the SAML keystore/truststore passwords and replication transport passwords. As a result, SSO fails and the CMS becomes inaccessible, except by disabling SAML authentication and using the manual login route.

 

The master key is stored in the JCR under /etc/key; it was never migrated to the filesystem bundle. The key value itself was not changed during the upgrade. I simply installed the CFP package, waited until it was complete and restarted the instance, as per the instructions. 

 

The workaround that resolved the issue (currently on test environments) was to rewrite the stored /etc/key value from the older obfuscated format to the raw value, so the newer crypto bundle could read it correctly. Since the underlying key value remained the same, existing encrypted data stayed valid, and decrypt operations started working again after a restart. I did this locally using a custom Groovy script.

 

I have not confirmed whether this is the intended or supported fix, and I have not found this behavior documented in the release notes or known issues.

 

Questions:

  1. Has anyone else seen this issue on AEM 6.5.25, CFP-25, or an earlier service pack?

  2. Did CFP-25 change how the /etc/key master key is read or encoded? If so, when did this behavior change?

  3. Is rewriting /etc/key from the older obfuscated format to the raw value the correct approach, or is the supported remediation to re-encrypt the affected passwords after the upgrade?

7 replies

Level 4
Jul 9, 2026 3:15 PM UTC

Hi ​@JibrilSa,

 

Your workaround is correct. CFP-25 updated the com.adobe.granite.crypto bundle, and the newer version no longer reads the obfuscated format that older AEM versions stored in /etc/key.

 

Rewriting /etc/key to the raw value is the right fix, not re-encrypting passwords. Since the underlying key didn’t change, all existing encrypted values stay valid.

 

To confirm: check the com.adobe.granite.crypto bundle version before/after in Felix console (/system/console/bundles). If the major version changed with CFP-25, that’s your root cause.

 

Recommend raising an Adobe Support ticket to get a hotfix or official confirmation, this type of crypto bundle regression should be documented in the release notes.

Translate
Level 1
Jul 27, 2026 6:54 AM UTC

Hi ​@JibrilSa ,

We are facing the same issue while installing the latest CFP, could you pls share the custom Groovy script you created to rewrite the key value?

 

Thanks in advance.

 

Translate
JibrilSaAuthor
Level 1
Jul 28, 2026 10:12 AM UTC

Hi ​@ash97

I've put it on GitHub here: https://github.com/saladj/aem65-saml-crypto-key-migration

The README covers the details/run order and one setup step you'll need before the scripts will run. I tidied things up a bit compared to what I ran on our environment so let me know if you run into any issues or how you get on.

Translate
Level 1
Jul 29, 2026 8:30 AM UTC

Thanks ​@JibrilSa, will let you know how it goes 

Translate
Level 2
Oct 27, 2026 7:47 AM UTC

Thanks for sharing the details. One related tip for anyone who later moves this estate to AEM as a Cloud Service: the crypto keys (hmac/master) are instance-specific, so encrypted values in content or configs will not decrypt in the cloud. Inventory where encrypted properties are used now; in AEMaaCS they should become Cloud Manager secret environment variables.

Translate
Mani_kumar_
Community Advisor
Community Advisor
Nov 10, 2026 9:23 PM UTC

Hi — you have already found the cause, and the fix is to migrate the stored key, not to re-encrypt anything or touch SAML. Posting this as a consolidated answer because the thread still has no accepted solution, and apalmstrike reported the same problem on a 6.5.24 test environment.

 

Root cause: older AEM instances keep the crypto master key in the repository at /etc/key. The old granite.crypto 3.4.16 bundle stored that key scrambled and unscrambled it on every read. The 3.5.x bundle (3.5.32 in 6.5.25) dropped the unscrambling step and does not migrate existing keys, so it reads the old scrambled bytes as if they were the key itself — the wrong key — and every decrypt fails with BadPaddingException: pad block corrupted. As akhil_merupula noted, CFP-25 is when the bundle changed. SAML is only where you notice it first, because it decrypts the keystore password on every anonymous request; the Global Trust Store, replication-agent passwords, and anything else holding an encrypted value are affected too. Check the version, not the service-pack number: read the granite.crypto version at /system/console/bundles. 3.4.16 and earlier expect the scrambled format; 3.5.x expects the plain format.

 

Fix — rewrite /etc/key/master and /etc/key/hmac from the old scrambled form to the plain form the new bundle expects. The key itself does not change, only how it is stored, so everything already encrypted with it still decrypts afterwards. Do this per instance (Author and Publish each have their own key), and only if your key is JCR-stored. First confirm it applies to you:

-  /etc/key exists with binary properties master and hmac;

-  grep “with JCR storage” crx-quickstart/logs/error.log shows “Starting Crypto Support (com.adobe.granite.crypto, 3.5.x) with JCR storage”;

-  the error is BadPaddingException: pad block corrupted under CryptoSupport.decrypt, usually below SamlAuthenticationHandler.requestCredentials and KeyStoreRepository.readKeyStorePassword.

If your key is file-based instead (under the com.adobe.granite.crypto.file bundle directory), this does not apply.

 

Steps (full scripts and detail are in your own repo, which I am linking rather than pasting blind: https://github.com/saladj/aem65-saml-crypto-key-migration):

1.  Back up first. In Package Manager, build a package with filter root /etc/key and download it. That zip is your rollback. Keep the instance closed to traffic until you have verified.

2.  Recover the two constants. The scripts ship with SALT and PW empty and refuse to run until you fill them in. Take a copy of the old com.adobe.granite.crypto-3.4.16 jar (crx-quickstart/launchpad/startup/1/com.adobe.granite.crypto-3.4.16.jar on an instance not yet upgraded, or from a backup/install media — an instance already on 6.5.25 no longer has it) and run either ./misc/recover-constants.sh /path/to/com.adobe.granite.crypto-3.4.16.jar (needs unzip + javap) or python3 misc/recover-constants.py /path/to/...jar (Python 3.6+, no JDK). Paste the two printed lines over the empty SALT and PW in BOTH 0-validate-e2e.groovy and 2-apply.groovy, exactly as printed — do not tidy up a doubled backslash in the password; Groovy turns \\ back into a single \ at runtime, and that is the most common way to make this fail with no useful error.

3.  Run 0-validate-e2e.groovy first (read-only, in the Groovy Console as an administrator — admin is what gets past the denyAllToEveryone ACL on /etc/key). Only apply if all three hold: master comes out at 16, 24, or 32 bytes (a valid AES key length); the CONTROL line fails (that is the bug you are fixing); and the PROOF line says DECRYPT OK. If any of those is wrong, stop and work out why — do not apply.

4.  Run 2-apply.groovy (it backs up, then rewrites the key), then restart AEM fully. The crypto bundle reads the key once at startup and keeps it in memory, so nothing changes until you restart.

5.  Verify, then open traffic: grep -c “Unable to decrypt” crx-quickstart/logs/error.log on the new log gives 0; an anonymous request to / returns a 302 to your IdP and saml.log looks normal; the Global Trust Store at /libs/granite/security/content/truststore.html lists your certificates instead of coming up empty; and a real SSO login works.

 

Do NOT delete /etc/key, and do NOT re-encrypt passwords — that does not fix this and will make it worse. Rollback warning: after migration the key is stored in plain form, which only the 3.5.x bundle understands. If you ever put the 3.4.16 bundle back, restore the scrambled key first (3-rollback.groovy, or reinstall your /etc/key package) and only then start the old bundle. Keep the *_obf_bak properties or the /etc/key package until you are sure you will not downgrade; 4-cleanup.groovy (which deletes the *_obf_bak backups) is optional and for much later.

 

On venumuvva2021’s question — is there an official take from Adobe on this? — I could not find one in this pass: no Adobe KB or documentation page for this granite.crypto 3.5.x key-format change was located, so the answer above rests on the bundle-version evidence, the validation script’s pass criteria, and the confirmed fix on test environments, not on an Adobe document. If Adobe Support has since published a KB or hotfix for this, that should be linked here instead.

Translate