i18n dictionary call giving 404 on Production Environment

[Rohan_Garg]

Hi AEM Community,

We are using i18n data dictionary in our project wherein the dictionary are hosted at the path - /apps/projName/i18n.

This node has 3 files - en-US.json and en_us.json.

On Dev and Stage environment the json is loading correctly.

On Production environment, we are getting the below errors -

GET /libs/cq/i18n/dict.en_us.json 404

GET /libs/cq/i18n/dict.en-US.json 404

If we check the error logs, we get the below entries -

06.12.2022 07:15:18.541 [cm-pxyzxyz-eabcxyz-aem-publish-5f64d868f6-hjnk6] *INFO* [163.116.195.116 [1670310918538] GET /libs/cq/i18n/dict.en_us.json HTTP/1.1] org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandsA/us/en/libs/cq/i18n/dict.en_us.json not found

As you can see the path "/content/brandsA/us/en" is appended to the original GET Request.

The request path "/libs/cq/i18n/dict" is bound to below servlet -

ServletResource, servlet=com.adobe.granite.i18n.impl.ResourceBundleExportServlet, path=/libs/cq/i18n/dict

Hence, there are 2 queries for this error -

1. Why is SlingRequestProcessorImpl showing resource path "/content/brandsA/us/en " added to the original request path ?

2. Why are Stage and Production environments responding differently when the artifacts of Stage are what is deployed to Production in AEMaaCS ?

@arunpatidar, @Jörg_Hoh, @B_Sravan, @kautuk_sahni, @markus_bulla_adobe, @Mohit_KBansal, @Bimmi_Soi, @BrianKasingli 

Thanks,

Rohan Garg

[arunpatidar]

Hi,

I think SlingRequestProcessorImpl is not adding the prefix but reporting the error that page not found.

org.apache.sling.engine.impl.SlingRequestProcessorImpl service: Resource /content/brandsA/us/en/libs/cq/i18n/dict.en_us.json not found

the second question about why is this behaving different in stage and in prod.

Can you check the publisher request logs(how request looks like/content/brandsA/us/en/libs/cq/i18n/dict.en_us.json or /libs/cq/i18n/dict.en_us.json) so see if dispatcher has some rules and only applied for PROD ?
Could be the osgi or other configuration for rewrite url/reverse mapping for prod?

please let us know if you find something as well. I am interested to know why there is difference behaviour. 

[Rohan_Garg]

The dispatcher's rules are absolutely identical for both environments. There are no rules which are exclusive only to PROD.

Any reverse mapping for PROD should have applied to STAGE as well.

AEM Request Logs -
Line 101751: 06/Dec/2022:05:18:01 +0000 [18313] -> GET /libs/cq/i18n/dict.en_us.json HTTP/1.1 [cm-pxyzabc-eabc123-aem-publish-5f64d868f6-hjnk6]
06/Dec/2022:05:18:01 +0000 [18313] <- 404 text/html 7ms [cm-pxyzabc-eabc123-aem-publish-5f64d868f6-hjnk6]

Line 102692: 06/Dec/2022:05:20:53 +0000 [18553] -> GET /libs/cq/i18n/dict.en_us.json HTTP/1.1 [cm-pxyzabc-eabc123-aem-publish-5f64d868f6-hjnk6]
06/Dec/2022:05:20:53 +0000 [18553] <- 404 text/html 7ms [cm-pxyzabc-eabc123-aem-publish-5f64d868f6-hjnk6]

So the publish is receiving request in correct format /libs/cq/i18n/dict.en_us.json but gets logged in the error log as /content/brandsA/us/en/libs/cq/i18n/dict.en_us.json.

[Rohan_Garg]

Also, I was checking the below URL -

https://github.com/apache/sling-org-apache-sling-engine/blob/master/src/main/java/org/apache/sling/e...

The resource not found exception might be due to below try block -

[markus_bulla_adobe]

Hi @Rohan_Garg!

My first recommendation would be to continue analysis to identify the component in the stack that is causing the issue.

1. First, please double check on the log files to identify where the rewriting is happening. This could potentially be handled on Dispatcher side (e. g. through rewrite rules) or on publish side.
2. If the rewrite is happening on the dispatcher, please double check on the rewrite rules in place and limit them to not cover/handle the relevant paths of your I18N resources.
3. If the rewrite is happening on the publish instance, there are different mechanisms that could be responsible: Sling Mapping defined under /etc/map, org.apache.sling.jcr.resource.internal.JcrResourceResolverFactoryImpl OSGi configuration or potentially others, such as custom transformers/filters/etc. in place (see this documentation).
4. The issue could also be not (directly) related to rewriting but to ACLs. If the according resource is not accessible due to a lack of permissions of the requesting user, the response would also be a 404. You can easily double check this by trying to request the resource with a user that has the administrators role.

Diverging behavior of different environments can have various root causes and it's hard to tell without further analysis. There are certain areas that usually must be different between e. g. staging and production. These might be domains (relevant e. g. to /etc/map entries), permissions/roles (a user might be assigned the admin role for stage but not for prod) as well as content (prod usually being the leading system for content creation) and others.

Hope that helps!

[Rohan_Garg]

Hi @markus_bulla_adobe, Thank for the amazingly detailed response!

For point #1 & #2, we have the below rule in our dispatcher -

#Allow i18n dictionary JSON /0107 { /type "allow" /path "/libs/cq/i18n/*" /extension "json" }

And as checked on both STAGE and PROD, the default action is set in logs as [actionnone].

Thus, the request is filtering through to publish unhindered.

For point #4 too, the issue is not related to ACLs, The requested resource is being served correctly when hitting the URL on publish.

Coming to point #3, our etc mappings for both environments on Stage and Prod are absolutely identical. When validating the request on local via /system/console/jcrresolver - The request is resolving correctly.

I will check up on further configurations once and update.

Thanks again!

[Rohan_Garg · Accepted solution]

Update -

We recently had our go live in which we switched our domain from https://preprod.brandsA.com to https://brandsA.com.

This domain switching auto resolved the issue. The console errors are no longer coming up in our Production environment thus aligning it with Stage.

@markus_bulla_adobe - Any suggestions on why switching the domain would resolve the issue ?

@arunpatidar - The issue seems to be related to domain. Switching the domain like I mentioned above has resolved this issue.

Thanks,

Rohan Garg

[markus_bulla_adobe]

Good to hear that the issue has been resolved.

As I mentioned in my initial reply, domain related configurations are definitely one thing that might influence the behavior on multiple levels. A couple of potential areas where domains might influence request serving are:

• Dispatcher configuration, especially VHost setup. Most configurations will have different VHosts for different domains and sometime redirect rules are also varying based on the domain of the request.
• Sling Mapping defined under /etc/map also takes the request domain into account when resolving/mapping the incoming request.
• I've seen custom implementations that take the domain header into account when serving a request, so your application might also be influencing behavior.

Hope that helps!

[Rohan_Garg]

Thanks a lot again for your inputs!
Actually, we are deploying the virtual hosts configuration out of a build script in NodeJS.

Similarly the etc mappings as well are generated by the build script.

This ensures that all the environments Dev, Stage, Prod have exactly the same set of vhosts, redirects, filters and other configurations.

I will take another look at the code maybe to see what can produce these different results with the exact same script!

Thank you again!