AEMaaCS: how to safely configure localized fulltext queries on pages, in a multitenant environment?

[FilippoDT]

Hi,

what is the recommended way to implement localized fulltext queries on pages on an AEMaaCS multi-tenant environment?

I was guessing it would have been possible to:

1. Create a custom index for each use case, potentially for each (tenant, language) pair.
2. For each index:
   1. Set tenant and/or language-wise includedPaths/excludedPaths, as described in https://jackrabbit.apache.org/oak/docs/query/lucene.html#include-exclude , or alternatively place the index under the language root node, https://jackrabbit.apache.org/oak/docs/query/lucene.html#non-root-index-definitions 
   2. Add a language specific analyzer.

But I found some warnings in the docs:

• The “Indexing best practices in AEM” tutorial page orders to “Avoid creating a custom index on the same node type as the OOTB index.”, https://experienceleague.adobe.com/en/docs/experience-manager-learn/foundation/development/understan... 
• The “Query and Indexing Best Practices” user guide seems to give way to multiple indexes on pages, “For nodetypes which have existing indexes (such as dam:Asset or cq:Page) prefer extension of OOTB indexes to the addition of new indexes.”, but doesn't tell what could happen, e.g. if the AEM application itself would behave badly, https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/query... 

Given the above warnings, is it safe to implement the strategy I was figuring?

Thanks!

[abhishekanand_]

Hi @FilippoDT 

To implement localized full-text queries in AEMaaCS multi-tenant environments, extend the OOTB cq:Page index rather than creating custom indexes for each tenant-language pair. By extending the OOTB indexes with specific configurations, you can achieve localized, tenant-specific full-text queries while adhering to AEMaaCS best practices.

 i guess Yes, it is safe to extend the OOTB cq:Page index with includedPaths and language-specific analyzers for tenant-language pairs in AEMaaCS. 

[FilippoDT]

Thanks for your response.

There is a point that is still not clear to me, regarding OOTB index extensions.

In order to have proper results in the SERP of each localized version of my website, I was thinking I would need a language specific analizer for each language, like I do with Solr, but in the docs I've found:

• The name of the custom index should have the form
  <indexName>-<productVersion>-custom-<customVersion> , https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/index... . It seems to me that only the custom extension having the lexicographically highest (productVersion, customVersion) pair would be used, given an indexName.
• “Note that currently only one analyzer can be configured per index”, https://jackrabbit.apache.org/oak/docs/query/lucene.html#analyzers 

So, given those AEMaaCS, constraints, it seems that it is not possible to have multilanguage search using the OOTB index extension technique.

Do you have any advice regarding the strategy outlined above?

Thanks

[konstantyn_diachenko]

Hi @FilippoDT,

I think extending of OOTB index should help you. You can create analyzer via composition: https://jackrabbit.apache.org/oak/docs/query/lucene.html#create-analyzer-via-composition, where you can create own analyzer according to your requirements. 

I would expect that you did correct structure for your content by locale like /content/brand1/en. /content/brand1/de etc. You can do fulltext search under site language root. It should help to do search only by desired locale.

[abhishekanand_]

Hi @FilippoDT 

AEMaaCS has limitations when it comes to configuring multiple analyzers in a single index. Because only one analyzer can be defined per index,

Create separate custom indexes for each language under unique index names, such as cqPage-en-custom and cqPage-fr-custom, each with its own language-specific analyzer. This way:

• Each language-specific index will use the appropriate analyzer for the locale.
• Queries can target the appropriate language index directly to get relevant results for that specific language.

using separate indexes per language with language-specific analyzers offers the best balance of relevance and maintainability. For multi-tenant setups, ensure each index is scoped properly using includedPaths to minimize performance impacts.

[kautuk_sahni]

@FilippoDT Did you find the suggestions helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!

[FilippoDT]

@kautuk_sahni I still have doubts, so I'm creating a PoC in order to check those suggestions, but still failing. If any of you can share the details of a multiple language setting, it would be very helpful.

I'm afraid I'm doing something wrong, so I'll detail what I've done till now:

1. Download the cq:Page OOTP index, namely /oak:index/cqPageLucene .
2. On localhost, copy /oak:index/cqPageLucene and paste as /oak:index/cqPageLucene-custom-1 . Sadly, the docs talks about <indexName>-<productVersion>-custom-<customVersion> as names for custom indexes, but the cq:Page OOTB index doesn't have the productVersion in its name, so I have to guess and try.
3. Set the Italian analyzer and includedPaths=[/content/mysite/language-masters/it] . Set reindex=true and save.
4. Checkpoint: using the query explanation tool, http://localhost:4502/libs/granite/operations/content/diagnosistools/queryPerformance.html , the custom index seems to work. Great.

But then I repeated the above steps, in order to create an index that uses an English analyzer:

• New index name, copied from /oak:index/cqPageLucene : /oak:index/cqPageLucene-custom-2 (thinking: perhaps "or higher" in the docs means: the customVersion should be higher, https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/index... . But will this new index be used instead of /oak:index/cqPageLucene-custom-1 ? Let's try and see.)
• includedPaths=[/content/mysite/language-masters/en]

Result:

the query performance tool says that the queries

select * from [cq:Page] as s where contains(s.*, 'test') and isdescendantnode(s, '/content/mysite/language-master/en')

select * from [cq:Page] as s where contains(s.*, 'test') and isdescendantnode(s, '/content/anothersite/language-master/en')

are using the (Italian) index /oak:index/cqPageLucene-custom-1 .

In the SDK (custom) indexes can be configured under e.g. /content/mysite/language-masters/it . I still haven't checked if it would work on AEMaaCS.

[konstantyn_diachenko]

Hi @FilippoDT,

This versioning is designed for 1 index: <indexName>-<productVersion>-custom-<customVersion>.

The cqPageLucene-custom-1 and cqPageLucene-custom-2 are the same index in the AEMaaCS, because indexName is cqPageLucene. The cqPageLucene-custom-2 will override cqPageLucene-custom-1. Locally, if you introduce new index version, disable previous index by change property type=disabled on the index definition node.

If you want to have one index for the Italian, another - for the English, I would suggest to put this into the index name like:

<someprefix>.cqPageLuceneIt-custom-1, <someprefix>.cqPageLuceneEn-custom-1. Note, someprefix should be less than or equal to 4 letters.

[FilippoDT]

Someone has marked the reply beginning with "I think extending of OOTB index should help you", by @konstantyn_diachenko , as the correct answer, without explaining how an OOTB index extension, that is unique, could have one analyzer that works both for, e.g., English and Italian.

I hope that who selected that comment could add the reason.

In the meanwhile, I'm unmarking that and accepting this one of the same author, because it has crucial details:

1. You can have only one OOTB index extension.

2. In order to have e.g. an English analyzer for some directories and an Italian analyzer for others, you have to create two custom indexes on the same nodetype of an OOTB index.

So the conclusion is: if you want to have localized search using AEM OOTB features, currently you have to go against Adobe best practices.

Thank you all for having spent your valuable time searching for a solution to the initial question.

[FilippoDT]

Having asked Adobe for support, they suggested to set a "tag" index selection policy, so that only my queries will use custom indexes.
The relevant documentation section is:
https://jackrabbit.apache.org/oak/docs/query/query-engine.html#index-selection-policy

This configuration should avoid the problems reported in Adobe best practices pages.

[FilippoDT]

@abhishekanand_ @konstantyn_diachenko thank you for your detailed examples.

In the documentation, section https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/index... , I see:

"

1. Customization of an OOTB index. These are indicated by appending -custom- followed by a numerical identifier to the original index name. For example: /oak:index/damAssetLucene-8-custom-1.

2. Fully custom index: It is possible to create an entirely new index from scratch. Their name must have a prefix to avoid naming conflicts. For instance: /oak:index/acme.product-1-custom-2, where the prefix is acme.

"

I'm afraid that naming the new indexes "cqPage-en-custom" or "<someprefix>.cqPageLuceneIt-custom-1" would not follow the convention for OOTB index customization, but would have the effect of creating fully custom indexes. And this is to be avoided, as warned in the references above.

*edit*

And for dam:Assets indexes the situation is even worse:

"
Introducing new indexes on the dam:Asset nodetype (particularly fulltext indexes) is strongly discouraged as these can conflict with OOTB product features leading to functional and performance issues.

"

[konstantyn_diachenko]

Hi @FilippoDT ,

I'd suggest to try creating the following indexes:

• /oak:index/<someprefix>.damAssetLucene-8-it with includedPaths=[/content/dam/<site>/it], where 8 is an OOTB damAssetLucene version.
• /oak:index/<someprefix>.damAssetLucene-8-en with includedPaths=[/content/dam/<site>/en], where 8 is an OOTB damAssetLucene version.
• /oak:index/<someprefix>.cqPageLucene-it with includedPaths=[/content/<site>/it]
• /oak:index/<someprefix>.cqPageLucene-en with includedPaths=[/content/<site>/en]

However, you will need to track OOTB indexes updates and sync your "custom" indexes with OOTB.

[FilippoDT]

Hi @konstantyn_diachenko ,

sorry, I didn't mention another relevant feature, https://jackrabbit.apache.org/oak/docs/query/lucene.html#design-considerations :

"8. Avoid overlapping index definition - Do not have overlapping index definition indexing same nodetype but having different includedPaths and excludedPaths. Index selection logic does not make use of the includedPaths and excludedPaths for index selection. Index selection is done only on cost basis and queryPaths. Having multiple definition for same type would cause ambiguity in index selection and may lead to unexpected results. Instead, have a single index definition for same type.

"

[konstantyn_diachenko]

@FilippoDT , okay, I see. Can you try to define includedPaths=[/content/dam/<site>/it] and queryPaths=[/content/dam/<site>/it] for each index then?