Question about customizing indexes on Cloud environments

[Preston]

We are experiencing slowdown while performing Query Builder queries against content fragments. While digging into this, I believe I discovered that it has to do with a field in the content fragments not being indexed. I added a custom index locally and the query improved from 200 ms to 2 ms.

According to the documentation here (https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/index...) creating new custom indexes that work against assets is discouraged, but modifications can be made to out of the box indexes. Does this mean those modifications are performed by Adobe or is that documentation simply missing from this article?

Preston

[narendiran_ravi · Accepted solution]

Hi @Preston , 

In AEMaaCS, if we want to modify the OOB index, we must copy the index definition from the Cloud environment CRX via packages. We need to include this as part of our code base by adding our custom index inside this. 

For the naming convention to be followed, please refer to the section - https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/index... in the same document.

Ex: /oak:index/damAssetLucene-11 is the index name in Cloud environment if you want to customize use /oak:index/damAssetLucene-11-custom-1. 

We need to do some updates in parent pom.xml and filter.xml of apps as well -https://experienceleague.adobe.com/en/docs/experience-manager-cloud-service/content/operations/index...

Please check and let me know if you need more details

[Preston]

I think the part that's throwing me is this from that documentation. 

"Introducing new indexes on the dam:Asset nodetype (particularly fulltext indexes) is strongly discouraged as these can conflict with OOTB product features leading to functional and performance issues. In general, adding additional properties to the current damAssetLucene-* index version is the most appropriate way to index queries on the dam:Asset nodetype (these changes will automatically be merged into a new product version of the index if it is subseqently released). If in doubt, contact Adobe Support for advice."

The wording that's throwing me is "adding additional properties to the current index version". So do they mean just creating a custom index based off of damAssetLucene-X and adding your own customizations? It's worded as if the index can be "edited". 

EDIT: I should also clarify that part of what's throwing me is that the fields in question (fields that are part of Content Fragment models) are defined in damAssetLucene in the "aggregates" section. 

<include8
        jcr:primaryType="nt:unstructured"
        path="jcr:content/data/master"/>

I'm not entirely clear on what this does, but any fields located under jcr:content/data/master don't seem to be particularly fast to query. Whereas if I add a custom index that includes this.

<jcrName
        jcr:primaryType="nt:unstructured"
        name="jcr:content/data/master/jcrName"
        propertyIndex="{Boolean}true"/>

Querying on jcrName goes from 200ms down to 2ms. 

[narendiran_ravi]

Yeah, the documentation could have been clear. Aggregates are useful when we want to combine the descendant properties of a node.  Refer to this link - https://www.aemcq5tutorials.com/tutorials/aem-oak-indexing-comprehensive-guide/. 

Even though you added the path in aggregates you still need to create an index for a property under that path. The custom index needs to be added under /oak:index/damAssetLucene-11-custom-1/indexRules/dam:Asset/properties as part of your code and deployed in the cloud.

[Preston]

Thanks for the reply. I asked this of @aanchal-sikka , but I'm curious if you have some insight as well. Currently since the query in question is using damAssetLucene, if we begin adding a custom index or indexes that refer to properties of content fragments that will mean that index won't be useful to queries of dam:Asset for media. Am I right to assume that you can have a custom index for media sit side-by-side with a custom index designed for content fragments and that AEM will just choose the appropriate index? 

[aanchal-sikka]

@Preston 

<include8
        jcr:primaryType="nt:unstructured"
        path="jcr:content/data/master"/>

This is used in context of fulltext search.

<jcrName
        jcr:primaryType="nt:unstructured"
        name="jcr:content/data/master/jcrName"
        propertyIndex="{Boolean}true"/>

 This is used in context of exact property search.

Ideal way is to

• copy existing damAssetLucene to create /oak:index/damAssetLucene-<latest>-custom-1 
• Add additional properties/configurations

I guess AEM Cloud now has the ability to merge new version of OOTB index with most recent version of old custom Index. This happens during new Cloud version upgrade. I am unable to find documentation. So, I would request to check with Adobe.

[Preston]

Thanks for the reply and especially for clearing up the different parts of the index. That makes sense. I'll ask Adobe about the proper way to format this. 

As an aside, we dug into this more today and it appears that we already have a custom index for Media Assets. Is there a way that you're aware of to make sure that a custom index designed for Assets is used when making a query for media Assets and that a custom index designed for Content Fragments is used when querying Content Fragments?

[aanchal-sikka]

@Preston 

By default, queries will use the index with the lowest expected cost (as in relational databases). But some cases, it is needed to specify which index(es) should be considered for a query:

• If you want to ensure a specific type of index is used for specific type of queries, for example custom indexes are used for custom queries.

Please try following

/jcr:root/content//element(*, nt:file)[jcr:contains(., 'test')] option(index tag x)

select * from [nt:file]
where ischildnode('/content')
and contains(*, 'test')
option(index tag [x])

https://jackrabbit.apache.org/oak/docs/query/query-engine.html

[Preston]

Interesting. Will this fall back to a different index if it doesn't hit with that index? 

[aanchal-sikka]

@Preston 

I haven't tried Index fallbacks any-time. Unaware if its possible.

[Preston]

Okay. Thanks. Thanks for all of your help, btw. 

[Preston]

One more question since you've been so helpful. Is there a way to tell how out of the box indexes are used? We're trying to figure out how to pick the right index to customize and we think "fragments" is the most helpful to us, but we can't find any documentation on these OOTB indexes and what their purpose is outside of digging into the index itself.