Adobe Experience Manager Sites & More

AmanK24 · 10/13/21

Hi Team,
We are facing search query traversal issue i.e. “The query read or traversed more than 100000 nodes. To avoid affecting other tasks, processing was stopped.”

We have assets stored under the “/content/dam/<project>/<sub-module>”. We are also using Asset share commons 1.94. version for the search page. The query generated while the page load is something like as below.:

2_group.0_path = "path1"

2_group.1_path = "path2"

2_group.2_path = "path3"

2_group.3_path = "path3"

2_group.4_path = "path4"

2_group.p.or = true

4_group.group.2_property = ./jcr:content/metadata/isSearchable

4_group.group.2_property.operation = unequals

4_group.group.2_property.value = false

5_group.mainasset = true

orderby = @jcr:score

orderby.sort = desc

p.guessTotal = 10

p.limit = 10

p.offset = 0

type = dam:Asset

The assets under this path /content/dam/<project> is more than 100000 count. The query runs under this path only to get the searchable assets. This path is enabled with CUG’s. The traversal issue come when users have access to only few assets or different sub module access.

Please provide the possible solution here for search enhancement. And FYI, the number of assets will continue to grow in future.

With Regards

Aman Kumar

CC:

lukasz-m · 10/14/21

Hi @AmanK24, it looks that query you have prepared is not covered by any index. Potential fixes are

Change the query in the way it will be covered by existing definition.
Create or extend existing index to support you query. You can use [1] to generate index definition.
Change traversal limit, however keeping in mind your comments that number of assets will increase, probably this will not be a long term solution.

You can also have a look on this page [2]

[1] http://oakutils.appspot.com/generate/index

[2] https://experienceleague.adobe.com/docs/experience-manager-65/developing/bestpractices/troubleshooti...

AmanK24 · 10/14/21

Thank you @lukasz-m for approaches.
I have few constraint here:

1. Assets Share Commons is used for generation of Query.

2. I need to make query with all the number of asset present that's the req. as we cant afford of skipping any search result.

3. I have already create custom Lucene index for dam:asset and required properties.

lukasz-m · 10/14/21

@AmanK24, so could you confirm if custom index solves the issue? If not, did you check query execution plan to verifies if your custom index is used. One last thing, I would assume that you have run re-index after you created custom index?

AmanK24 · 10/14/21

Hi @lukasz-m No I am getting issue even after creating custom index. As i am saying we have already more than 100000 assets.

Yes, the custom index is only picked.

Also yes re index after creation of custom index.

What i feel is my index count might be more than 100000.

lukasz-m · 10/14/21

@AmanK24you can check index size and number of elements in index under /system/console/status-oak-index-stats, can you share query execution plan - this could be useful for further discussion.

AmanK24 · 10/18/21

@lukasz-m ,

Please find the state of the custom index as below.
Type : lucene
Async : true
Async lane name : async
Last indexed up to : 2021-10-17T03:06:48.112Z
Last updated time : 2021-10-15T23:08:43.111Z
Size : 501.3 MB
Estimated entry count : 1184743

Note: The core application functionality of the application is to maintain all digital assets in one central location and allow users to search and download assets.

Please find the plan as below:

[dam:Asset] as [a] /* lucene:abc-asset-custom(/oak:index/abc-asset-custom) +:ancestors:/content/dam/path1 +:notNullProps:jcr:content/metadata/abc:isSearchable where ([a].[./jcr:content/metadata/abc:isSearchable] <> 'false') and (isdescendantnode([a], [/content/dam/path1])) */ union [dam:Asset] as [a] /* lucene:abc-asset-custom(/oak:index/abc-asset-custom) +:ancestors:/content/dam/path2 +:notNullProps:jcr:content/metadata/abc:isSearchable where ([a].[./jcr:content/metadata/abc:isSearchable] <> 'false') and (isdescendantnode([a], [/content/dam/path2])) */ union [dam:Asset] as [a] /* lucene:abc-asset-custom(/oak:index/abc-asset-custom) +:ancestors:/content/dam/path3 +:notNullProps:jcr:content/metadata/abc:isSearchable where ([a].[./jcr:content/metadata/abc:isSearchable] <> 'false') and (isdescendantnode([a], [/content/dam/path3])) */ union [dam:Asset] as [a] /* lucene:abc-asset-custom(/oak:index/abc-asset-custom) +:ancestors:/content/dam/path4 +:notNullProps:jcr:content/metadata/abc:isSearchable where ([a].[./jcr:content/metadata/abc:isSearchable] <> 'false') and (isdescendantnode([a], [/content/dam/path4])) */ union [dam:Asset] as [a] /* lucene:abc-asset-custom(/oak:index/abc-asset-custom) +:ancestors:/content/dam/path4 +:notNullProps:jcr:content/metadata/abc:isSearchable where ([a].[./jcr:content/metadata/abc:isSearchable] <> 'false') and (isdescendantnode([a], [/content/dam/path4])) */

lukasz-m · 10/18/21

@AmanK24, could you please share query execution plan?

Ankur_Khare · 10/19/21

Could u pls check these threads-

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/the-query-read-more-than-5...

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/the-query-read-or-traverse...

I suspect custom index is incorrect-

Check below link to generate the index based on the query-

http://oakutils.appspot.com/generate/index

And try again.

AmanK24 · 10/20/21

Thank you @Ankur_Khare , We have referred http://oakutils.appspot.com/generate/index and generated the index. Our system currently hold 11TB+ data which results in the total number of 1.1 millions assets.
And when we perform search, we have to perform on the entire assets.

These assets are categorized in subfolder with CUG enabled.

Ankur_Khare · 10/20/21

For me this number"The query read or traversed more than 10000 nodes. To avoid affecting other tasks, processing was stopped" doesn't seems to be that large and u can try increasing the limit-

http://localhost:4502/system/console/jmx/org.apache.jackrabbit.oak%3Aname%3Dsettings%2Ctype%3DQueryE...

This is not the permanent solution and a permanent solution would be to optimize the query.

https://github.com/AdobeDocs/experience-manager-65.en/blob/master/help/sites-developing/troubleshoot...