Expand my Community achievements bar.

SOLVED

FulltextIndex$FulltextPathCursor| Index-Traversed 20000 nodes with filter Filter

Avatar

Level 3

I have a crawler to run JCR-SQL2 queries on author instance using AEM scheduler. Sometimes I would get WARN logs from AEM and the logs look something like this:

 

*WARN*| [sling-default-4]| org.apache.jackrabbit.oak.plugins.index.search.spi.query.FulltextIndex$FulltextPathCursor| Index-Traversed 20000 nodes with filter Filter...

 

Sometimes the node counts from the logs would go from 10,000 to 40,000. Is this something I need to be worried about? What's the maximum threshold on the node number?

 

 

 

 

1 Accepted Solution

Avatar

Correct answer by
Level 4

Does it tell you "To avoid affecting other tasks, processing was stopped"? If so, then your query is not executed correctly because your index needs to be rewritten to have a less amount of records (another solution is reconfiguring the index-traversing limit for Query Engine service).

If you don't have that error, then it's just a warning that you are traversing, so the processing is just slower than it could be. The usual deal when you have a full-text search This warning should be displayed for every 10000 records (for full-text indexes) until the error described above happens, then processing will be stopped.

 

Just in case, check for more info on this page https://jackrabbit.apache.org/oak/docs/query/query-troubleshooting.html and here: https://www.aemcq5tutorials.com/tutorials/aem-oak-indexing-comprehensive-guide/
They seem to be related.

A sample quote from the last page, the setting system property oak.traversing.warn should fix your warning:


Another case is, it might happen that the TraversingCursor still serve the query if an existing index cost is higher than traversing a particular sub-tree.
Analyze your query log file and check for below log statement:

23.09.2017 15:01:08.633 *WARN* [qtp1054115870-158] org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy Traversed 10000 nodes (30000 index entries) using index foo with filter Filter(query=select * from [nt:base] where [foo] is not null, path=*, property=[foo=[is not null]])

Note:- This is highlighting that the query is being handled by an index. ContentMirrorStoreStrategy is used only the property index. Nevertheless, the index is becoming big. more than 10000 nodes by default. The actions from here to be taken are if there’s any way to optimize the query itself or increase the threshold. In our specific example the query can be optimised by providing a specific node type and/or a path restriction. Otherwise the threshold can be configured by the system property oak.traversing.warn. See OAK-2720 for more details.

 

View solution in original post

2 Replies

Avatar

Correct answer by
Level 4

Does it tell you "To avoid affecting other tasks, processing was stopped"? If so, then your query is not executed correctly because your index needs to be rewritten to have a less amount of records (another solution is reconfiguring the index-traversing limit for Query Engine service).

If you don't have that error, then it's just a warning that you are traversing, so the processing is just slower than it could be. The usual deal when you have a full-text search This warning should be displayed for every 10000 records (for full-text indexes) until the error described above happens, then processing will be stopped.

 

Just in case, check for more info on this page https://jackrabbit.apache.org/oak/docs/query/query-troubleshooting.html and here: https://www.aemcq5tutorials.com/tutorials/aem-oak-indexing-comprehensive-guide/
They seem to be related.

A sample quote from the last page, the setting system property oak.traversing.warn should fix your warning:


Another case is, it might happen that the TraversingCursor still serve the query if an existing index cost is higher than traversing a particular sub-tree.
Analyze your query log file and check for below log statement:

23.09.2017 15:01:08.633 *WARN* [qtp1054115870-158] org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy Traversed 10000 nodes (30000 index entries) using index foo with filter Filter(query=select * from [nt:base] where [foo] is not null, path=*, property=[foo=[is not null]])

Note:- This is highlighting that the query is being handled by an index. ContentMirrorStoreStrategy is used only the property index. Nevertheless, the index is becoming big. more than 10000 nodes by default. The actions from here to be taken are if there’s any way to optimize the query itself or increase the threshold. In our specific example the query can be optimised by providing a specific node type and/or a path restriction. Otherwise the threshold can be configured by the system property oak.traversing.warn. See OAK-2720 for more details.

 

Avatar

Community Advisor

@aemUser2345 

To fix index issues

  • Use the Oak index generation tool - to generate index definition.
  • Add indexing under node oak:index.
  • Trigger the re-index.

you can also use the tool to analyze :

http://<AEMurl>:<port>/libs/granite/operations/content/diagnosistools/queryPerformance.html
 

https://experienceleague.adobe.com/docs/experience-manager-65/deploying/practices/best-practices-for...

https://experienceleague.adobe.com/docs/experience-cloud-kcs/kbarticles/KA-17492.html?lang=en