Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

Indexes going in Traversal

Avatar

Avatar
Give Back
Level 2
Kundan_Ray1
Level 2

Likes

3 likes

Total Posts

13 posts

Correct Reply

0 solutions
Top badges earned
Give Back
Ignite 1
Validate 1
Shape 1
Boost 3
View profile

Avatar
Give Back
Level 2
Kundan_Ray1
Level 2

Likes

3 likes

Total Posts

13 posts

Correct Reply

0 solutions
Top badges earned
Give Back
Ignite 1
Validate 1
Shape 1
Boost 3
View profile
Kundan_Ray1
Level 2

30-08-2020

Hi ,

Although we have indexes (property,lucene ) defined for most of the queries,  few of the indexes are going in traversal

 

For Ex :

 

On Restart of AEM

 

 

*WARN* [Apache Sling Repository Startup Thread] org.apache.jackrabbit.oak.plugins.index.search.spi.query.FulltextIndex$FulltextPathCursor Index-Traversed 140000 nodes with filter Filter(query=SELECT * FROM [nt:base] WHERE ISDESCENDANTNODE([/content]) AND [cq:master] IS NOT NULL, path=/content//*, property=[cq:master=[is not null]])

 

Therefore it's taking a long time for restart.

 

OOTB ntBaseLucene

Numdocs count - 5725504/oak:index/ntBaseLucene

 

Property Field Info - /oak:index/ntBaseLucene cq:master 165543

But logs we see it's going for Index-Traversed although we have index defined.

 

What is the reason for Index Traversal ? This is happening for lower numdocs indexes also. How do we fix this ?

Accepted Solutions (1)

Accepted Solutions (1)

Avatar

Avatar
Give Back 3
Level 4
SundeepKatepally
Level 4

Likes

76 likes

Total Posts

76 posts

Correct Reply

21 solutions
Top badges earned
Give Back 3
Boost 10
Boost 25
Boost 50
Affirm 10
View profile

Avatar
Give Back 3
Level 4
SundeepKatepally
Level 4

Likes

76 likes

Total Posts

76 posts

Correct Reply

21 solutions
Top badges earned
Give Back 3
Boost 10
Boost 25
Boost 50
Affirm 10
View profile
SundeepKatepally
Level 4

31-08-2020

I have got similar issue. 

I have reindexed all OOTB indexes and setting reindex flag = true , then this issue got resolved.

Answers (3)

Answers (3)

Avatar

Avatar
Ignite 1
Level 4
Andrei_Dantsou
Level 4

Likes

35 likes

Total Posts

29 posts

Correct Reply

14 solutions
Top badges earned
Ignite 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile

Avatar
Ignite 1
Level 4
Andrei_Dantsou
Level 4

Likes

35 likes

Total Posts

29 posts

Correct Reply

14 solutions
Top badges earned
Ignite 1
Boost 5
Boost 3
Boost 25
Boost 10
View profile
Andrei_Dantsou
Level 4

31-08-2020

Hi @Kundan_Ray1,

Oak chooses the indexer with the lowest estimated cost. (https://docs.adobe.com/content/help/en/experience-manager-65/deploying/deploying/queries-and-indexin...).
You can go to Tools > Operations > Dashboard > Diagnosis > Query Performance > Explain Query (/libs/granite/operations/content/diagnosistools/queryPerformance.html), execute you query and check costs for ntBaseLucene vs traversal.
query-perf.png


If the cost for ntBaseLucene is higher, it might indicate that indexes are broken/not actual due to various reasons (such as improper instance shutting down during indexing) and thus triggering re-index might be considered for ntBaseLucene (set the reindex property to true within /oak:index/ntBaseLucene). It is best to re-index during quiet periods (for example, not during a large content ingest), and ideally during maintenance windows when AEM's load is known and controlled (https://docs.adobe.com/content/help/en/experience-manager-65/deploying/deploying/troubleshooting-oak...).

However, I guess querying is not only reason for slow instance starting up. While activation the MSM core bundle ("Day Communique 5 WCM Multi Site Management Core") invokes the query you mentioned - "SELECT * FROM [nt:base] WHERE ISDESCENDANTNODE([/content]) AND [cq:master] IS NOT NULL" and checks if found nodes exist (please see the stack trace in the thread com.day.cq.wcm.msm.impl.LiveCopyFinderProvider.buildBloomFilter - https://experienceleaguecommunities.adobe.com/t5/adobe-experience-cloud-ideas/msm-core-fails-to-star...). The query returns live copy configuration nodes (e.g. /content/we-retail/us/en/community/members/jcr:content/cq:LiveSyncConfig) and according to your message, the number of found nodes is large (>Numdocs count - 5725504) so processing them might introduce a delay. I'd also recommend reviewing the content structure to ensure that no unnecessary nodes are present (such as unnecessary pages + their live copies, etc.)

Regards

Avatar

Avatar
Contributor
Employee
hamidk92094312
Employee

Likes

103 likes

Total Posts

240 posts

Correct Reply

38 solutions
Top badges earned
Contributor
Shape 1
Ignite 1
Give Back 50
Give Back 5
View profile

Avatar
Contributor
Employee
hamidk92094312
Employee

Likes

103 likes

Total Posts

240 posts

Correct Reply

38 solutions
Top badges earned
Contributor
Shape 1
Ignite 1
Give Back 50
Give Back 5
View profile
hamidk92094312
Employee

30-08-2020

It's not clear to me if this is your query or its an OOTB query ? It's searching "nt:base". The Oak index definition suggest such query to have an index with the following properties:

 

- compatVersion = 2
- async = "async"
- jcr:primaryType = oak:QueryIndexDefinition
- evaluatePathRestrictions = true
- type = "lucene"
+ indexRules
+ nt:base
+ properties
+ master
- name = "cq:master"
- propertyIndex = true
- notNullCheckEnabled = true

 

If the existing index does not have the suggested properties you may add it and rebuild the index to verify if it makes any improvements.

Avatar

Avatar
Give Back
Level 2
Kundan_Ray1
Level 2

Likes

3 likes

Total Posts

13 posts

Correct Reply

0 solutions
Top badges earned
Give Back
Ignite 1
Validate 1
Shape 1
Boost 3
View profile

Avatar
Give Back
Level 2
Kundan_Ray1
Level 2

Likes

3 likes

Total Posts

13 posts

Correct Reply

0 solutions
Top badges earned
Give Back
Ignite 1
Validate 1
Shape 1
Boost 3
View profile
Kundan_Ray1
Level 2

26-09-2020

Hi @Andrei_Dantsou , Thanks for your inputs . We have given our inputs to Adobe Product Engineering as we have seen this issue happening only in 6.5 and not in 6.3 , with the same number of livecopy nodes present there. As this is a generic search query triggered during restart where result count is more than 10k, it goes for index-traversal. In future, this may be removed from startup msm core bundle activation to on-demand.