Assets Share Commons : The query read or traversed more than 100000 nodes Issue. | Community
Skip to main content
A
AmanK24
Level 2
October 14, 2021

Assets Share Commons : The query read or traversed more than 100000 nodes Issue.

  • October 14, 2021
  • 2 replies
  • 3021 views

Hi Team,
                We are facing search query traversal issue i.e. “The query read or traversed more than 100000 nodes. To avoid affecting other tasks, processing was stopped.”

We have assets stored under the “/content/dam/<project>/<sub-module>”. We are also using Asset share commons 1.94. version for the search page. The query generated while the page load is something like as below.:

2_group.0_path = "path1"

2_group.1_path = "path2"

2_group.2_path = "path3"

2_group.3_path = "path3"

2_group.4_path = "path4"

2_group.p.or = true

4_group.group.2_property = ./jcr:content/metadata/isSearchable

4_group.group.2_property.operation = unequals

4_group.group.2_property.value = false

5_group.mainasset = true

orderby = @6655266:score

orderby.sort = desc

p.guessTotal = 10

p.limit = 10

p.offset = 0

type = dam:Asset

 

The assets under this path /content/dam/<project> is more than 100000 count. The query runs under this path only to get the searchable assets.  This path is enabled with CUG’s. The traversal issue come when users have access to only few assets or different sub module access.

Please provide the possible solution here for search enhancement. And FYI, the number of assets will continue to grow in future.

 

With Regards

Aman Kumar

 

CC: 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

2 replies

lukasz-m
Community Advisor
lukasz-mCommunity Advisor
Community Advisor
October 14, 2021

Hi @amank24, it looks that query you have prepared is not covered by any index. Potential fixes are

  1. Change the query in the way it will be covered by existing definition.
  2. Create or extend existing index to support you query. You can use [1] to generate index definition.
  3. Change traversal limit, however keeping in mind your comments that number of assets will increase, probably this will not be a long term solution.

You can also have a look on this page [2]

 

[1] http://oakutils.appspot.com/generate/index

[2] https://experienceleague.adobe.com/docs/experience-manager-65/developing/bestpractices/troubleshooting-slow-queries.html?lang=en

    A
    AmanK24Author
    Level 2
    October 14, 2021

    Thank you @lukasz-m  for approaches.
    I have few constraint here:

    1. Assets Share Commons is used for generation of Query.

    2. I need to make query with all the number of asset present that's the req. as we cant afford of skipping any search result.

    3. I have already create custom Lucene index for dam:asset and required properties.

     

     

      lukasz-m
      Community Advisor
      lukasz-mCommunity Advisor
      Community Advisor
      October 14, 2021

      Hi @lukasz-m  No I am getting issue even after creating custom index. As i am saying we have already more than 100000 assets.

      Yes, the custom index is only picked.

      Also yes re index after creation of custom index.

      What i feel is my index count might be more than 100000.


      @amank24you can check index size and number of elements in index under /system/console/status-oak-index-stats, can you share query execution plan - this could be useful for further discussion.

        Ankur_Khare
        Community Advisor
        Ankur_KhareCommunity Advisor
        Community Advisor
        October 20, 2021

        Could u pls check these threads-

         

        https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/the-query-read-more-than-500000-nodes-in-memory/m-p/294303

         

        https://experienceleaguecommunities.adobe.com/t5/adobe-experience-manager/the-query-read-or-traversed-more-than-100000-nodes-to-avoid/td-p/357042

         

        I suspect custom index is incorrect-

        Check below link to generate the index based on the query-

         

        http://oakutils.appspot.com/generate/index

         

        And try again.

         

          A
          AmanK24Author
          Level 2
          October 20, 2021

          Thank you @ankur_khare , We have referred http://oakutils.appspot.com/generate/index and generated the index. Our system currently hold 11TB+ data which results in the total number of  1.1 millions assets.
          And when we perform search, we have to perform on the entire assets.

           

                 These assets are categorized in subfolder with CUG enabled.

            Ankur_Khare
            Community Advisor
            Ankur_KhareCommunity Advisor
            Community Advisor
            October 20, 2021

            For me this number"The query read or traversed more than 10000 nodes. To avoid affecting other tasks, processing was stopped" doesn't seems to be that large and u can try increasing the limit-

             

            http://localhost:4502/system/console/jmx/org.apache.jackrabbit.oak%3Aname%3Dsettings%2Ctype%3DQueryEngineSettings

            This is not the permanent solution and a permanent solution would be to optimize the query.

            https://github.com/AdobeDocs/experience-manager-65.en/blob/master/help/sites-developing/troubleshooting-slow-queries.md

             

              Terms & ConditionsAccessibility statement

              Loading footer...