Too many dam nodes need to be traversed

[ForrestLi]

I hvave more than 800000 images in the dam. When I publish or unpublish a page, AEM will auto find the reference imges for this page,  it will take a lot of time. This caue a page successfully be published will take serveral minutes. 

Does anyone know how can I reduce the search time? 

Where and how can I create a index for the query of the follow log. 

POST /bin/wcmcommand HTTP/1.1] org.apache.jackrabbit.oak.plugins.index.Cursors$TraversingCursor Traversed 810000 nodes with filter Filter(query=select [jcr:path], [jcr:score], * from [dam:Asset] as a where [jcr:content/contentFragment] = 'true' and contains(*, '/content/demo/test') and isdescendantnode(a, '/content/dam') order by [jcr:created] desc option(index tag [visualSimilaritySearch]) /* xpath: /jcr:root/content/dam//element(*, dam:Asset)[(jcr:content/@contentFragment = 'true' and jcr:contains(., '/content/demo/test'))] order by :created descending option (index tag visualSimilaritySearch) */ fullText="/content/demo/test", path=/content/dam//*, property=[:indexTag=[visualSimilaritySearch], jcr:content/contentFragment=[true]]); consider creating an index or changing the query

AEM : 6.5

Service Package : 6.5.11

Thanks,

Forrest

[lukasz-m]

Hi @ForrestLi, in general there are really good guidance about search indexes. This could be good starting point for you.

• https://experienceleague.adobe.com/docs/experience-manager-65/deploying/practices/best-practices-for...
• https://experienceleague.adobe.com/docs/experience-manager-65/deploying/deploying/queries-and-indexi...

Looking on the information form the log you have shared, I was able to quickly generate below index rules, using Oak Index Definition Generator, you can try to use it.

<?xml version="1.0" encoding="UTF-8"?><jcr:root xmlns:dam="http://www.day.com/dam/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0">
  <myIndex compatVersion="{Long}2" async="async" jcr:primaryType="oak:QueryIndexDefinition" evaluatePathRestrictions="{Boolean}true" type="lucene">
    <indexRules jcr:primaryType="nt:unstructured">
      <dam:Asset jcr:primaryType="nt:unstructured">
        <properties jcr:primaryType="nt:unstructured">
          <contentFragment name="jcr:content/contentFragment" propertyIndex="{Boolean}true" jcr:primaryType="nt:unstructured"/>
          <created name="jcr:created" ordered="{Boolean}true" jcr:primaryType="nt:unstructured"/>
        </properties>
      </dam:Asset>
    </indexRules>
  </myIndex>
</jcr:root>

Alternatively, if this query is related to OOTB AEM functionality, you can also contact Adobe Support team to provide package with proper index definition or extension to existing one.

Please also keep in mind one important thing, you should avoid creating new index if existing can be extended.

[ForrestLi]

I add the property "contentFragment" under the damAssetLucene index. It is show me "The index definition has changed without reindexing." How can I make this new propery to work?

[Alisahali]

Adding that property might be helpful but not the solution. Add properties tags and refresh to the damAssetLucene node as given below:

"tags": [
"visualSimilaritySearch"
]
"refresh": true

Note that tags is a multi String and refresh is a boolean value. Run the query tool right after and you'll notice that it'll now use damAssetLucene index.

[asarafali]

Hi Alisahali

Can you please share me the screen shot with the below changes ,

"tags": [
"visualSimilaritySearch"
]
"refresh": true

[Jörg_Hoh]

Can you run that query on the queryPerformance tool [1]? This query should use the damAssetLucene index.

And then please raise a support ticket with Adobe support and reference this thread.

thanks,

Jörg 

[1] https://experienceleague.adobe.com/docs/experience-manager-65/administering/operations/operations-da...

[ForrestLi]

I run the query on the queryPerformance tool, it is showing "No indexes were used. This is a traversal query."

[Alisahali]

Hey Forrestli, I ran into the same problem this morning and resolved it by adding tags and refresh properties to damAssetLucene index. Note that reindexing is not required after you add them. Your query should start using damAssetLucene right after. Also, it is a known issue with one of the 6.5 service pack. Follow the answer that I posted earlier. Thanks! 

[Alisahali · Accepted solution]

Hello there,

When trying to move/delete/publish either Content Fragments or Sites/Pages, there is an issue when Content Fragment references are fetched, as the background query will fail; i.e. the functionality will not work.
To ensure correct operation you need to add the following properties to the index definition node /oak:index/damAssetLucene (no re-indexing is required) :

"tags": [
"visualSimilaritySearch"
]
"refresh": true

Hope that helps. Thanks!

[ForrestLi]

Hi Alisahali, Is the new node like this?

[Alisahali]

No, you don't need a new node. Just add them as new properties directly to the damAssetLucene node please. Sorry for the confusion. For example - it should be at the same level as the reindex property if you were to perform reindexing i.e. directly against the damAssetLucene node and not the 'property' node under it.

[ForrestLi]

Like this?

[Alisahali]

Hang on. Lemme attach a screenshot

[Alisahali]

Like this

[ForrestLi]

Thanks Alisahali. After I changed the property of the damAssetLucene, my query currently is using the index "damAssetLucene(/oak:index/damAssetLucene)". 

[tomek-niedzwiedz]

Hi, I'm facing the same problem and the query keeps using traversal, even after adding the properties. Judging from the replies, I understand this is the location of the properties, can you confirm? For some reason, I don't see a screenshot in your previous reply.

[tomek-niedzwiedz]

Not sure what happened, but I just retried that on another instance and it worked. The one where it originally failed seems fine too. I can confirm that this is the right config.

[Alisahali66]

Hello there,

Sorry for the late response, but yes that is the correct configuration

Tip: you can also use the query performance tool to check if the query is using a given index or not for testing purposes.

Regards,

S.A

[SundeepKatepally]

rebuild all of the indexes and ensure that there is no traversals.

Also, Try to indentify the list of properties that OOTB reference search executes. Ensure that those properties are indexed.

[LyleX]

Thanks, fixed my issue.