Expand my Community achievements bar.

July 31st AEM Gems Webinar: Elevate your AEM development to master the integration of private GitHub repositories within AEM Cloud Manager.
SOLVED

Too many dam nodes need to be traversed

Avatar

Level 3

I hvave more than 800000 images in the dam. When I publish or unpublish a page, AEM will auto find the reference imges for this page,  it will take a lot of time. This caue a page successfully be published will take serveral minutes. 

 

Does anyone know how can I reduce the search time? 

 

Where and how can I create a index for the query of the follow log. 

 

 

POST /bin/wcmcommand HTTP/1.1] org.apache.jackrabbit.oak.plugins.index.Cursors$TraversingCursor Traversed 810000 nodes with filter Filter(query=select [jcr:path], [jcr:score], * from [dam:Asset] as a where [jcr:content/contentFragment] = 'true' and contains(*, '/content/demo/test') and isdescendantnode(a, '/content/dam') order by [jcr:created] desc option(index tag [visualSimilaritySearch]) /* xpath: /jcr:root/content/dam//element(*, dam:Asset)[(jcr:content/@contentFragment = 'true' and jcr:contains(., '/content/demo/test'))] order by :created descending option (index tag visualSimilaritySearch) */ fullText="/content/demo/test", path=/content/dam//*, property=[:indexTag=[visualSimilaritySearch], jcr:content/contentFragment=[true]]); consider creating an index or changing the query

 

 

AEM : 6.5

Service Package : 6.5.11

 

Thanks,

Forrest

1 Accepted Solution

Avatar

Correct answer by
Level 4

Hello there,

When trying to move/delete/publish either Content Fragments or Sites/Pages, there is an issue when Content Fragment references are fetched, as the background query will fail; i.e. the functionality will not work.
To ensure correct operation you need to add the following properties to the index definition node /oak:index/damAssetLucene (no re-indexing is required) :

 

"tags": [
"visualSimilaritySearch"
]
"refresh": true

 

Hope that helps. Thanks!

View solution in original post

19 Replies

Avatar

Community Advisor

Hi @ForrestLi, in general there are really good guidance about search indexes. This could be good starting point for you.

Looking on the information form the log you have shared, I was able to quickly generate below index rules, using Oak Index Definition Generator, you can try to use it.

<?xml version="1.0" encoding="UTF-8"?><jcr:root xmlns:dam="http://www.day.com/dam/1.0" xmlns:jcr="http://www.jcp.org/jcr/1.0">
  <myIndex compatVersion="{Long}2" async="async" jcr:primaryType="oak:QueryIndexDefinition" evaluatePathRestrictions="{Boolean}true" type="lucene">
    <indexRules jcr:primaryType="nt:unstructured">
      <dam:Asset jcr:primaryType="nt:unstructured">
        <properties jcr:primaryType="nt:unstructured">
          <contentFragment name="jcr:content/contentFragment" propertyIndex="{Boolean}true" jcr:primaryType="nt:unstructured"/>
          <created name="jcr:created" ordered="{Boolean}true" jcr:primaryType="nt:unstructured"/>
        </properties>
      </dam:Asset>
    </indexRules>
  </myIndex>
</jcr:root>

Alternatively, if this query is related to OOTB AEM functionality, you can also contact Adobe Support team to provide package with proper index definition or extension to existing one.

Please also keep in mind one important thing, you should avoid creating new index if existing can be extended.

Avatar

Level 3

I add the property "contentFragment" under the damAssetLucene index. It is show me "The index definition has changed without reindexing." How can I make this new propery to work?

 

ForrestLi_0-1647590861554.png

 

 

Avatar

Level 4

Adding that property might be helpful but not the solution. Add properties tags and refresh to the damAssetLucene node as given below:

 

"tags": [
"visualSimilaritySearch"
]
"refresh": true

 

Note that tags is a multi String and refresh is a boolean value. Run the query tool right after and you'll notice that it'll now use damAssetLucene index.

Avatar

Level 2

Hi Alisahali

 

Can you please share me the screen shot with the below changes ,

 

"tags": [
"visualSimilaritySearch"
]
"refresh": true

 

 

Avatar

Employee Advisor

Can you run that query on the queryPerformance tool [1]? This query should use the damAssetLucene index.

And then please raise a support ticket with Adobe support and reference this thread.

 

thanks,

Jörg 

 

[1] https://experienceleague.adobe.com/docs/experience-manager-65/administering/operations/operations-da...

Avatar

Level 3

I run the query on the queryPerformance tool, it is showing "No indexes were used. This is a traversal query."

Avatar

Level 4

Hey Forrestli, I ran into the same problem this morning and resolved it by adding tags and refresh properties to damAssetLucene index. Note that reindexing is not required after you add them. Your query should start using damAssetLucene right after. Also, it is a known issue with one of the 6.5 service pack. Follow the answer that I posted earlier. Thanks! 

Avatar

Correct answer by
Level 4

Hello there,

When trying to move/delete/publish either Content Fragments or Sites/Pages, there is an issue when Content Fragment references are fetched, as the background query will fail; i.e. the functionality will not work.
To ensure correct operation you need to add the following properties to the index definition node /oak:index/damAssetLucene (no re-indexing is required) :

 

"tags": [
"visualSimilaritySearch"
]
"refresh": true

 

Hope that helps. Thanks!

Avatar

Level 4

No, you don't need a new node. Just add them as new properties directly to the damAssetLucene node please. Sorry for the confusion. For example - it should be at the same level as the reindex property if you were to perform reindexing i.e. directly against the damAssetLucene node and not the 'property' node under it.

Avatar

Level 3

Thanks Alisahali. After I changed the property of the damAssetLucene, my query currently is using the index "damAssetLucene(/oak:index/damAssetLucene)". 

Avatar

Level 2

Hi, I'm facing the same problem and the query keeps using traversal, even after adding the properties. Judging from the replies, I understand this is the location of the properties, can you confirm? For some reason, I don't see a screenshot in your previous reply.

tomekniedzwiedz_0-1652282712196.png

 

Not sure what happened, but I just retried that on another instance and it worked. The one where it originally failed seems fine too. I can confirm that this is the right config.

Avatar

Level 1

Hello there,

 

Sorry for the late response, but yes that is the correct configuration

 

Tip: you can also use the query performance tool to check if the query is using a given index or not for testing purposes.

 

Regards,

S.A

Avatar

Community Advisor

rebuild all of the indexes and ensure that there is no traversals.

 

Also, Try to indentify the list of properties that OOTB reference search executes. Ensure that those properties are indexed.