How to handle huge amount of Content Fragments with GraphQL | Adobe Higher Education
Skip to main content
1
12nalk123
Level 2
March 8, 2023
Resuelto

How to handle huge amount of Content Fragments with GraphQL

  • March 8, 2023
  • 2 respuestas
  • 2678 visualizaciones

We have following structure in our project

  • /content/dam/customerA
    • it contains 300k+ Content Fragments of type CFMofA
    • GraphQL Endpoint customer-A
  • /content/dam/customerB
    • it contains 10 Content Fragments of type CFMofB
    • GraphQL Endpoint customer-B

Now we try to send a GraphQL query like this to get a list of all Content Fragments of customer B to Endpoint B

query { CFMofBList{ items { ... } } }

But unfortunately we always run into this error:

"Exception while fetching data (/CFMofBList) : The query read or traversed more than 100000 nodes. To avoid affecting other tasks, processing was stopped."
 
And in the "Query Performance" tools we can see that internally AEM creates this query:
SELECT asset.* FROM [dam:Asset] AS asset WHERE ISDESCENDANTNODE(asset, '/content/dam') AND asset.[jcr:content/contentFragment] = CAST('true' AS BOOLEAN) AND asset.[jcr:content/data/cq:model] = '/conf/customerB/settings/dam/cfm/models/CFMofB' ORDER BY asset.[jcr:path]

Explaining the query shows that this index is used "damAssetLucene(/oak:index/damAssetLucene)"

 

 

What can we do here? Best for us would be to restrict the path of the query to "/content/dam/customerB" but this is not supported by AEM as far as I know and the query is generated internally by the AEM GraphQL implementation. Increasing the read limit in org.apache.jackrabbit.oak.query.QueryEngineSettingsService did not help either.

Este tema ha sido cerrado para respuestas.
Mejor respuesta de aanchal-sikka

hello @12nalk123 

 

I guess the index doesn't have all properties defined.

Example: when i checked on my local damAssetLucene doesn't index "jcr:content/data/cq:model"

 

So, AEM might be using this index to get results, but they are still too many to be verified for cq:model.

 

Please try adding it to damAssetLucene , index and then check

2 respuestas

ChitraMadan
Community Advisor
ChitraMadanCommunity Advisor
Community Advisor
March 8, 2023

Hi @12nalk123 ,

 

We too faced the same issue with content fragments in AEM 6.5.

 

Installing this package fixed the issue for us, hopefully it should work for you too.

AEM Content Fragment with GraphQL Index Package 1.0.3

 

Thanks,

Chitra

    1
    12nalk123Autor
    Level 2
    March 8, 2023

    Thank you for the suggestion but we have already installed "AEM Content Fragment with GraphQL Index Package 1.0.5".

    Also I forgot to mention we have AEM 6.5.14

      ChitraMadan
      Community Advisor
      ChitraMadanCommunity Advisor
      Community Advisor
      March 9, 2023

      Hi @12nalk123 ,

       

      Did you try re-indexing?

        aanchal-sikka
        Community Advisor
        aanchal-sikkaCommunity AdvisorRespuesta
        Community Advisor
        March 9, 2023

        hello @12nalk123 

         

        I guess the index doesn't have all properties defined.

        Example: when i checked on my local damAssetLucene doesn't index "jcr:content/data/cq:model"

         

        So, AEM might be using this index to get results, but they are still too many to be verified for cq:model.

         

        Please try adding it to damAssetLucene , index and then check

        Aanchal Sikka
          1
          12nalk123Autor
          Level 2
          March 9, 2023

          Hi @aanchal-sikka ,

           

          thank you very much for this hint. After adding the property to the index the query works now.

          The only question I have now is why the intended index "fragments" from the "AEM Content Fragment with GraphQL Index Package" is not used and instead damAssetLucene is used?

           

          Edit: Now I know why it was not picked, the index has the property "tags=['fragments']" so the query would need to look like this in order to be picked:

           

          SELECT asset.* FROM [dam:Asset] AS asset WHERE ISDESCENDANTNODE(asset, '/content/dam') AND asset.[jcr:content/contentFragment] = CAST('true' AS BOOLEAN) AND asset.[jcr:content/data/cq:model] = '/conf/xxx/settings/dam/cfm/models/xxx' ORDER BY asset.[jcr:path] option(index tag fragments)

           

          But that is not the case so the damAssetLucence is picked.

           

            K
            khannapiyush36
            Level 2
            August 30, 2023

            Hi @12nalk123,

            Thanks so much for sharing the insights on option(index tag fragments). I struggled for this property update even after creating a valid index for node dam:cfVariationNode. But still it did not pick up. After going over you post & doing a comparision with OOTB asset index, I figured out the gap. Thanks again!

            Términos y condicionesAccessibility statement

            Loading footer...