How to handle huge amount of Content Fragments with GraphQL

[12nalk123]

We have following structure in our project

• /content/dam/customerA
  • it contains 300k+ Content Fragments of type CFMofA
  • GraphQL Endpoint customer-A
• /content/dam/customerB
  • it contains 10 Content Fragments of type CFMofB
  • GraphQL Endpoint customer-B

Now we try to send a GraphQL query like this to get a list of all Content Fragments of customer B to Endpoint B

query {
  CFMofBList{
    items {
      ...
    }
  }
}

But unfortunately we always run into this error:

"Exception while fetching data (/CFMofBList) : The query read or traversed more than 100000 nodes. To avoid affecting other tasks, processing was stopped."

And in the "Query Performance" tools we can see that internally AEM creates this query:

SELECT asset.* FROM [dam:Asset] AS asset WHERE ISDESCENDANTNODE(asset, '/content/dam') AND asset.[jcr:content/contentFragment] = CAST('true' AS BOOLEAN) AND asset.[jcr:content/data/cq:model] = '/conf/customerB/settings/dam/cfm/models/CFMofB' ORDER BY asset.[jcr:path]

Explaining the query shows that this index is used "damAssetLucene(/oak:index/damAssetLucene)"

What can we do here? Best for us would be to restrict the path of the query to "/content/dam/customerB" but this is not supported by AEM as far as I know and the query is generated internally by the AEM GraphQL implementation. Increasing the read limit in org.apache.jackrabbit.oak.query.QueryEngineSettingsService did not help either.

[ChitraMadan]

Hi @12nalk123 ,

We too faced the same issue with content fragments in AEM 6.5.

Installing this package fixed the issue for us, hopefully it should work for you too.

AEM Content Fragment with GraphQL Index Package 1.0.3

Thanks,

Chitra

[12nalk123]

Thank you for the suggestion but we have already installed "AEM Content Fragment with GraphQL Index Package 1.0.5".

Also I forgot to mention we have AEM 6.5.14

[ChitraMadan]

Hi @12nalk123 ,

Did you try re-indexing?

[12nalk123]

yes, we also excluded "/content/dam/customerA" once from the index and then the query worked.

But excluding the whole path in damAssetLucene is not really an option.

[aanchal-sikka · Accepted solution]

hello @12nalk123 

I guess the index doesn't have all properties defined.

Example: when i checked on my local damAssetLucene doesn't index "jcr:content/data/cq:model"

So, AEM might be using this index to get results, but they are still too many to be verified for cq:model.

Please try adding it to damAssetLucene , index and then check

[12nalk123]

Hi @aanchal-sikka ,

thank you very much for this hint. After adding the property to the index the query works now.

The only question I have now is why the intended index "fragments" from the "AEM Content Fragment with GraphQL Index Package" is not used and instead damAssetLucene is used?

Edit: Now I know why it was not picked, the index has the property "tags=['fragments']" so the query would need to look like this in order to be picked:

SELECT asset.* FROM [dam:Asset] AS asset WHERE ISDESCENDANTNODE(asset, '/content/dam') AND asset.[jcr:content/contentFragment] = CAST('true' AS BOOLEAN) AND asset.[jcr:content/data/cq:model] = '/conf/xxx/settings/dam/cfm/models/xxx' ORDER BY asset.[jcr:path] option(index tag fragments)

But that is not the case so the damAssetLucence is picked.

[khannapiyush36]

Hi @12nalk123,

Thanks so much for sharing the insights on option(index tag fragments). I struggled for this property update even after creating a valid index for node dam:cfVariationNode. But still it did not pick up. After going over you post & doing a comparision with OOTB asset index, I figured out the gap. Thanks again!