Adobe Experience Platform Query Service

[SundarSritharan]

Hi Team,

We have huge dataset with TBs of data in AEP . In order to fetch data from project firefly , we are using query service api calls. Seems like it is very slow in performance. Is there any other way to fetch AEP dataset's data very faster or to boost up query performance?

@dr_venture.

[dr_venture · Accepted solution]

There will always be a small additional call time added when using Firefly actions to access the AEP query service.  It is another layer between the client and the data.  That being said it should be a small addition when compared to calling query service directly.
It depends on your use case but the value added by using Firefly Actions is the ability to use the State and File storage layer to do caching of data. 

Please see this thread on caching of data

https://experienceleaguecommunities.adobe.com/t5/project-firefly-questions/action-result-max-size-an...

It depends on your use case but the slight call time hit might be worth it if you can cache some of the data and increase your applications response time greatly.

[maggod]

I've seen similar issues with query service.

Its not about the minor time which is added by your firefly application. Its about the performance of the query service itself.

Yes, you can try using some caching concept. But even with that, at some point you need to refresh. Right? And if it then takes 2 mins (yes, I've seen this!) in any case your custom marketing application's user experience is destroyed.

The overarching question for me is: Is it recommended by Adobe to use the query service at all for custom applications? (no matter if Firefly or other)

[dr_venture]

good points

you could use firefly and put in an Alarm (cron) job that would refresh the cached dataset independent of the users interactions.  The user would never hit the 2min refresh but it would still be there.

Not trying to under play the need for fast preformant API's of course.  It would be ideal if this caching was not needed at all.  I understand the fustration. I don't work in AEP engineering or product management so I dont really know the answer to your question about correct use.  I would recomend asking this question in their forums.  

BTW if you get a good answer that changes what we know about the use of that API please circle back and let me know.  I would love to understand it better as well. 

[Jacob-DDdev]

I'd love to see the ability to define a custom index on AEP datasets. Not sure if Adobe is worried about the extra storage costs involved, but the performance gains would make the tool way more approachable.

Idea submitted, here.