Data Distiller vs Query Service in AEP

[SaurabhPa5]

what is difference between Data Distiller and Query Service in AEP? In what ways are they similar or different?

[Asheesh_Pandey]

@SaurabhPa5  In AEP, Data Distiller and Query Service are closely related but distinct capabilities for working with data stored in the AEP Data Lake. This FAQ document answers your questions about Query Service and Data Distiller. It also includes commonly seen error codes while using the “Queries” product for data validation or writing transformed data back to the data lake.

[Imbalakumar]

Hi @SaurabhPa5 

Query Service and Data Distiller are distinct, complementary components that provide specific data querying capabilities.

Query Service is designed for ad hoc queries to explore, validate, and experiment with ingested data without altering the data lake.

In contrast, Data Distiller focuses on batch queries that transform and enrich data, with results stored back into the data lake for future use. Batch queries in Data Distiller can be scheduled, monitored, and managed, supporting deeper data processing and manipulation that Query Service alone does not facilitate.

You can refer the adobe documentation to learn more about these differences and capabilities - https://experienceleague.adobe.com/en/docs/experience-platform/query/troubleshooting-guide

Thanks,

Bala

[Shubham_AEP · Accepted solution]

This confusion is very normal for someone who heard the Data Distiller very first time, this might have happened because of the Query service UI.

Yes both Data Distiller and Query service have use the same UI in AEP, the only difference between Query Service and Data Distiller in Adobe Experience Platform (AEP) is mostly about evolution, features, and capabilities.

Query Service - A raw SQL engine for validation, exploration and analyzing data in AEP.

Data Distiller - Data Distiller is a package offering that includes a subset of the functionalities from Adobe Experience Platform. With Data Distiller you can perform post-ingestion data preparation (such as cleaning, shaping, and manipulation) for real-time customer profile or analytical use cases by executing batch queries in Query Service. Your use of Data Distiller is dependent on your entitlement for Experience Platform-based applications.

So till the time you are performing Validation, exploration or general anlayzation of your Post ingested data, this will serve you as the Query Service Feature, The moment you use this output to insert into the another dataset, or to map this output in audience we call it Data DIstiller Audience, now this feature is known as the Data Distiller capability.

Scenario                                                  Cappability called in that case
If Exploring data with ad hoc SQL queries                          - Query Service
If Creating scheduled jobs or reusable datasets                 - Data Distiller
If Preparing data for Journey Orchestration or RTCDP       - Data Distiller
If Debugging datasets interactively                                     - Query Service

Product Document for better Understanding: https://experienceleague.adobe.com/en/docs/experience-platform/query/data-distiller/overview

https://experienceleague.adobe.com/en/docs/experience-platform/query/ui/overview

[KevinFosterDX]

Another way to look at it is that you cannot use SQL statements such as INSERT or CREATE TABLE without a license to Data Distiller. Nor can you schedule any queries without that license.

Without a license to Data Distiller you are limited to interactive SELECT statements executed in the AEP UI or via a SQL client tool such as psql.