SUGGESTION : Computing on real time streaming

[jayakrishnaaparthasarathy]

Hi Team,

We have data coming from different sources, let’s say sources as SOURCE X, SOURCE Y SOURCE Z and sources will be kept on adding as per business requitement. Those SOURCES X Y Z contains the data about customer profile and their respective purchase history. All those are being streamed through realtime and we are good on the same.

Question: we would like to compute some of the attribute data from these different sources . Like SOURCE X attribute1 + SOURCE Y attribute2 + SOURCE Z attribute3. So, since these data are streaming in real time with XDM compatible we are not able to perform any calculation. However, I have suggested to do that computation around the actual source system itself (in this case it is AWS) and ingest those attributes alone in separate schema and datasets itself.

Would it sound good way of having the computation data on the AEP system? Looking for the long-term approach to do this activity. Else please do let me know on other way around to perform this calculation.  

[Anuhya-Y]

@jayakrishnaaparthasarathy 

It is advisable to compute data outside AEP and push data via streaming into AEP. This way latest data would be reflected in profile in real time.

For some reason, if you are looking for computation in AEP. Yes ,it is possible using query service/Data Distiller which allows customers to perform post-ingestion data preparation and save the computed data in a new profile enabled dataset. In this case computed data may not reflect in real time in profile as the data travels from data lake to profile store.

[jayakrishnaaparthasarathy]

@Anuhya-Y thanks for your reply. Yes, Data distiller will perform data transformation for the post-ingested data only and that too will take only batch method to send the enriched customer data through distiller. So, it is not in our choise for this use case.

lets wait for other experts answer on the same. 

[ChetanyaJain]

@jayakrishnaaparthasarathy AEP is not designed for ETL or heavy transformation. If any of that needs to be done, the recommendation would be to manage this outside AEP. You can use tools like (Ab-Initio, Informatica Powercenter, Talend, Snowflake, etc) to consolidate data from various sources and apply required transformations/filters. 

As you mentioned, Data Distiller is post-ingestion and will also require a license - it may work for specific scenarios but it totally depends on the use case at hand.

[jayakrishnaaparthasarathy]

Hi @ChetanyaJain Thank you so much for your reply, I agree that AEP doesn't support ETL and my question doesn't fall in ETL process and we are not expecting AEP to transform any bulk data being ingested in to AEP. However, my intial impression was that  if anybody could have done the real time computing over the attributes using any features. Unfortunately, seems AEP doesn't support any real time computation (though it supports batch, calculation field). So as suggested, it is good and best practice to have all those computing outside the AEP (whichever is sourse system) and send the actual data in realk time to AEP.

[Anil_Umachigi · Accepted solution]

To add all the comments here, it will eventually depend on how often you will receive new request and how soon can you make those edits to data source. 

Although in Alpha, does Computed attributes can be considered for future to get this done especially for temporary attributes for use cases. 

In our implementation experience we have for now relied on Query service ( Data Data Distiller) to enrich profiles on daily basis for list of calculated insights for profile, A disclaimer here, this can get really heavy and complex if the list becomes bigger as time goes on. 

Hope that helps 

Anil 

[jayakrishnaaparthasarathy]

Hi @Anil_Umachigi , thanks for your reply. Hope, you are referring the same and would be one of the valued featutes from AEP once it is in LIVE to leverage same. Really looking forward on this computed attributes.
https://experienceleague.adobe.com/docs/experience-platform/profile/computed-attributes/overview.htm...

[Anil_Umachigi]

@jayakrishnaaparthasarathy  Yes that's the one!