SUGGESTION : Computing on real time streaming | Community
Skip to main content
jayakrishnaaparthasarathy
Community Advisor
jayakrishnaaparthasarathyCommunity Advisor
Community Advisor
October 27, 2022
Solved

SUGGESTION : Computing on real time streaming

  • October 27, 2022
  • 3 replies
  • 1604 views

Hi Team,

We have data coming from different sources, let’s say sources as SOURCE X, SOURCE Y SOURCE Z and sources will be kept on adding as per business requitement. Those SOURCES X Y Z contains the data about customer profile and their respective purchase history. All those are being streamed through realtime and we are good on the same.

Question: we would like to compute some of the attribute data from these different sources . Like SOURCE X attribute1 + SOURCE Y attribute2 + SOURCE Z attribute3. So, since these data are streaming in real time with XDM compatible we are not able to perform any calculation. However, I have suggested to do that computation around the actual source system itself (in this case it is AWS) and ingest those attributes alone in separate schema and datasets itself.

Would it sound good way of having the computation data on the AEP system? Looking for the long-term approach to do this activity. Else please do let me know on other way around to perform this calculation.  

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Anil_Umachigi

To add all the comments here, it will eventually depend on how often you will receive new request and how soon can you make those edits to data source. 

Although in Alpha, does Computed attributes can be considered for future to get this done especially for temporary attributes for use cases. 

 

In our implementation experience we have for now relied on Query service ( Data Data Distiller) to enrich profiles on daily basis for list of calculated insights for profile, A disclaimer here, this can get really heavy and complex if the list becomes bigger as time goes on. 

 

Hope that helps 

Anil 

 

3 replies

Anuhya-Y
Community Advisor
Anuhya-YCommunity Advisor
Community Advisor
October 27, 2022

@jayakrishnaaparthasarathy 

It is advisable to compute data outside AEP and push data via streaming into AEP. This way latest data would be reflected in profile in real time.

 

For some reason, if you are looking for computation in AEP. Yes ,it is possible using query service/Data Distiller which allows customers to perform post-ingestion data preparation and save the computed data in a new profile enabled dataset. In this case computed data may not reflect in real time in profile as the data travels from data lake to profile store.

    jayakrishnaaparthasarathy
    Community Advisor
    jayakrishnaaparthasarathyCommunity AdvisorAuthor
    Community Advisor
    October 27, 2022

    @anuhya-y thanks for your reply. Yes, Data distiller will perform data transformation for the post-ingested data only and that too will take only batch method to send the enriched customer data through distiller. So, it is not in our choise for this use case.

    lets wait for other experts answer on the same. 

    ChetanyaJain-1
    Community Advisor
    ChetanyaJain-1Community Advisor
    Community Advisor
    October 29, 2022

    @jayakrishnaaparthasarathy AEP is not designed for ETL or heavy transformation. If any of that needs to be done, the recommendation would be to manage this outside AEP. You can use tools like (Ab-Initio, Informatica Powercenter, Talend, Snowflake, etc) to consolidate data from various sources and apply required transformations/filters. 

    As you mentioned, Data Distiller is post-ingestion and will also require a license - it may work for specific scenarios but it totally depends on the use case at hand.

      jayakrishnaaparthasarathy
      Community Advisor
      jayakrishnaaparthasarathyCommunity AdvisorAuthor
      Community Advisor
      November 1, 2022

      Hi @chetanyajain-1 Thank you so much for your reply, I agree that AEP doesn't support ETL and my question doesn't fall in ETL process and we are not expecting AEP to transform any bulk data being ingested in to AEP. However, my intial impression was that  if anybody could have done the real time computing over the attributes using any features. Unfortunately, seems AEP doesn't support any real time computation (though it supports batch, calculation field). So as suggested, it is good and best practice to have all those computing outside the AEP (whichever is sourse system) and send the actual data in realk time to AEP.

      Anil_Umachigi
      Adobe Employee
      Anil_UmachigiAdobe EmployeeAccepted solution
      Adobe Employee
      October 31, 2022

      To add all the comments here, it will eventually depend on how often you will receive new request and how soon can you make those edits to data source. 

      Although in Alpha, does Computed attributes can be considered for future to get this done especially for temporary attributes for use cases. 

       

      In our implementation experience we have for now relied on Query service ( Data Data Distiller) to enrich profiles on daily basis for list of calculated insights for profile, A disclaimer here, this can get really heavy and complex if the list becomes bigger as time goes on. 

       

      Hope that helps 

      Anil 

       

        jayakrishnaaparthasarathy
        Community Advisor
        jayakrishnaaparthasarathyCommunity AdvisorAuthor
        Community Advisor
        November 1, 2022

        Hi @anil_umachigi , thanks for your reply. Hope, you are referring the same and would be one of the valued featutes from AEP once it is in LIVE to leverage same. Really looking forward on this computed attributes.
        https://experienceleague.adobe.com/docs/experience-platform/profile/computed-attributes/overview.html?lang=en

          Anil_Umachigi
          Adobe Employee
          Anil_UmachigiAdobe Employee
          Adobe Employee
          November 1, 2022

          @jayakrishnaaparthasarathy  Yes that's the one! 

          Terms & ConditionsAccessibility statement

          Loading footer...