Data prep while ingesting data

[arpan-garg]

Hi ,

I have some data preparation that needs to be done while ingesting activity data from a source via streaming API's. Specifically, I am looking to set the eventType as "online.seller.event" if the person associated with the event is an online seller, and as "online.buyer.event" if the person associated with the event is an online buyer. However, information on whether the person is an online buyer or seller is only available in the profile dataset and is not present in the event table in the source system. I was wondering if there is a way to perform this calculation on-the-fly using the data-prep functions in AEP. Perhaps by querying the profile dataset associated with the ECID and setting the eventType to "online.buyer.event" if the person is an online buyer.

I would appreciate any guidance or suggestions you may have on this matter. Thank you for your time.

[arijitg]

Sorry @arpan-garg AEP has limited ETL capability and data-prep functions does not support such target table lookup feature that you're expecting.

[arpan-garg]

Thanks for the info @arijitg , whats the ideal way to achieve this ? Would you recommend doing all such calculations outside AEP and then ingest the data.

[arijitg · Accepted solution]

That's correct @arpan-garg it's recommended that we do all joins, lookup, aggregations etc to the source and send the finalized data to AEP.

[Tof_Jossic]

Not entirely sure if this helps but since ETL is mentioned here, I thought I'd point to the following document: 

• Developing ETL Integrations for AEP - https://experienceleague.adobe.com/docs/experience-platform/etl/home.html?lang=en