Expand my Community achievements bar.

Check out the 3rd Edition of the AEP Community Lens for all the latest releases, resources, and Community updates
SOLVED

Data prep while ingesting data

Avatar

Level 4

Hi ,

I have some data preparation that needs to be done while ingesting activity data from a source via streaming API's. Specifically, I am looking to set the eventType as "online.seller.event" if the person associated with the event is an online seller, and as "online.buyer.event" if the person associated with the event is an online buyer. However, information on whether the person is an online buyer or seller is only available in the profile dataset and is not present in the event table in the source system. I was wondering if there is a way to perform this calculation on-the-fly using the data-prep functions in AEP. Perhaps by querying the profile dataset associated with the ECID and setting the eventType to "online.buyer.event" if the person is an online buyer.

I would appreciate any guidance or suggestions you may have on this matter. Thank you for your time.

1 Accepted Solution

Avatar

Correct answer by
Employee

That's correct @arpang16406580 it's recommended that we do all joins, lookup, aggregations etc to the source and send the finalized data to AEP.

4 Replies

Avatar

Employee

Sorry @arpang16406580 AEP has limited ETL capability and data-prep functions does not support such target table lookup feature that you're expecting.

Avatar

Level 4

Thanks for the info @arijitg , whats the ideal way to achieve this ? Would you recommend doing all such calculations outside AEP and then ingest the data.

 

 

Avatar

Correct answer by
Employee

That's correct @arpang16406580 it's recommended that we do all joins, lookup, aggregations etc to the source and send the finalized data to AEP.

Avatar

Employee

Not entirely sure if this helps but since ETL is mentioned here, I thought I'd point to the following document: 

page footer