DataStream filtering before data getting ingested to AEP dataLake

[Pradeep-Jaiswal]

Consider dataStream have 10 fields from profile enabled schema. is there a way to restrict few fields in AEP dataLake ingestion, while getting all fields forwarded to other service such as Adobe Analytics ? So out of 10 fields , only 3 fields goes into dataLake but all 10 field goes into adobe analytics ? IS this even a possibility, I dont find any filtering feature in data mapping UI. 

[Abie]

Hi @Pradeep-Jaiswal ,

One way I could think of is - If you wish to prevent some fields from reaching the destination, then don't have those fields in the destination schema. The datastream forwards the events data to destinations like AEP dataset, analytics etc. If AEP dataset doesn't have a valid field for the incoming event data, it won't be mapped and will be ignored. Similarly, if there is no mapping for a given field in analytics, it will be ignored.

Unfortunately, there is no OOTB solution for this problem, but only such tricks to solve it. I have bumped into a similar problem in the past, investigated datastreams quite deeply but no luck.

I hope it helps. Thanks!