Column-level filtering for Web SDK data ingested into AEP Profile
Hi,
We are currently reviewing how to control which fields of WebSDK datastream are ingested into Real-Time Customer Profile while still retaining the full dataset in the Data Lake.
For the Adobe Analytics source connector, Adobe provides the Column-level filtering capability, which allows specific columns to be selected for ingestion into Profile while the remaining columns continue to be available in the Data Lake:
For example, with Analytics data we could configure the ingestion so that only eVar1, eVar2 and eVar3 are included in Profile, while the rest of the Analytics fields are still ingested into the Data Lake but do not contribute to Profile.
We would like to achieve the same behavior for data collected directly through Adobe Experience Platform Web SDK.
I have the following questions:
-
Is there an equivalent column-level filtering capability for Web SDK datasets, where we can select which XDM fields are ingested into Profile while keeping the complete event payload in the Data Lake?
-
If this capability is not currently available for Web SDK data, what is the recommended Adobe architecture or approach for preventing unnecessary fields from being ingested into Profile while still retaining them in the Data Lake for analytical use?
Our objective is mainly to minimize unnecessary data stored in Profile, while preserving the richer event data in the Data Lake for downstream analytics and use cases such as Customer Journey Analytics.
Thanks in advance for any guidance or recommended best practices.