Historical Data ingestion for new additional fields

[Pradeep-Jaiswal]

when a new field is introduced in the schema after it already been enabled for the profile and have the data ingested, how do we ingest the historical data for newly introduce field ? One approach I am ware is create a new dataSet with primary key and new field, and then use that dataset to ingest both historical and incremental data. however this approach may not be scalable, if we introduce new field every quarter, then it would lead to multiple dataset

[AnkitJasani29]

Hi @Pradeep-Jaiswal ,

New dataset is always not needed, you can use existing dataset as well with updated schema.

Prepare a file or batch with the historical data including:

1. The identity field (primary key) used to identify existing profiles (e.g., email, ECID).
2. The new field(s) with historical values.

Use batch ingestion or a dataflow to upload the historical file into the same dataset.

Adobe will merge new data with the existing profiles based on the identity.