Historical Data ingestion for new additional fields

[Pradeep-Jaiswal]

when a new field is introduced in the schema after it already been enabled for the profile and have the data ingested, how do we ingest the historical data for newly introduce field ? One approach I am ware is create a new dataSet with primary key and new field, and then use that dataset to ingest both historical and incremental data. however this approach may not be scalable, if we introduce new field every quarter, then it would lead to multiple dataset

[AnkitJasani29]

Hi @Pradeep-Jaiswal ,

New dataset is always not needed, you can use existing dataset as well with updated schema.

Prepare a file or batch with the historical data including:

1. The identity field (primary key) used to identify existing profiles (e.g., email, ECID).
2. The new field(s) with historical values.

Use batch ingestion or a dataflow to upload the historical file into the same dataset.

Adobe will merge new data with the existing profiles based on the identity.

[Pradeep-Jaiswal]

This approach would work for the profile or lookup based dataset, but how about event based datasets ? Even if we find unique Key for each historical event and re-ingest them with only new attributes (excluding existing attributes) it would be appended instead of overwritten records.

[kautuk_sahni]

@Pradeep-Jaiswal Just checking in — were you able to resolve your issue?
We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!