Historical Data ingestion (event class) for new additional fields

Question

When a new field is introduced in the event schema after it already been enabled for the profile and have the data ingested, how do we ingest the historical data for newly introduce field ?

Some of the solution i could think of are,

option1) One approach I am ware is create a new dataSet with primary key and new field, and then use that dataset to ingest both historical and incremental data. however this approach may not be scalable. if we introduce new field every quarter, then it would lead to multiple datasets.

option2) Wipe out historical data from the dataset by deleting the batch which had ingested that historical data, and re-ingest the data with all attribute. this is time consuming, and can disturb existing prod setup.

option3) use data distiller to backfill the historical data. can that be done technically as event based schema class doesnt support upsert?

Are there any best recommended approach ?

Michael_Soprano · Answer

Do you want to ingest only one field or whole record (with multiple fields)?

If you already have dataset populated with event data and want to update all the events which are already in the dataset with ONLY ONE FIELD then the best way is:

1. Delete old dataset

Create new one
Ingest records (with new to field) to the new dataset

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded