When a new field is introduced in the event schema after it already been enabled for the profile and have the data ingested, how do we ingest the historical data for newly introduce field ?
Some of the solution i could think of are,
option1) One approach I am ware is create a new dataSet with primary key and new field, and then use that dataset to ingest both historical and incremental data. however this approach may not be scalable. if we introduce new field every quarter, then it would lead to multiple datasets.
option2) Wipe out historical data from the dataset by deleting the batch which had ingested that historical data, and re-ingest the data with all attribute. this is time consuming, and can disturb existing prod setup.
option3) use data distiller to backfill the historical data. can that be done technically as event based schema class doesnt support upsert?
Are there any best recommended approach ?
