Historical Data ingestion for new additional fields | Community
Skip to main content
Pradeep-Jaiswal
Pradeep-Jaiswal
Level 5
May 19, 2025
Question

Historical Data ingestion for new additional fields

  • May 19, 2025
  • 2 replies
  • 383 views

when a new field is introduced in the schema after it already been enabled for the profile and have the data ingested, how do we ingest the historical data for newly introduce field ? One approach I am ware is create a new dataSet with primary key and new field, and then use that dataset to ingest both historical and incremental data. however this approach may not be scalable, if we introduce new field every quarter, then it would lead to multiple dataset

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

2 replies

AnkitJasani29
AnkitJasani29
Level 6
May 20, 2025

Hi @pradeep-jaiswal ,

New dataset is always not needed, you can use existing dataset as well with updated schema.

Prepare a file or batch with the historical data including:

  1. The identity field (primary key) used to identify existing profiles (e.g., email, ECID).
  2. The new field(s) with historical values.

Use batch ingestion or a dataflow to upload the historical file into the same dataset.

Adobe will merge new data with the existing profiles based on the identity.

    Pradeep-Jaiswal
    Pradeep-JaiswalAuthor
    Level 5
    July 2, 2025

    This approach would work for the profile or lookup based dataset, but how about event based datasets ? Even if we find unique Key for each historical event and re-ingest them with only new attributes (excluding existing attributes) it would be appended instead of overwritten records.

    kautuk_sahni
    Community Manager
    kautuk_sahniCommunity Manager
    Community Manager
    July 2, 2025

    @pradeep-jaiswal Just checking in — were you able to resolve your issue?
    We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!

    Kautuk Sahni
    Terms & ConditionsAccessibility statement

    Loading footer...