Skip to main content
Level 2
Nov 17, 2026 11:47 AM UTC
Question

Profile Records in Data Lake

I would like to clarify how profile records are physically stored in the AEP Data Lake when using a dataset based on the XDM Individual Profile class.

  1. Assume that a Profile-enabled dataset receives the following first batch:

customerId | name | city

123 | John | Warsaw

456 | Anna | Krakow

 

  1. Second ingestion: 

customerId | name | city

123 | John | Wroclaw

How the data would like in the Data Lake? Like that?

customerId | name | city

123 | John | Warsaw

456 | Anna | Krakow

123 | John | Wroclaw

1 reply

SatheeskannaK
Adobe Champion, Community Advisor, UG Leader
Adobe Champion, Community Advisor, UG Leader
Nov 17, 2026 3:03 PM UTC

Hi ​@Dex 666, Yes, you’re correct.

Data Lake (dataset level):

  • Ingestion is append-only and batch-based. Batches are immutable, so there are no upserts and nothing is overwritten.
  • After the two ingestions, the dataset holds 3 rows: 123/John/Warsaw, 456/Anna/Krakow, and 123/John/Wroclaw.
  • Data is stored as Parquet files grouped by batch, with system metadata per row (such as ingestion time).

Profile Store (where “latest wins” applies)

  • When these batches are also loaded into the Profile Store, where attributes are merged per profile.
  • John’s city becomes Wroclaw, and Anna is unchanged. Profile Store keeps no attribute history.
  • With the default Timestamp Ordered merge policy, the newest timestamp wins. For batch ingestion without an explicit timestamp, ingestion time is used. For streaming, the record timestamp is used when provided.

Things to consider,

  • Identity is required: customerId must be defined as an identity (primary identity plus namespace) so Profile can tie rows to the same person.
  • Ordering matters: A batch with an older timestamp can lose to a newer value already in Profile.
  • Merge policies: Timestamp Ordered is the default. Dataset Precedence lets a specific dataset always override another, regardless of timestamp. Merge policies apply when the same attribute comes from multiple datasets, and they are evaluated at profile read time (segmentation, activation), not at ingestion.
  • Retention: Data Lake retention (dataset TTL or deletion) is separate from Profile Store retention.
Translate
Thanks, Sathees