Clarification on Dataset Lookup Node Behavior with Duplicate Lookup Keys in AJO | Community
Skip to main content
R
RakeshK1
Level 2
June 26, 2026
Solved

Clarification on Dataset Lookup Node Behavior with Duplicate Lookup Keys in AJO

  • June 26, 2026
  • 2 replies
  • 53 views

Hello ALL,

I have a question regarding the new Dataset Lookup node in Adobe Journey Optimizer (AJO).

Scenario

I have configured a Dataset Lookup using the following lookup key:

Lookup Key: 123

Sample dataset:

Column1 (Lookup Key) Column2 Timestamp
123 Yes 6/26
123 No 6/26
123 Yes 6/25

Question

How does the Dataset Lookup node behave when there are multiple records matching the same lookup key?

Specifically:

  • Does AJO return the first matching record, last matching record, most recent record based on timestamp, or is the behavior non-deterministic?
  • Is there any documented logic or prioritization used when duplicate lookup key values exist in the dataset?
  • Are duplicate lookup keys supported, or is the lookup key expected to be unique within the dataset?

Observation

In my test, when the lookup key 123 matches multiple records, the journey consistently evaluates to the "Others" condition rather than the expected paths.

I would like to understand:

  1. Whether this is the expected behavior when duplicate records exist for the lookup key.
  2. If there are any best practices or requirements to enforce uniqueness on the lookup field.
  3. Whether there is official documentation describing how Dataset Lookup handles duplicate matches.

Any clarification would be appreciated.

Thank you

Best answer by RakeshK1

Follow-up Questions

Based on the explanation above, let's assume that each individual batch contains unique lookup key values, but the same lookup key can exist across multiple batches over time.

In that scenario:

  • When a Dataset Lookup is performed, will AJO always retrieve the record from the latest ingested batch, or is there another prioritization logic that determines which record is returned?

My use case involves unsubscribe data, where duplicate records for the same customer are common. The latest record should ideally determine the current subscription status.

I am evaluating alternative approaches and would appreciate guidance on best practices.

One option I am aware of is implementing a Marketing Action / suppression check directly on the Email node as a last-mile validation or create Targeting Rules or audience-based filtering( I assume the profile should be enabled for profile to use this) which in my case, is not possible.

Could you clarify the following:

  1. Are there recommended patterns in AJO for handling unsubscribe datasets that contain historical duplicates?
  2. Are there any best-practice architectures for enforcing the latest unsubscribe status before message delivery in my scenario?

Understanding how AJO resolves duplicate records across Dataset Lookups will help us determine the most reliable approach for unsubscribe compliance.

Thank you.

2 replies

DavidKangni
Community Advisor
DavidKangniCommunity Advisor
Community Advisor
June 28, 2026

Hi ​@RakeshK1,

I had the same issue and here is the answer I got from Adobe PM team.

Hi David, got an answer for you:

https://experienceleague.adobe.com/en/docs/journey-optimizer/using/data-management/lookup-aep-data
Learn how to use Adobe Experience Platform datasets in Journey Optimizer Decisioning and personalization capabilities.
we do not pickup a random value, unless
you have a batch of data that has duplicate KEYs
 so if you have a key twice in the dataset then it will be a random choice
the unique identity that you are using for the primary key must be unique in the batch of data

 

Thanks,

David

David Kangni
R
RakeshK1AuthorAccepted solution
Level 2
June 29, 2026

Follow-up Questions

Based on the explanation above, let's assume that each individual batch contains unique lookup key values, but the same lookup key can exist across multiple batches over time.

In that scenario:

  • When a Dataset Lookup is performed, will AJO always retrieve the record from the latest ingested batch, or is there another prioritization logic that determines which record is returned?

My use case involves unsubscribe data, where duplicate records for the same customer are common. The latest record should ideally determine the current subscription status.

I am evaluating alternative approaches and would appreciate guidance on best practices.

One option I am aware of is implementing a Marketing Action / suppression check directly on the Email node as a last-mile validation or create Targeting Rules or audience-based filtering( I assume the profile should be enabled for profile to use this) which in my case, is not possible.

Could you clarify the following:

  1. Are there recommended patterns in AJO for handling unsubscribe datasets that contain historical duplicates?
  2. Are there any best-practice architectures for enforcing the latest unsubscribe status before message delivery in my scenario?

Understanding how AJO resolves duplicate records across Dataset Lookups will help us determine the most reliable approach for unsubscribe compliance.

Thank you.

I
IamCGK
Level 3
June 29, 2026

Hey ​@RakeshK1 

I’d encountered a similar scenario and found a peculiar way to resolve it. The recommended solve is to use uniquely generated values for (uuid/guid) _id during data ingestion for differentiating duplicate records. This way the lookup store consistently fetches the latest record ingested (the old ones will be cleared/replaced from the lookup store).

If you want to generate unique _id records within AEP, then kindly follow the below steps and see if that works for you

  • In AEP, Navigate to Workflows section and launch  Map CSV to XDM schema

     

  • Select the target dataset and data.csv file you want to ingest
  • During data mapping/data prep step, select the “New Field type” and “Add Calculated field” option. This opens a separate tool box for you to add functions.

     

  • Use uuid() data prep function and map it to _id and hit Finish

     

Once the workflow is done, the lookup store will be updated with newest record (latest record ingested). I was able to confirm this by replicating a similar setup.

 

Now some answers for your questions

  • Does AJO return the first matching record, last matching record, most recent record based on timestamp, or is the behavior non-deterministic?
    •  The behavior is non-deterministic when there are duplicate records with no uniqueness enforced
  • Is there any documented logic or prioritization used when duplicate lookup key values exist in the dataset?
    • No, it is not mentioned in the documentation. But there is a key general recommendation mentioned in this document
  • Are duplicate lookup keys supported, or is the lookup key expected to be unique within the dataset?
    • Generally lookup keys are recommended to be unique however as per the above technique, the lookup cache can be cleared out/replaced by the unique _id records. 

Hope these answers your questions!

 

Regards,

Ganesh Kumar C

R
RakeshK1Author
Level 2
June 29, 2026

​@IamCGK Are you suggesting that we ingest records using the same UUID for the same entity? This would cause the new record to replace the existing one, allowing the lookup to always return the most recent value.

If so, there is already a lot of data ingested and lot of journeys already use this as source. Is there some other way possible?.

Checking with Adobe Support on the side. 

I
IamCGK
Level 3
June 29, 2026

​@RakeshK1 - Not sure I understand your question properly. If I understand your question correct, then you’re looking to fetch the latest ingested record per lookup key right?

 

If that’s the case, I meant pairing the lookup key along with the uuid() function — which generates different uuid() value during data ingestion — Not the same uuid value for the same entity. This means this will only replace that particular lookup key record and the remaining records based on different lookup keys remain as-is. 

 

As an example, the first record from the below table will always be chosen unless a new record with a different uuid() & same lookup key is ingested 

 

Column1 (Lookup Key) Column2 _id (Ingested latest to oldest)
123 Yes 4a44d3ea-9c58-4767-8d5e-2a38645a4e9e
123 No b63490fd-b4c8-45a1-bce4-859b12b90d29
123 Yes ad7a384c-e63c-4371-9a44-10946d34afa6
Terms & ConditionsAccessibility statement

Loading footer...