Validate Merge Policy with choice DataSetPrecedence

[Luca_Lattarini]

Hi All,

I am validating the Merge Policy has been creating for two different datasets (Dataset 1 and Dataset 2)based on the same XDM Profile schema. Assuming We chose to set DataSetPrecedence as Merge Policy Type with order 1) Dataset 1 2) Dataset 2.

If I retrieve the profile based on email as primary identity, which attribute i will visualize if they have the same attribute but with different value? An example Dataset 1 has attribute loyaltyCard = Gold and Dataset 2 has attribute loyaltyCard = Silver?

Thanks

[ChetanyaJain]

Hi @Luca_Lattarini,

The simple answer is it will set loyaltyCard = Gold - since priority is Dataset 1, irrespective of when the record is ingested.

Thanks!

[ChetanyaJain · Accepted solution]

Hope this helps!

[Luca_Lattarini]

Thanks @ChetanyaJain  very helpful as usual. 

[Danny-Miller]

@ChetanyaJain @Luca_Lattarini You might want to test it out, but I believe field 2 in your example @ChetanyaJain ends up evaluating to "def".

[Luca_Lattarini]

@Danny-Miller in the case the attribute of DS with priority 1 is empty, the final evaluation will show the value of the attribute (not empty) of DS with priority 2? 

[Danny-Miller]

That is my understanding, but I haven't tested this specific scenario.  Worth testing out.

[ChetanyaJain]

@Danny-Miller I was checking with Bitun as well, and he was sure there's no preference for non-null fields. Also, we should have a mechanism to adjust at the field level that will be a super helpful feature for future roadmap.

@Luca_Lattarini please let us know the results once you test.

[Luca_Lattarini]

@ChetanyaJain @Danny-Miller After test,  I confirm that there no preference for non null fields. I agree that it could be a useful feature to be added.

[Danny-Miller]

@Luca_Lattarini @ChetanyaJain When you say "no preference", I want to make sure I understand... In the test you did you are saying field 2 ended up being?

1. [field doesn't exist on profile]
2. "def"

Also, just wanted to make sure you loaded the data in the order laid out in the example.

Thanks for testing this.

BTW, I so rarely use this because of a few reasons, but good to know how it works.

[bitunsen-2022]

Hi Danny,

The testing was done in the order it was laid out in the example Chetanya had put together.

In AEP, we only have 2 options/ways to define the construction of profile unification:

• Tiimestamp based
• Dataset precedence

In this testing, based on the selected merge policy (Dataset precedence), the Dataset 1 wins (as it has highest precedence). As, in Dataset 1, "field 2" is NULL, the unified profile will not have this "field 2". This is how "Dataset precedence" works.

What I understood from this thread is - Luca is looking for an additional Option ("Data Completeness") - if my winning dataset does not have any value to a particular field and other loosing datasets are having values for that field, then the populated value needs to be selected from the next lower precedent dataset.

This is a very common Data survivorship functionality of Master Data Management.

@Luca_Lattarini ,

I second @Danny-Miller's thought, I also rarely use or plan to use "Dataset precedence". If I decide to use this, I would perform a deep level analysis to understand why I really need this.

From my own perspective - this is a core MDM problem and MDM is solving this for more than a decade. This should be handled outside AEP, must be in MDM or in ETL layer (if no MDM is available and you have this requirement). We need to understand the sole purpose of CDP - what it is meant to solve - we should not consider CDP as a magic box.

If we start solving all the data problems in CDP, then it will loose the feasibility - #tradeoff.

My perspective may be wrong from others point of view and I am always open to discuss on this.

[Luca_Lattarini]

@bitunsen-2022 

I wanted to verify again what u mentioned above where ->

In this testing, based on the selected merge policy (Dataset precedence), the Dataset 1 wins (as it has highest precedence). As, in Dataset 1, "field 2" is NULL, the unified profile will not have this "field 2". This is how "Dataset precedence" works.

And If I search under Profile->Browser, the Merge Policy with precedence 1) DS1 2) DS2  . In the case DS1 has some empty field, It will be showed under profile attributes the DS2 field which is not empty.

Thanks

Luca

[Prithwiraj_Deb]

This question and following discussion really cleared my doubt and I also agree with the suggestion that AEP should not try to mimic every feature of MDM. The field level validation of null values for data-set precedence will not give proper cost benefit advantage and I don't see any important field to be "Optional" and field is mandatory the XDM validation during ingestion would have failed anyway.