Issue in identifying latest information of a type of Experience Event

[bitunsen-2022]

Hello Team,

Need some guidance on resolving one issue related to ExperienceEvent while using them in creating Segment. Though it is mutable, we had to choose one of my dimension data as ExperienceEvent as that is having 1-to-many relationship with my customer record (array will also not work as number of records are sometimes more than 100 per customer record). For example: customer has many accounts, and account details (example: status, monthly average balance) can get changed in time. So one customer can have multiple events for a same account with different values in a certain timeframe. In this situation, how to get the latest event of the account, if I have to build a segment based on the latest information of the account?

Also looking for some suggestion if there is any better way of modeling the XDM schemas for 1 to many relationships for mutable dimensions ? Array option as Record schema wont work, as during update via real time or batch, we have to rebuild the whole array.

Thanks in advance for the help.

[ericknee · Accepted solution]

@bitunsen-2022 its a bit hard to grok the issue without a diagram but I'll do my best to answer this.

In the relational world what you are describing a 1:N relationship to another entity object called account. It would typically be modeled like you see below.  With Experience Platform we only support a single entity defined as Person which is composed of fragments of either traits of that person (XDM Individual Profile) or behaviors of that person (XDM Experience Event). In the relational world to bridge the gap between two different entities you would build a bridge table as you see below with Customer Account.  

When you go to model this in Experience Platform you cannot think relationally anymore as our system is built on NoSQL. This means that other entities have to be denormalized back into one of the fragment types (XDM Individual Profile or XDM Experience Event).  Below is what we would be expected to be modeled:

You have to denormalize the bridge table and the entity into an array on the profile if you want to know the current accounts each person owns. This does mean you have to think through the data engineering side of this.

• Batch feeds - you will need to perform some form of data engineering to also ensure you re-create this array each time you update the customer record.
  • (optionally) - you could create two datasets (XDM Individual Profile) where one dataset contains the customer traits minus the account details and another dataset contains the cusId + account details in an array.
• Stream feeds - you will need to build a serverless function to construct the array of accounts and append to the customer record any time you have an update if you are doing full record re-instatement
  • (optionally) - create two datasets like in batch but just stream the pieces in separately 

The only reason to model accounts as events in this scenario is if there is a business case where the customer is asking the history of the how account is changing over time. That doesn't appear to be the case here so I would not model this as an event as that is the intention from the relational model.

[bitunsen-2022]

Thanks a lot Erick for your detailed level of response. I really appreciate the way you responded. You appropriately represented the model. 

I was bit hesitant in representing Accounts using array in profile, because I have some customers who are having more than 5000 accounts. I know those are edge cases (3% of total customer population), still wanted to understand if there is any size limit or restrictions per profile for an array (I am aware of the total size limit per profile following the guardrails).

[Ankita05]

How did you resolve this and what is the approach you took to address this issue?
I have the similar situation to get the latest status of the subscription and segment on top of it.
But I also know that array's have some storage limitations. 

[ericknee]

@bitunsen-2022 there is a limitation the array size which you read about in the docs here. In short arrays should be <500 elements deep if you are de-normalizing back into a XDM Individual Profile class-based schema.  I question the value of modeling accounts as array here because unless you have a business use case you are just creating bloat for an engineering exercise.  Usually what I've seen is array's created that hold the most recent 10, last 10, etc.  

What is the business use case for bringing the accounts into the model?  Are you modeling for a B2B scenario?  

Additional context is needed to provide direction beyond what I've shared above as the business case is incredibly important here.