Lookup entity volume and visualization

[StephDRX]

Dear,

We are implementing a lookup entity containing static information (such as store information). We have created a relationship and activated both schemas and datasets for profile.

Our source needs to update these static information frequently through a streaming source.

There is a limit of 5GB on these lookup entities. A few questions following this limit:

- Can you confirm you can update records in the lookup entity by using an existing primary identity and updating other attributes (e.g. store address for a given store ID) ?

- Does the 5GB limit apply to the dataset of the lookup entity? Or only the data in the profile store (i.e. the latest updated version of the record)? If it applies to the dataset, our regular updates on store records could end up polluting the dataset and we could reach the 5GB limit. If the 5GB limit is on the profile store, we would be OK.
- How can you visualize the data stored in the profile store (i.e. the latest updated version of the record)? Could you give an example of a query?

Kind regards,
Stephan

[arijitg · Accepted solution]

Hi @StephDRX 

- Can you confirm you can update records in the lookup entity by using an existing primary identity and updating other attributes (e.g. store address for a given store ID) ? Yes you can update records based on primary identity, latest record will override older one (considering record class lookup).

- Does the 5GB limit apply to the dataset of the lookup entity? Or only the data in the profile store (i.e. the latest updated version of the record)? If it applies to the dataset, our regular updates on store records could end up polluting the dataset and we could reach the 5GB limit. If the 5GB limit is on the profile store, we would be OK. 5 GB limit applies on the applies to the dataset of the lookup entity, not just the data in the profile store.

- How can you visualize the data stored in the profile store (i.e. the latest updated version of the record)? Could you give an example of a query? To use query best practice is to use now() in data prep to pass the ingestion time. OOTB External Audit field group can be used to host that. Then you can sort your query result to get the latest record for that Primary Identity path. 

Regards,

Arijit Ghosh