How to verify that data ingested into a dataset processed to Profile successfully.

[Luca_Lattarini]

Hi All,

How can I validate/check if the data ingested into dataset processed to Profile successfully?

Thanks

[Lokesh_Shivalingaiah]

one of the quick way is to grab few random IDs from the data set and try to search from the Profile.

[ChetanyaJain]

Hi @Luca_Lattarini 

To get the stats on how many records were ingested, how many moved to profile, how many ids were generated... all this can be derived from the Monitoring dashboard. You can filter by days, check with specific dataflow as well.

For records that failed to move to the profile, I am not sure which API can be utilized to fetch that. But there should be a way to find that.

As @Lokesh_Shivalingaiah suggested you can randomly perform sanity check as well by searching. However for large data to find the profiles that couldn't make it to UPS - I will check if I can find more details.

Thanks,

Chetanya

[poojasinha · Accepted solution]

HI @Luca_Lattarini 

1. Check for the status of the batch in the dataset activity tab.

2. You can use the Retrieve Batch Catalog API to receive the same metrics that you see in the dataset activity tab.

[Luca_Lattarini]

Hello @poojasinha @Lokesh_Shivalingaiah @ChetanyaJain ,

I did some test and I found that check Dataset> Activity and Identies > Browser are the two ways for my marketers. These two ways allow me to monitor from the ingestion part to the Profile side.Is it correct consider that Query Service and Monitoring Batch covers the same ingestion part of Dataset> Activity ?

Thanks

[poojasinha]

Hi @Luca_Lattarini 

If marketers are comfortable using the UI they should rely on the Dataset>Activity and Monitoring Tabs to get the metrics/ information from ingestion to profile.  The identities tab will not give them this information.

Thanks,

Pooja

[Luca_Lattarini]

@poojasinha Understand , Thanks

[aaanuk2016]

Hey,

I have data ingested successfully into dataset few days ago but it has not yet got added to profile. The schema is enabled for "Profile". I noticed that I had only one identifier which was also PK. Shall making an existing field as an additional Identifier, resolve the issue (data not yet added to profile)?

What about the existing data? Do I need to reimport the data?

Thanks.

[poojasinha]

As long as their is one primary identity defined (not identity) on the dataset the data should get ingested in the profile store. Data enters profile store at the time of ingestion so you will have to reload the data to make sure it reaches profile store. Run the Catalog API to lookup the dataset and make sure it has the following two attributes set in the unifiedProfile tag: enabled=true and enabledAt=the timestamp of when the dataset was enabled for profile. If it is not set then disable/enable the profile toggle in the UI and check with the API again. Re-ingest the data again.