Why do I see multiple profiles in a single request?

[Pareek1]

I am trying to send an audience to a webhook to test the output of my custom created DSDK, but in the output I see several duplicate profiles and some of those occur in a single request as well. I am ingesting the profiles using the HTTP API from Postman, please tell me what might be causing this issue.

[abhinavbalooni]

Hey @Pareek1 

When you say duplicate profiles. Which field is the identity here ? Is it marked as an identity field in AEP ? Did you check under profiles? Is it coming up as one profile ?

[Pareek1]

Hi @abhinavbalooni the Email Address is the primary identity here, yes it is marked as an identity field in AEP, yes I had checked in profiles and it is coming up as one profile. I am using a pebble template in my server config, that might be causing the issue but I am unsure how, so adding the unescaped template.
There are many profiles that are coming up only once, but some profiles have this problem.

{% for profile in input.profiles %}
{
   "body": {
     "CreateLeadRequest": {
       "MessageId": null,
       "MessageType": null,
       "IntObjectName": null,
       "IntObjectFormat": null,
       "ListOfANTM Creat Lead IO": {
         "LeadInfo": [ {
           "Email Address": "{{profile.attributes.Email_Address}}",
           "Reference_Number":"{{profile.attributes.Reference_Number}}",
            "Response_Create_Date":"{{profile.attributes.Response_Create_Date}}",
            "Routing_Code":"{{profile.attributes.Routing_Code}}",
            "Vendor_Lead_Id":"{{profile.attributes.Vendor_Lead_Id}}",
            "Vendor_Source":"{{profile.attributes.Vendor_Source}}"
                   }]
       }
     }
   }
 }{% if not loop.last %},{% endif  %}
{%endfor%}

[abhinavbalooni]

Thanks for the response @Pareek1 

I am not familiar with Pebble so won't try to comment on it. But here is my hypothesis. HTTP API streaming destination sends a profile through whenever there is an audience update or a profile update. As you mentioned, this is happening for a few profiles, my thinking is that in these cases, the user profile might be getting updated frequently in a very short duration which might be getting picked up due to this reason.

To test the anive hypothesis, try looking at the complete snapshot of the profile for the exported piece. Try exporting as much of the profiles as possible to see what is the delta in the exports. This seems to be the only reason why it might be happening.

Cheers,

Abhinav

[Pareek1]

Hi @abhinavbalooni, thanks for responding, I found out that the pebble template isn't the issue, because I tried activating the audience to another webhook destination using HTTP API connector available in AEP, but I got the duplicates for the same profiles in the HTTP API connection as well. As you have said, looks like the profiles are getting updated frequently in a very short duration, but I am not sure why that might be happening.
I had one more related question, what happens if I send duplicate profiles which have exact same attributes?

[abhinavbalooni]

Hey @Pareek1 

I don't see a challenge in handling duplicate profiles if those are exact replicas. I would have just chosen the latest payload one for my purpose, as these are edge cases.

Cheers,

Abhinav

[Pareek1]

Hi @abhinavbalooni, what if I add a profile with same primary id (Email Address here), with other attributes being different, will that update the previously existing profile? Does this thing depend on the merge policy? 

[abhinavbalooni · Accepted solution]

@Pareek1 Now you are talking about data ingestion into AEP instead of sending data out of AEP.

To answer your question, it depends on a few factors. If you have upsert enabled on the dataset which you are using to ingest profile information, then your profile would be filled with information for fields which are new to the profile and the existing fields would get updated (Upsert - Update + Insert). If your dataset is not upsert enabled, the latest profile information sent through would overwrite the existing profile. This would hold true in line with a time based merge policy. Since the data in this scenario is being sent to only one dataset, hence dataset precedence merge policy would not be relevant/applicable.

Hope the above helps.

Cheers,

Abhinav

[Pareek1]

Hi @abhinavbalooni, Thanks a lot. This will surely help me.

[abhinavbalooni]

All good @Pareek1 Happy to help. 

You can accept the solution if it works for you so that others can look for the same bit that worked for you.

Cheers,

Abhinav