Expand my Community achievements bar.

Enable a dataset for UPSERT profile using APIs which ingestion will work on this

Avatar

Level 2

Once the dataset is enabled for Upsert functionality via API, can it also be used for Streaming ingestion?

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

7 Replies

Avatar

Community Advisor

Hello @Snowwhite5378 

 

Yes you can use streaming ingesting after a dataSet has been enabled for upsert, but please be aware there are some limitations and inbound payload changes.

 

Please refer to this article.

 

https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/upserts

Avatar

Level 2

Hi @brekrut 

 

Problem Statement  :

Once data is ingested through streaming for a particular profile. Ex. (name :xyz , country :spain )in this we need to update the name of an attribute value . For that profile without getting the data override /deleted of other attributes .Is there any way to update one field and retain all the data for that profile  .

Is there any way to do this and if yes can you please let me know the steps .

 

Avatar

Community Advisor

Hello @Snowwhite5378 

 

If you are streaming data for into a dataSet which has been enabled for upsert, you will need to use the merge operation as part of the inbound payload.

 

https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/upserts#payload-with-identi...

 

When a dataSet is not configured for upsert the last write of data upon the dataset will be reflected upon profile.

Avatar

Level 2

Hi @brekrut 

Still facing the same issue 

Steps i followed

Created Two dataset

  • Profile dataset In this dataset i have enable this for profile and upsert
  • Identity dataseIn this i have disable the profile and enable the identity service as per doc 
  • I have ingested one record through streaming in profile dataset i can see  profile is created 
  • when again i have ingested same record with same individual id just to update specific attribute that attribute got updated but rest got deleted 
  • i have follow this payload to send the incremental record 
  • curl -X POST 'https://dcs.adobedc.net/collection/9aba816d350a69c4abbd283eb5818ec3583275ffce4880ffc482be5a9d810c4b' \
    -H 'Content-Type: application/json' \
    -H 'x-adobe-flow-id: d5262d48-0f47-4949-be6d-795f06933527' \
    -d '{
    "header": {
    "flowId" : "d5262d48-0f47-4949-be6d-795f06933527",
    "imsOrgId": "{ORG_ID}",
    "datasetId": "62259f817f62d71947929a7b",
    "operations": {
    "data": "merge",
    "identity": "create",
    "identityDatasetId": "6254a93b851ecd194b64af9e"
    }
    },
    {
     "body": {
           
            "xdmEntity": {
                "tenantID": {
                    "individualId": "678996061",
                    "patchTestObject": {
                        "country": "singapore"
                       
                    }
                },
                "_id": "/uri-reference",
               
                "createdByBatchID": "/uri-reference",
                "extSourceSystemAudit": {
                    "createdBy": "Sample value",
                    "createdDate": "2018-11-12T20:20:39+00:00",
                    "externalID": "Sample value",
                    "externalKey": {
                        "sourceID": "Sample value",
                        "sourceInstanceID": "Sample value",
                        "sourceKey": "Sample value",
                        "sourceType": "Sample value"
                    },
                   
                    "lastUpdatedDate": "2024-06-05T05:19:15+00:00"
                   
                }
             
            }
        }

    }

Avatar

Community Advisor

Hello @Snowwhite5378 

 

Can you please try this payload for upsert againt the data collection.  Notice the xdmEntity is changed to xdmUpdate

{
"header": {
"msgType": "xdmEntityUpdate",
"msgId": "{{guid}}",
"xactionId": "{{current_timestamp}}{{guid}}",
"createdAt": "{{epoch_time}}",
"originalTimestamp": "{{epoch_time}}",
"datasetId": "<dataSetID>",
"dataStreamId": "",
"imsOrgId": "<IMSORG>",
"xdmSchema": {
"name": "_xdm.context.profile",
"version": "1.0"
},
"source": {
"name": "<sourceName>"
}
},
"body": {
"xdmEntityUpdates": {
"_<tenant>": {
<Attributes are here>

}
}
}
}

 

Avatar

Level 1

Enabling dataset for Profile, using the API (isUpsertEnabled) is only for enabling upsert on batch ingestion. 

sreeCharan73_0-1718018208487.png


To use the Streaming Upserts, you could either use XDM compatible or non XDM payloads.

Although, your payload looks like XDM payload, for streaming upsert, you could adopt XDM inCompatible ingestion, and follow the https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/upserts#:~:text=this%20exam... and remove the XDM Entity object(in the above) and make sure to have the mapping in the data prep (Dataflow).