How to use data prep functions for streaming data via HTTP API

[xliu]

Hi there,

Here is our use case: We have an event schema and in the schema there is an array field: "Status | string[]". And we send event data payload to AEP in the below format: 

"_companyname": {
    "user_id": "1234567",
    "test_id": "123456",
    "status": ['CREATED'],
},
"eventType": "test",
"producedBy": "test",
"timestamp": "2023-03-18T14:43:53.000Z",
"_id": "xxxxxxxxx"

and it works as expected, there is an event in AEP associated with the profile. But for the field 'status', if we would like to append some value to it, how should I  update the payload by using these data prep functions? Or are there any other solutions? We tried the payload below, but failed. (_id is identical for both payloads)

"_companyname": {
    "user_id": "1234567",
    "test_id": "123456",
    "status": "upsert_array_append(['UPDATED'])",
},
"eventType": "test",
"producedBy": "test",
"timestamp": "2023-03-18T14:43:53.000Z",
"_id": "xxxxxxxxx"

[Kumar_Saurabh_ · Accepted solution]

Hi @xliu,

To use any dataprep function for data transformation, we have to apply the logic to the dataflow created to onboard this streaming HTTP API source data.

This option to add logic is available while creating the dataflow under the Mapping section—Add Calculated Field.

Documentation link for Creating HTTP API with Dataprep logic:

https://experienceleague.adobe.com/en/docs/experience-platform/sources/ui-tutorials/create/streaming...

Navigate to the - Map Standard Fields in the documentation.

Documentation for all the available Data Prep Functions including upsert array:

https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/functions

Regards,

Kumar Saurabh

[xliu]

Thank you very much for your reply @Kumar_Saurabh_ 

I tried to follow the doc you shared, but I have some questions:

1. When I create mapping fields and upload a sample data file, am I supposed to upload a JSON file in the below format? And the format is according to the page here https://developer.adobe.com/experience-platform-apis/references/streaming-ingestion/#tag/Streaming-I... 

{
    "header":
    {
        "schemaRef":
        {
            "id": "https://ns.adobe.com/companyname/schemas/123345",
            "contentType": "application/vnd.adobe.xed+json;version=1"
        },
        "flowId": "12345",
        "datasetId": "12345"
    },
    "body":
    {
        "xdmMeta":
        {
            "schemaRef":
            {
                "id": "https://ns.adobe.com/loyaltyone/schemas/123345",
                "contentType": "application/vnd.adobe.xed+json;version=1"
            }
        },
        "xdmEntity":
        {
            "_companyname":
            {
                "userid": "888",
                "test_id": "123456",
                "status":
                [
                    "UPDATED_0208"
                ]
            },
            "eventType": "test",
            "producedBy": "self",
            "timestamp": "2023-03-25T14:43:53.000Z",
            "_id": "8880-123457"
        }
    }
}

2. When I create a calculated field, I cannot select the array field from left side bar, in this example, I cannot see the "Status" field, do I miss any steps?

Thank you!

[Kumar_Saurabh_]

Hi @xliu ,

There are 2 points to consider in this solution:-

1) The source data should not be already formatted as an array. You will have to modify your source to pass the Status field value as a string, and then use data prep function to Append Array.

2) The _id column value must be unique for different events else the data will be rejected for the UPS ingestion.

Hope this helps!

Regards,
Kumar Saurabh