How to use data prep functions for streaming data via HTTP API | Community
Skip to main content
X
xliu
Level 3
March 26, 2024
Solved

How to use data prep functions for streaming data via HTTP API

  • March 26, 2024
  • 1 reply
  • 1277 views

Hi there,

 

Here is our use case: We have an event schema and in the schema there is an array field: "Status | string[]". And we send event data payload to AEP in the below format: 

"_companyname": { "user_id": "1234567", "test_id": "123456", "status": ['CREATED'], }, "eventType": "test", "producedBy": "test", "timestamp": "2023-03-18T14:43:53.000Z", "_id": "xxxxxxxxx"

 

and it works as expected, there is an event in AEP associated with the profile. But for the field 'status', if we would like to append some value to it, how should I  update the payload by using these data prep functions? Or are there any other solutions? We tried the payload below, but failed. (_id is identical for both payloads)

"_companyname": { "user_id": "1234567", "test_id": "123456", "status": "upsert_array_append(['UPDATED'])", }, "eventType": "test", "producedBy": "test", "timestamp": "2023-03-18T14:43:53.000Z", "_id": "xxxxxxxxx"

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Kumar29917170hcyp

Hi @xliu,

To use any dataprep function for data transformation, we have to apply the logic to the dataflow created to onboard this streaming HTTP API source data.

 

This option to add logic is available while creating the dataflow under the Mapping section—Add Calculated Field.

 

Documentation link for Creating HTTP API with Dataprep logic:

https://experienceleague.adobe.com/en/docs/experience-platform/sources/ui-tutorials/create/streaming/http

Navigate to the - Map Standard Fields in the documentation.

 

Documentation for all the available Data Prep Functions including upsert array:

https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/functions


Regards,

Kumar Saurabh

1 reply

K
Adobe Employee
Kumar29917170hcypAdobe EmployeeAccepted solution
Adobe Employee
March 27, 2024

Hi @xliu,

To use any dataprep function for data transformation, we have to apply the logic to the dataflow created to onboard this streaming HTTP API source data.

 

This option to add logic is available while creating the dataflow under the Mapping section—Add Calculated Field.

 

Documentation link for Creating HTTP API with Dataprep logic:

https://experienceleague.adobe.com/en/docs/experience-platform/sources/ui-tutorials/create/streaming/http

Navigate to the - Map Standard Fields in the documentation.

 

Documentation for all the available Data Prep Functions including upsert array:

https://experienceleague.adobe.com/en/docs/experience-platform/data-prep/functions


Regards,

Kumar Saurabh

X
xliuAuthor
Level 3
March 27, 2024

Thank you very much for your reply @kumar29917170hcyp 

 

I tried to follow the doc you shared, but I have some questions:

1. When I create mapping fields and upload a sample data file, am I supposed to upload a JSON file in the below format? And the format is according to the page here https://developer.adobe.com/experience-platform-apis/references/streaming-ingestion/#tag/Streaming-Ingestion/operation/sendMessage 

 

{ "header": { "schemaRef": { "id": "https://ns.adobe.com/companyname/schemas/123345", "contentType": "application/vnd.adobe.xed+json;version=1" }, "flowId": "12345", "datasetId": "12345" }, "body": { "xdmMeta": { "schemaRef": { "id": "https://ns.adobe.com/loyaltyone/schemas/123345", "contentType": "application/vnd.adobe.xed+json;version=1" } }, "xdmEntity": { "_companyname": { "userid": "888", "test_id": "123456", "status": [ "UPDATED_0208" ] }, "eventType": "test", "producedBy": "self", "timestamp": "2023-03-25T14:43:53.000Z", "_id": "8880-123457" } } }

 

2. When I create a calculated field, I cannot select the array field from left side bar, in this example, I cannot see the "Status" field, do I miss any steps?

 

Thank you!

 

    K
    Adobe Employee
    Kumar29917170hcypAdobe Employee
    Adobe Employee
    March 27, 2024

    Hi @xliu ,

    There are 2 points to consider in this solution:-


    1) The source data should not be already formatted as an array. You will have to modify your source to pass the Status field value as a string, and then use data prep function to Append Array.

    2) The _id column value must be unique for different events else the data will be rejected for the UPS ingestion.

     

    Hope this helps!

    Regards,
    Kumar Saurabh

    Terms & ConditionsAccessibility statement

    Loading footer...