Real-Time Customer Data Platform

abhinavbalooni · 7/29/24

Today, we are going to talk about a method of onboarding data into RTCDP which is discussed a lot and helps stream data into Adobe Experience Platform from pretty much any third party. We are going to cover the HTTP API streaming source. So let’s get started.

Source

Navigate to the Sources in the Experience Platform UI.

Above, you would see ‘Set up’ the first time you try setting it up. Since, I’ve set up an account at my end, it shows up as Add Data.

Once you’ve created the account, you move on to Adding data. The screen looks something like below:

Go ahead and click on Add data.

Select your dataset, you would like to stream data to. Review and finish.

Next step is where most of the end users get confused. How to test whether the end point is working fine for them or not.

Testing Using Postman

Next step is to get the streaming endpoint details to stream test data to. Select your dataflow from the list of dataflows for the account. Scroll to the bottom in the details that show up on the right of the screen. You should see details for using with the API endpoint. Something like the below:

Open up postman, and create a new POST request. Feed the streaming endpoint from above into the request url. The other two values, dataflow id and the dataset id will come in handy when creating the POST Body request.

In my example, I am trying to update a consent field group in my profile. You can do pretty much anything you want to with the streaming endpoint.

My payload looks something like the below (yours would look different depending on your schema)

{
  "header": {
    "schemaRef": {
      "id": "https://ns.adobe.com/spnam/schemas/<put in your schema id here> ",
      "contentType": "application/vnd.adobe.xed-full+json;version=1"
    },
    "flowId": "<feed the dataflow id from the dataflow details here",
    "datasetId": "<feed the dataset id from the dataflow details here>"
  },
  "body": {
    "xdmMeta": {
      "schemaRef": {
        "id": "https://ns.adobe.com/spnam/schemas/<put in your schema id here> ",
        "contentType": "application/vnd.adobe.xed-full+json;version=1"
      }

    },
    "xdmEntity": {

      
  "_id": "1112121",
  "_repo": {
    "createDate": "2024-07-28T12:00:00-06:00",
    "modifyDate": "2024-07-28T12:00:00-06:00"
  },
  "consents": {
    "collect": {
      "val": "y"
    },
    "idSpecific": {
      "ECID": {
        "<ECID value removed from here>": {
          "adID": {
            "idType": "IDFA",
            "val": "y"
          },
          "collect": {
            "val": "y"
          },
          "marketing": {
            "email": {
              "reason": "Sample value",
              "time": "2018-11-12T20:20:39+00:00",
              "val": "y"
            },
            "push": {
              "reason": "Sample value",
              "time": "2024-05-15T20:20:39+00:00",
              "val": "n"
            },
            "sms": {
              "reason": "Sample value",
              "time": "2018-11-12T20:20:39+00:00",
              "val": "y"
            }
          },
          "personalize": {
            "content": {
              "val": "y"
            }
          },
          "share": {
            "val": "y"
          }
        }
      }
    },
    "marketing": {
      "any": {
        "reason": "Sample value",
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "call": {
        "reason": "Sample value",
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "commercialEmail": {
        "reason": "Sample value",
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "email": {
        "reason": "Sample value",
        "subscriptions": {
          "key": {
            "subscribers": {
              "key": {
                "source": "Sample value",
                "time": "2018-11-12T20:20:39+00:00"
              }
            },
            "topics": [
              "Sample value"
            ],
            "type": "Sample value",
            "val": "y"
          }
        },
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "fax": {
        "reason": "Sample value",
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "postalMail": {
        "reason": "Sample value",
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "preferred": "email",
      "push": {
        "reason": "Sample value",
        "subscriptions": {
          "key": {
            "subscribers": {
              "key": {
                "source": "Sample value",
                "time": "2018-11-12T20:20:39+00:00"
              }
            },
            "topics": [
              "Sample value"
            ],
            "type": "Sample value",
            "val": "y"
          }
        },
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      },
      "sms": {
        "reason": "Sample value",
        "subscriptions": {
          "key": {
            "subscribers": {
              "key": {
                "source": "Sample value",
                "time": "2018-11-12T20:20:39+00:00"
              }
            },
            "topics": [
              "Sample value"
            ],
            "type": "Sample value",
            "val": "y"
          }
        },
        "time": "2018-11-12T20:20:39+00:00",
        "val": "y"
      }
    },
    "metadata": {
      "time": "2018-11-12T20:20:39+00:00"
    },
    "personalize": {
      "content": {
        "val": "y"
      }
    },
    "share": {
      "val": "y"
    }
  },
  "createdByBatchID": "/uri-reference",
  "identityMap": {
    "ECID": [
      {
        "authenticatedState": "ambiguous",
        "id": "<ECID value removed from here>",
        "primary": true
      }
    ]
  }

    }
  }
}

Once you have the POST body request ready. Your request should look something like the below:

Now go ahead and press send. If your request is correctly structured, you will see a 200 OK response like below.

Once done, it should show up in the dataflow runs in a few minutes. You can check the profile and should see the latest data reflecting near real time. Have a look at the highlighted attributes which show the latest hit reflecting in my case.

You can also verify that the ingestion was successful by looking at the dataflow run and should also show up in the dataset as a batch ingestion. As you can see below, the dataflow is processing the hit that was sent through.