Issue with Streaming API

[Antony6790]

Hi All,

Calling Streaming API and trying to ingest data into the dataset but getting below error. 

"The transformed message is empty. Please check the mapping set cc70ab1fe7e044da9d5883be25cb05d1 and version 0 and XDM schema _xyz.schemas.b1a161669dd4f9f9392ccf1c182868f6270a5e9eccc5cd95 and schema version 1 and try again. Message details:"

Using the schemaregistry API, I got the SchemaID and using it in the API call. Also in the payload I'm sending all the attributes in the same order how I added in the dataflow mapping.

From the error I think streaming Ingestion API is trying to use old Schema version even though Im using schemaID of latest version 1.1 or somewhere schema version is not matching.

Any suggestions how to resolve this error?

Thanks

[DavidRoss91]

Hi @Antony6790 

Could you send a copy of the payload you are sending? You can remove any sensitive info. 

[Antony6790]

Hi David,

Below is the POSTMAN payload.

{
"messages": [
{
"header": {
"schemaRef": {
"id": "https://ns.adobe.com/xyz/schemas/<schemaID>",
"contentType": "application/vnd.adobe.xed-full+json;version=1.0"
},
"imsOrgId": "xyz",
"datasetId": "xyz",
"source": {
"name": "xyz"
}
},
"body": {
"xdmMeta": {
"schemaRef": {
"id": "https://ns.adobe.com/xyz/schemas/<schemaID>",
"contentType": "application/vnd.adobe.xed-full+json;version=1.0"
}
},
"xdmEntity": {
"state": "ABC",
"address": "Testing4",
"owner_id": 6,
"id": "A6794",
"address2": "Schema new",
"zip": "T1J 3Z6",
"name": "Testing4",
"city": "Testing4",
"pfs_id": 1

}
}
}
]
}

[brekrut]

Hello @Antony6790, 

Your version number in the schema should always match 1 when sending in a streaming /POST to a data collection.

You can use the following as reference for the body.

{
    "header": {
        "schemaRef": {
            "id": "<schemaID>>",
            "contentType": "application/vnd.adobe.xed-full+json;version=1.0"
        },
        "imsOrgId": "<<IMSORG>>",
        "datasetId": "<<dataSetID>>",
        "source": {
            "name": "<<dataflowName>> "
        }
    },
    "body": {
        "xdmMeta": {
            "schemaRef": {
                "id": "<schemaID>>",
                "contentType": "application/vnd.adobe.xed-full+json;version=1.0"
            }
        },
        "xdmEntity": {
            "_<<tenant>>": {
         
            }
        }
    }
}

[Antony6790]

Hi Brekut,

Yes, I'm using 1.0 as version. However in the schemaregistry I see the version for this schema as 1.1 and tried this version also, but still getting same error.

Thanks.

[DavidRoss91]

@Antony6790 might not matter but I see "messages" at the top of your payload. Can you try starting with just the following below?

{
"header": {

[Antony6790]

Without Messages I'm getting error. As per our AD documentation it's needed for successful API calls.

Unfortunately our AD team rolled off and not working with us anymore. So I'm reverse engineering and trying to make this work in our DEV sandbox.

Thanks.

[Rite18]

@Antony6790 it seems to be an error due to transformation and mapping during the ingestion process, you have payload but during the process, throwing error may be one of the below 

- transformation logic if any 

-  schema mapping 

-  data field type/ required field