I have created a dataflow for ingestion of data into a dataset based on a schema. One of the attributes in the schema is of datatype String.
While for most events, this attribute is coming as String from upstream, for certain other events, it is coming as a JSON object. I see that there are warnings in the dataflow run regarding this datatype mismatch. I want to know that in these cases, whether AEP drops the entire event because of this datatype mismatch error or drops only that specific attribute and rest of the data in the event is ingested? What is the default behavior in this case?
1. When you a create dataset, you have an option to enable partial ingestion. But partial ingestion is for, Let's say you are ingesting 10 records, if one record is not matching the syntax it will insert 9 other records but drops one.
2. In your case, it is one record and one attribute in that record has a data type mismatch. So obviously it will not insert the rest of the attributes and drops that record (entire event data)
Thanks @ShashiCh for the reply. Are you sure it will drop the entire record? Is there any Adobe documentation that can confirm this? I am asking because I have seen conflicting behaviors when I see the reporting in datasets and dataflows. Datasets shows that particular records have failed but dataflows show that they were records with warnings.
Hi @SameerCh1 - What @ShashiCh mentioned is correct , if a certain attribute in a event does not comply to the schema requirements the whole event will be dropped. I am not sure if we have a Adobe document for this but what you can do to test it is to add a streaming api which send data to this dataset and make a call to this streaming api by appending ?syncValidation=true in the url. It will tell you directly what the error is plus you can verify that the record will not be inserted in the datalake.
Hi @arpan-garg thanks for the reply. When I tested this myself, I found that my events (where I sent an Object instead of String for a control attribute) were getting ingested into AEP and I was able to query them from the dataset. All the other attributes from the event had values and only the control attribute had NULL. If I can see the record in the dataset, that means AEP is accepting the record but only ignoring that attribute, right?
@ShashiCh@arpan-garg I heard back from Adobe support. They have confirmed that if the attribute has a datatype mismatch then the attribute is nullified but the rest of the record is ingested. It will be marked under Records with warnings in the dataflow.
