If there is an error for one of the attributes in an event, is the entire event dropped by AEP or only that specific attribute? | Community
Skip to main content
S
SameerCh1
Level 2
July 2, 2024
Solved

If there is an error for one of the attributes in an event, is the entire event dropped by AEP or only that specific attribute?

  • July 2, 2024
  • 1 reply
  • 1745 views

I have created a dataflow for ingestion of data into a dataset based on a schema. One of the attributes in the schema is of datatype String.

While for most events, this attribute is coming as String from upstream, for certain other events, it is coming as a JSON object. I see that there are warnings in the dataflow run regarding this datatype mismatch. I want to know that in these cases, whether AEP drops the entire event because of this datatype mismatch error or drops only that specific attribute and rest of the data in the event is ingested? What is the default behavior in this case?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by SameerCh1

Thanks for confirming this. So both ingestion (streaming and batch) behaves in the same way? Also the behavior is same in upsert operations?

 

I have not checked this thoroughly yet. I'll try to confirm on this.


Yes both works the same. Unless a record is marked as Records with failures, it does not get dropped.

1 reply

S
ShashiCh
Level 3
July 3, 2024

1. When you a create dataset, you have an option to enable partial ingestion. But partial ingestion is for, Let's say you are ingesting 10 records, if one record is not matching the syntax it will insert 9 other records but drops one.

 

2. In your case, it is one record and one attribute in that record has a data type mismatch. So obviously it will not insert the rest of the attributes and drops that record (entire event data)

    S
    SameerCh1Author
    Level 2
    July 3, 2024

    Thanks @shashich for the reply. Are you sure it will drop the entire record? Is there any Adobe documentation that can confirm this? I am asking because I have seen conflicting behaviors when I see the reporting in datasets and dataflows. Datasets shows that particular records have failed but dataflows show that they were records with warnings.

      arpan-garg
      Community Advisor
      arpan-gargCommunity Advisor
      Community Advisor
      July 5, 2024

      Hi @sameerch1  - What @shashich mentioned is correct , if a certain attribute in a event does not comply to the schema requirements the whole event will be dropped. I am not sure if we have a Adobe document for this but what you can do to test it is to add a streaming api which send data to this dataset and make a call to this streaming api by appending ?syncValidation=true in the url. It will tell you directly what the error is plus you can verify that the record will not be inserted in the datalake.

       

      Thanks,

      Arpan

      Terms & ConditionsAccessibility statement

      Loading footer...