Batch Ingestion - format & pattern check

[Michael_Soprano]

XXX

There is a table like that. 

Assuming that I have got a field in batch which do not meet the requirements of Schema field (pattern, format). 

How would be a behaviour of such a batch:

a) batch would land in Data Lake 

b) batch would be aborted ? 

[Asheesh_Pandey · Accepted solution]

@Michael_Soprano In my understanding, If a field in your batch data doesn't meet the schema requirements (like wrong format, pattern, or data type), the entire batch will fail. (Note: It is true only if the issue is with required fields OR identity fields that will block the batch, partial ingestion can still upload data upto certain percentage based on the batch ingestion settings). You’ll see the failure listed in the Monitoring UI with specific error details. Best to validate your data ahead of time using Data Prep or the Schema Registry API to avoid surprises!

[KumarRishii]

If a batch contains fields that do not meet schema requirements (like pattern or format), the behavior depends on validation rules:

Strict Validation: The batch is aborted, and no data lands in the Data Lake.
Soft Validation: The batch may still land in the Data Lake, but problematic fields could trigger warnings, be ignored, or sent to an error bucket.

Check your system's ingestion configuration to confirm the behavior.

[Michael_Soprano]

Where you set up this validation - strict/soft?

[kautuk_sahni]

@KumarRishii Just checking in — were you able to resolve your issue?
We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!