Batch Ingestion - format & pattern check | Community
Skip to main content
Michael_Soprano
Michael_Soprano
Level 10
June 29, 2025
Solved

Batch Ingestion - format & pattern check

  • June 29, 2025
  • 3 replies
  • 420 views

XXX

 

There is a table like that. 

Assuming that I have got a field in batch which do not meet the requirements of Schema field (pattern, format). 

How would be a behaviour of such a batch:

a) batch would land in Data Lake 

b) batch would be aborted ? 

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by Asheesh_Pandey

@michael_soprano In my understanding, If a field in your batch data doesn't meet the schema requirements (like wrong format, pattern, or data type), the entire batch will fail. (Note: It is true only if the issue is with required fields OR identity fields that will block the batch, partial ingestion can still upload data upto certain percentage based on the batch ingestion settings). You’ll see the failure listed in the Monitoring UI with specific error details. Best to validate your data ahead of time using Data Prep or the Schema Registry API to avoid surprises!

3 replies

Asheesh_Pandey
Community Advisor
Asheesh_PandeyCommunity AdvisorAccepted solution
Community Advisor
June 29, 2025

@michael_soprano In my understanding, If a field in your batch data doesn't meet the schema requirements (like wrong format, pattern, or data type), the entire batch will fail. (Note: It is true only if the issue is with required fields OR identity fields that will block the batch, partial ingestion can still upload data upto certain percentage based on the batch ingestion settings). You’ll see the failure listed in the Monitoring UI with specific error details. Best to validate your data ahead of time using Data Prep or the Schema Registry API to avoid surprises!

- Asheesh
    KumarRishii
    KumarRishii
    Level 5
    June 30, 2025

    If a batch contains fields that do not meet schema requirements (like pattern or format), the behavior depends on validation rules:

    Strict Validation: The batch is aborted, and no data lands in the Data Lake.
    Soft Validation: The batch may still land in the Data Lake, but problematic fields could trigger warnings, be ignored, or sent to an error bucket.

    Check your system's ingestion configuration to confirm the behavior.

      Michael_Soprano
      Michael_SopranoAuthor
      Level 10
      July 1, 2025

      Where you set up this validation - strict/soft?

      kautuk_sahni
      Community Manager
      kautuk_sahniCommunity Manager
      Community Manager
      July 2, 2025

      @kumarrishii Just checking in — were you able to resolve your issue?
      We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!

      Kautuk Sahni
      Terms & ConditionsAccessibility statement

      Loading footer...