Enable error diagnostics/partial ingestion from Query Service

[supratim320]

Hi All,

This is a follow up to my previous post . During the insert from query service, I realized there are some ill-formatted emails in my source data. This is causing a failure of the whole batch and nothing gets inserted. Using the source/direct file upload into a dataset, we can handle such scenario by enabling:

a. Error diagnostic toggle

b. Enable partial ingestion toggle

Here is my error log (from query service):

58000 Batch query execution gets : [failed reason ErrorCode: 58000 Batch query execution gets : [Errors(Some(INGEST-1211-400),Some(List()),Some(Parsing errors occurred when converting or validating the data. Please enable error diagnostics and ingest data again to preview or download an error log.))]]

How do I enable error diagnostics or is there any mechanism to enable partial ingestion from query service?

Thanks for your response!

[DavidRoss91]

@supratim320 

 To enable error diagnostics, follow these steps:

1. Navigate to the Dataset: Go to the Adobe Experience Platform UI and find the dataset you're working with.
2. Access Dataset Settings: Look for settings or options related to the dataset, often found under "Settings" or "Properties".
3. Enable Error Diagnostics: Within the dataset settings, locate the option for enabling error diagnostics. This might be labeled something like "Enable Error Diagnostic Toggle" or similar.
4. Save Changes.

As for partial ingestion I'm not sure if it can be done via query service. But I found this:

Configure Batch Properties: When setting up your batch, look for options related to error handling or data validation. There should be a setting to enable partial batch ingestion. This might be labeled as "Enable Partial Ingestion" or something similar.

[Pradeep_Kumar_Srivastav · Accepted solution]

Hi @supratim320 , Unfortunately, you cannot directly enable partial ingestion in Adobe Experience Platform (AEP) using the Query Service.

We need to understand Partial Ingestion and Query Service - While Partial Ingestion: allows us to ingest only valid records from a batch, even if some records contain errors, Query Service: is primarily used for retrieving data from AEP, not for modifying ingestion settings, It's designed for querying and analyzing data.

You can use the Query service to filter out ill-formatted emails and can retrieve the correct/valid email records only by writing a regEx for the valid email address. The result will be to have only well formatted email/valid email address and dataset ingestion should work in that case.

Let me know if this helps.

[brekrut]

Hello @supratim320 

As noted above the writing of records into a dataset via query service does not allow for the partial ingestion of records.  When there is a row which does not meet the criteria of the column(s) all records for the batch(select) will not be inserted into the dataSet.