Enable error diagnostics/partial ingestion from Query Service

Question

Hi All,This is a follow up to my previous post . During the insert from query service, I realized there are some ill-formatted emails in my source data. This is causing a failure of the whole batch and nothing gets inserted. Using the source/direct file upload into a dataset, we can handle such scenario by enabling:a. Error diagnostic toggleb. Enable partial ingestion toggleHere is my error log (from query service):58000 Batch query execution gets : [failed reason ErrorCode: 58000 Batch query execution gets : [Errors(Some(INGEST-1211-400),Some(List()),Some(Parsing errors occurred when converting or validating the data. Please enable error diagnostics and ingest data again to preview or download an error log.))]]How do I enable error diagnostics or is there any mechanism to enable partial ingestion from query service? Thanks for your response!

Pradeep_Kumar_Srivastav · Accepted Answer

Hi @supratim320 , Unfortunately, you cannot directly enable partial ingestion in Adobe Experience Platform (AEP) using the Query Service.

We need to understand Partial Ingestion and Query Service - While Partial Ingestion: allows us to ingest only valid records from a batch, even if some records contain errors, Query Service: is primarily used for retrieving data from AEP, not for modifying ingestion settings, It's designed for querying and analyzing data.

You can use the Query service to filter out ill-formatted emails and can retrieve the correct/valid email records only by writing a regEx for the valid email address. The result will be to have only well formatted email/valid email address and dataset ingestion should work in that case.

Let me know if this helps.

DavidRoss91 · Answer

@supratim320

To enable error diagnostics, follow these steps:

Navigate to the Dataset: Go to the Adobe Experience Platform UI and find the dataset you're working with.
Access Dataset Settings: Look for settings or options related to the dataset, often found under "Settings" or "Properties".
Enable Error Diagnostics: Within the dataset settings, locate the option for enabling error diagnostics. This might be labeled something like "Enable Error Diagnostic Toggle" or similar.
Save Changes.

As for partial ingestion I'm not sure if it can be done via query service. But I found this:

Configure Batch Properties: When setting up your batch, look for options related to error handling or data validation. There should be a setting to enable partial batch ingestion. This might be labeled as "Enable Partial Ingestion" or something similar.

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded