Enable Partial Ingestion for a Dataset already receiving clickstream data from the Web SDK

[JacobBa6]

I have configured a datastream/dataset/dataflow to receive clickstream data from a website the has an event driven data layer and the Adobe WebSDK.

Looking at the dataflow runs, there are some batches failing due to typeOf errors (expected string, found integer).

The developers in charge of the event driven data layer are having trouble replicating the specific typeOf error to be able to fix it in the data layer, and the data in question is part of the product array so it would be cumbersome to add code checks in the tag manager to convert the data to the proper type.

There are options to enable partial ingestion of a dataset within a typical error threshold, but i don't see any way to apply this setting to a dataset that is already receiving data from the webSDK. 

The toggle in the picture appears to just be for dragging/dropping files into the dataset.

Is there something in the UI i'm missing? I don't think it is worthwhile to make data layer changes or tag manager changes because 2 out of 80k records are causing a batch to fail, but I think that AEP should still ingest the batch if only 2 out of 80k records are failing.

[Harveer_SinghGi1]

Hi @JacobBa6 ,

The partial ingestion is a feature that can be used to successfully ingest all the correct data into Adobe Experience Platform while all the incorrect data is batched separately, along with details as to why it is invalid.

Now, when you are doing a batch ingestion like uploading a file in the UI, you are given a toggle to enable this feature so that all the correct records of that file get ingested successfully and all the erroneous records are combined in a separate sort of mini batch which is marked as failed. When there is a streaming ingestion happening through Web SDK, partial ingestion is automatically enabled, you don't need to manually configure it. What this means is that for Web SDK ingestion all the correct records will get ingested and the failed batches that you see will only have the erroneous records in them. So for your example, if 2 out of 80k records have error, those two will be added to a mini batch which gets marked as failed batch and the remaining records will get ingested successfully.

You can retrieve failed batches using the Data Access API to validate this by examining the batch file which lists all only the failed records in a batch. These batch files can also be used to check what are the other attributes for failed records like page URL that developers can use to further investigate the issue.

Cheers!