Record deletion not reflecting in Data Lake – Expected behavior or issue?

How did you delete? it depends on the way how you deleted the data.

Hey ​@Solix Cloud AI 

I assume you’re trying the delete records as a batch from a dataset. If that’s the case, then that could be an UI Issue/Batch deletion might be queued.

My recommendation would be to verify whether that batch to be deleted has actually been removed from the data lake with the help of following steps.

1. Grab the batch ID either from UI or API that you want to delete. Refer this documentation for grabbing batch ids via API. 
2. Delete that batch using this API. This method doesn’t physically clean data immediately but mark that batch as “inactive” and eventually the garbage will be collected asynchronously with status “deleted”. The status can be confirmed from the above API. [Note: Deletion of batches only removes records from data lake but not from profile store]

   

    

3. Verify the deletion by reviewing the metadata of that dataset via this API . This returns the necessary meta objects, which includes both lake and profile storage in bytes.

So, to answer your questions

• It appears that the deletion of a batch is handled asynchronously (UI/API) to safely remove the data from AEP (regardless of data retention policy)
• Please follow the above methods to verify
• I don’t think there is any way to configure/control that behavior other than setting retention policies

Hope this helps!

Cheers,
Ganesh Kumar