Issue with Record Deletion – Not Cleaning Up from Datalake

[user48]

Hi Community,

I’m encountering an issue while trying to delete records in our sandbox and would appreciate any insights or help.

Here’s the situation:

• I created a delete record request via the UI for two records in our sandbox. The request was completed successfully, but the records were only cleaned up from UPS and not from the datalake.
• Additionally, according to the documentation, I am unable to find the option to select whether to delete records from a specific dataset or from all datasets.

Has anyone faced a similar issue or knows how to resolve this? Any guidance would be greatly appreciated.

I’ve attached screenshots for reference:

1. Screenshot of the successful completion of the delete request
2. Screenshot from UI showing the absence of the dataset selection functionality as per document.
3. Screenshot of the dataset selection option from the AEP document

Thanks in advance!

[Harveer_SinghGi1]

Hi @user48 ,

The option to choose single or multiple datasets is available to organizations who have been moved to the Delta format. If you are not seeing this option then most probably your organization have not undergone delta migration yet. I suspect that record deletion from only UPS and not data lake might also be linked to the same reason but not 100% sure. As record deletion is a beta feature we don't have a lot of documentation around it's workings. I'd suggest you reach out to support team for more clarity on first part of your query.

Cheers!

[DavidRoss91]

To tag onto @Harveer_SinghGi1 's answer--

The doc states:

To improve efficiency and make dataset operations less expensive, organizations who have been moved to the Delta format can delete data from the Identity Service, Real-Time Customer Profile, and the data lake. This type of user is referred to as delta-migrated. Users from organizations who have been delta-migrated can choose to delete records from either a single or all datasets. Users from organizations that have not undergone delta migration are unable to selectively delete records from either a single dataset or all datasets

So, it would appear that this is the reason for your issues you are having.

[user48]

Thank you @DavidRoss91 and @Harveer_SinghGi1 ,

Could you please help clarify what "Delta Migrated" refers to? Additionally, how do I find out if our organization has been migrated to the Delta format?

If we have not been migrated, could you please explain the process for migrating to Delta?

[DavidRoss91]

Hi @user48 

I pulled this basic definition from MS documentation:

"Delta Format is an open-source storage layer that brings reliability to data lakes"

I would recommend reaching out to Adobe Support through a ticket, to gather more information direct to your organization for further details on the delta migration.

[Harveer_SinghGi1]

Hi @user48 ,

To know more about Delta Lake you can read these articles,

https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform-blogs/massive-data-proces...

https://docs.delta.io/latest/delta-intro.html

And as you'll see the defination is similar to what @DavidRoss91 has shared, documentation states,

“Delta Lake is an open source project that enables building a Lakehouse architecture on top of data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes, such as S3, ADLS, GCS, and HDFS.”

On the migration status I agree with @DavidRoss91 and it seems you'd have to contact support to check that because as per my understanding Adobe is moving it's AEP instances to delta and only Adobe will have visibility of when the instances belonging to your organization are planned for migration.

Cheers!

[kautuk_sahni]

@user48 Did you find the suggestions helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!

[user48 · Accepted solution]

Sure @kautuk_sahni.

Thank you, @Harveer_SinghGi1  and @DavidRoss91 , for your help. I reached out to Adobe, and they clarified that for records to be deleted from the Data Lake, the organization needs to be 'Delta-migrated.' However, even for Delta-migrated organizations, hygiene requests are not processed against the Data Lake, as this capability is still in 'limited release' and not yet generally available.

Currently, to enable a full-service level for the Data Lake, we need to raise a ticket and provide the necessary confirmation.