Deletion of Records from Individual Profile and Custom Class Datasets

[trojan_horse]

Hi all,

I have an urgent request and would appreciate a quick response.

We need to expire or delete old records from both individual profile and custom class datasets in AEP. The records should be removed from both the Profile Store and the Data Lake, and this process should be automated on a schedule - for example, deleting any records older than 30 days.

How can this be done in AEP? My understanding is that the data retention policy only works for experience event datasets.

Thanks

[itsMeTechy]

AEP/RTCDP does not natively support automatically deleting profile records based on their age (e.g., removing records older than 30 days). The platform is designed to maintain comprehensive customer profiles and enable a full 360-degree customer view over time.

While there are mechanisms to delete identities from the profile store, these are primarily intended to fulfill GDPR or other regulatory compliance requirements. These options have specific guardrails and limitations and are not suitable for general data lifecycle management.

If your use case strictly requires removing records older than 30 days, one practical approach would be to periodically ingest only the current valid profiles into a new dataset and then drop the old dataset. This ensures that only the most recent records remain available in both the data lake and profile store.

[Devyendar · Accepted solution]

@trojan_horse can you share further info on what is your objective of deleting the data every 30 days? Is it primarily to control profile volume (for license), dataset size or compliance requirement.

If it is,

Profile Volume: Instead of deleting, setting up TTL for Experience Events and Pseudonymous profile expiry

Dataset size: Instead of deleting the dataset as suggested by @itsMeTechy, deleting Batches as a schedule job from external solution like Lamda through Batches API is better as you don't need to deal with manual dataset creation every time.

Compliance: Privacy Job API can be used

[Karuppiah_Sakthivel]

@trojan_horse 

In AEP, there’s no native feature to automatically delete or expire profile or custom class records based on age — that capability only exists for Experience Event datasets through the retention policy.

If the goal is to keep your data fresh or limit the volume of profiles stored, a better long-term approach is to design your data pipeline to handle lifecycle logic before ingestion. For example, filter or exclude records older than 30 days within your source or transformation process so only current data is sent to AEP.

This avoids unnecessary reprocessing, keeps the Profile Store lean, and prevents dependency on delete operations inside AEP, which are intended mainly for compliance requests. It’s generally cleaner and more reliable to manage this logic upstream rather than within the platform itself.

[Devyendar]

@Karuppiah_Sakthivel managing source side filtering is not ideal either, 3 or 6 months down the lane the data ingestion AEP will be older then 30 days and data needs to be managed within AEP, either by deleting batches, applying TTLs to control profile volumes or deleting the entire datasets (although not suggested as it would be a breaking change on the data).