Handling events in profiles and datalake

[ButhpurKiran]

We have events recorded in the profiles sections which are configured for 45days as a set. And, on rolling days post 45days, the events will be removed from the profile section. However, the events will be stored in datalake for 13months and can be retrieved as an when required as a whole set. Exploring from adobe documentation, there is a way to hold the specific events and can be achieved through SQL. Has anyone used this approach? Any insights or usecases will be highly appreciated. Thanks inadvance.

Kind regards,

Kiran Buthpur 

[AnkitJasani29]

Hi @ButhpurKiran ,

Specific scenarios haven’t come up for me yet, but one option is to use query service to isolate the events you want, write them into a new dataset, and then enable that dataset for Real-Time Profile ingestion.

Specifically, I found the above approach in the first paragraph of the reference link below.

https://experienceleague.adobe.com/en/docs/experience-platform/query/home

Thanks,

Ankit

[pradnya_balvir]

Hi @ButhpurKiran ,

Your idea to retain or “rehydrate” specific event data beyond the profile TTL (Time-To-Live) using SQL in the Data Lake is absolutely valid and aligns with how many advanced Adobe RTCDP users structure their data lifecycle.

Common Use Cases

1. Re-evaluation of Profile Behavior Over Longer Windows

   • E.g., "Users who viewed product A 3 months ago and now came via email"

   • Event expired from profile, but still present in Data Lake

2. Backfill Audience Segments

   • Historical re-segmentation based on new logic or campaign timing changes

3. Custom Aggregations for Summary Datasets

   • Summarize specific events per user per month and store that back into a profile-enabled dataset (summarized)

4. Data Science/Attribution Use Cases

   • Feature engineering on long-term behavioral signals not available in profile store

Best Practices

1. Create a Profile-enabled summary dataset

   • Schema: XDM ExperienceEvent or Custom Schema

   • Include: Identity, event type, and minimal attributes

2. Schedule the query in Query Service

   • Run daily or weekly to process "expired" but relevant event data

3. Tag these rehydrated events clearly

   • Use a marker field (e.g., event_source = 'rehydrated_from_datalake') to differentiate

4. Avoid overloading Profile with too much legacy data

   • Use summarized or roll-up records (e.g., “last purchased in past 90 days”)

[kautuk_sahni]

@ButhpurKiran Just checking in — were you able to resolve your issue?
We’d love to hear how things worked out. If the suggestions above helped, marking a response as correct can guide others with similar questions. And if you found another solution, feel free to share it — your insights could really benefit the community. Thanks again for being part of the conversation!