Does TTL affect customer AI?

[SahuSa1]

From the AEP Architecture diagram I think Customer AI works at datalake level but not at profile store level, so I think TTL set on an events type dataset will not affect customer AI data requirements. Please let me know if my understanding is correct. 

Thanks

[Parvesh_Parmar]

Hello @SahuSa1 ,

Your understanding is mostly correct, but let me provide additional clarity:

Customer AI and Data Lake Operations:

Customer AI relies on data stored in the data lake for training its models and making predictions. If datasets in the data lake are managed with expiration policies (TTL), the availability of critical data for Customer AI can be impacted. However, if expiration dates or processing rules are configured properly, data will not be deleted prematurely.

Best Practices for Managing Data Retention for Customer AI:

1. Align Data Retention with Customer AI Requirements:

   • If you anticipate that Customer AI models require one year of historical data, ensure the data is retained in the data lake for at least one year.
   • You can achieve this by extending TTL settings or disabling TTL for datasets critical to Customer AI operations.

2. Understand the Purpose of AEP:

   • AEP is a data platform designed for customer experience management, stitching the data from different source, segmentation and activation,  not a long-term data warehouse.
   • If your use case requires long-term data storage for Customer AI or other tools, it is best to manage historical data outside of AEP in a dedicated data warehouse or data lake solution.
   • If you keep data for long term in DWH or other cloud like azure or AWS and run these models there for customer AI, you will have more performance, customization power, budget friendly as compare to do in AEP.

Hope it will help!

Best regards,

Parvesh

[SahuSa1]

Hi @Parvesh_Parmar , Thank you so much for your quick response.

Please correct me if I am wrong, I had the understanding that TTL only impacts the profile store and it does not affect the data stored in the datalake. Only I guess, the pseudonymous profile data would get expired from both profile store and datalake if it has no activity within the TTL set period, due to its "all or nothing" concept.

(Source - https://www.youtube.com/watch?v=We3RN-41Dik)

So,TTL set on dataset should not affect the data requirements for customer AI right? 

Also, can TTL be applied so that data also gets expired from datalake along with profile store? 

Please enlighten me.

Thanks,

Sambit Sahu

[Parvesh_Parmar · Accepted solution]

Hello @SahuSa1 , 

I have just cross-checked Adobe's documentation on Experience Event Expirations (Experience Event Expirations | Adobe Experience Platform).

According to the documentation, Experience Event expirations will delete data from the Profile Store, but not from the Data Lake.

Regarding Pseudonymous Profiles, while it is not explicitly mentioned, I believe it follows the same functionality. Here’s the relevant documentation for your reference: Pseudonymous Profile data expiration | Adobe Experience Platform.

If your customer is directly using Data Lake data for training data models or predictions, or if any out-of-the-box Adobe data model is running on the Data Lake, there should be no impact. However, if you are planning to build something that relies on event data and pseudonymous data and you have expiration rules set, I recommend cross-checking this with Adobe for confirmation. That way, if any issues arise, they can provide the necessary support.

Kind regards,
Parvesh

[SahuSa1]

Thanks @Parvesh_Parmar , I had asked the same to Adobe.. this is what I got as reply - "Customer AI analyzes the datasets, the event TTL is applied on the profile store"

Will try to check with them how Pseudonymous data expiration works at data lake level.

Thanks,

Sambit Sahu