Does TTL affect customer AI?

[SahuSa1]

From the AEP Architecture diagram I think Customer AI works at datalake level but not at profile store level, so I think TTL set on an events type dataset will not affect customer AI data requirements. Please let me know if my understanding is correct. 

Thanks

[Parvesh_Parmar]

Hello @SahuSa1 ,

Your understanding is mostly correct, but let me provide additional clarity:

Customer AI and Data Lake Operations:

Customer AI relies on data stored in the data lake for training its models and making predictions. If datasets in the data lake are managed with expiration policies (TTL), the availability of critical data for Customer AI can be impacted. However, if expiration dates or processing rules are configured properly, data will not be deleted prematurely.

Best Practices for Managing Data Retention for Customer AI:

1. Align Data Retention with Customer AI Requirements:

   • If you anticipate that Customer AI models require one year of historical data, ensure the data is retained in the data lake for at least one year.
   • You can achieve this by extending TTL settings or disabling TTL for datasets critical to Customer AI operations.

2. Understand the Purpose of AEP:

   • AEP is a data platform designed for customer experience management, stitching the data from different source, segmentation and activation,  not a long-term data warehouse.
   • If your use case requires long-term data storage for Customer AI or other tools, it is best to manage historical data outside of AEP in a dedicated data warehouse or data lake solution.
   • If you keep data for long term in DWH or other cloud like azure or AWS and run these models there for customer AI, you will have more performance, customization power, budget friendly as compare to do in AEP.

Hope it will help!

Best regards,

Parvesh

[SahuSa1]

Hi @Parvesh_Parmar , Thank you so much for your quick response.

Please correct me if I am wrong, I had the understanding that TTL only impacts the profile store and it does not affect the data stored in the datalake. Only I guess, the pseudonymous profile data would get expired from both profile store and datalake if it has no activity within the TTL set period, due to its "all or nothing" concept.

(Source - https://www.youtube.com/watch?v=We3RN-41Dik)

So,TTL set on dataset should not affect the data requirements for customer AI right? 

Also, can TTL be applied so that data also gets expired from datalake along with profile store? 

Please enlighten me.

Thanks,

Sambit Sahu