Accessing data , Data retention and data cleanup | Community
Skip to main content
arpan-garg
Community Advisor
arpan-gargCommunity Advisor
Community Advisor
May 8, 2023
Solved

Accessing data , Data retention and data cleanup

  • May 8, 2023
  • 2 replies
  • 4624 views

Hello,

 

I have a few questions regarding accessing data through the Profile API, data retention, and data cleanup.

1. I have recently ingested data into AEP using the streaming API and I am currently accessing it through the Real-Time Profile API. I understand that the data is stored in the profile store for 7 days, but I am unsure if the Profile API will still provide a response after this period. Could you please clarify this for me?

2. Additionally, I am curious about the length of time data remains in the data-lake before Adobe archives it. Should I be concerned about backing up the data I have ingested into AEP, or does it remain there indefinitely? This document says that data stays for 7 days https://experienceleague.adobe.com/docs/experience-cloud-kcs_en/kbarticles/KA-19958.html?lang=en

3.Finally, I would like to know how to delete event data in the data lake that is no longer relevant. Can you provide some guidance on this?

 

Thank you for your time and assistance,

Arpan

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
    Best answer by arijitg

    Hi @arpan-garg 

     

    1. If you didn't enable any TTL window of 7 days you should still be able to get the response, though new sandboxes comes with a default TTL period of 30 Days so after that period data will be removed.

    2. Data Lake is meant as a temporary data zone to get the data to Profile and is meant to contain data only for 7 days with the sole purpose of preparing the data to be ingested into Profile. However, there are no enforced guardrails around this limit. Data is neither deleted nor made inaccessible after 7 days.

    3. You can delete data from data-lake on batch level (not possible on record level) you can list out old batches which are not required anymore and use batch deletion API. Below is a reference guide -->

     

    Batch Ingestion API Guide | Adobe Experience Platform

    2 replies

    arijitg
    Adobe Employee
    arijitgAdobe EmployeeAccepted solution
    Adobe Employee
    May 8, 2023

    Hi @arpan-garg 

     

    1. If you didn't enable any TTL window of 7 days you should still be able to get the response, though new sandboxes comes with a default TTL period of 30 Days so after that period data will be removed.

    2. Data Lake is meant as a temporary data zone to get the data to Profile and is meant to contain data only for 7 days with the sole purpose of preparing the data to be ingested into Profile. However, there are no enforced guardrails around this limit. Data is neither deleted nor made inaccessible after 7 days.

    3. You can delete data from data-lake on batch level (not possible on record level) you can list out old batches which are not required anymore and use batch deletion API. Below is a reference guide -->

     

    Batch Ingestion API Guide | Adobe Experience Platform

      arpan-garg
      Community Advisor
      arpan-gargCommunity AdvisorAuthor
      Community Advisor
      May 8, 2023

      Hi @arijitg - 

       

       

       

      1. Where can I see and set the TTL? If the time allocated in TTL is reached does it mean I can't access the profile via profile API ? I was assuming that profile API will always give the response irrespective of how old is data.

      2. Is there a way to delete the data which is not relevant for a business usecase. For example delete all the events which has timestamp before a year.

       

       

      arijitg
      Adobe Employee
      arijitgAdobe Employee
      Adobe Employee
      May 8, 2023

      Hi @arpan-garg 

       

      1. TTL can be set by product team if you raise a request to them, you can't set/access it directly. I think default TTL window is mentioned when license is provisioned else you can raise a request to verify. TTL only removes events from profile store so if you've profile data you can still access it but you won't be able to see any event data associated, they'll be removed in rolling manner.

       

      2. For data lake "No" (TTL will remove events from profile store but not from data lake)

      arijitg
      Adobe Employee
      arijitgAdobe Employee
      Adobe Employee
      May 8, 2023

      Is there a way to delete data with scheduled queries? 


      No @arpan-garg , you can schedule queries to override profile attributes with Null and make them empty but you can't delete them. 

      Terms & ConditionsAccessibility statement

      Loading footer...