Skip to main content
KishorPh1
Level 1
Jan 30, 2026 9:19 AM UTC
Solved

Timestamp Based Data ingestion in AEP

We are using CSV based data ingestion from ADSL 2.0 source connector. There is a field called lastUpdatedDate which is datetime field. Normal ingestion is working fine. But I want to do ingestion based on the lastUpdatedDate field. Issue I am facing is the source system is not able to send me incremental file and always sends full file. So i need to ingest only those records whose lastUpdatedDate field is newer than one we have.. This feature exists in some form in DB connectors but not present by defualt in CSV based ingestion. Possible reason for this is when we are ingesting the data from CSV and setting up rules, we dont have access to AEP dataset and hence can not figure out last updated timestamp to compare… Let me know if this is possible in some way.

Regards, Kishor

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Tof_Jossic
Tof_JossicAccepted solution
Adobe Employee
Jan 30, 2026 10:23 AM UTC

​@KishorPh1 

I believe there is no built‑in way in ADLS 2.0 CSV sources to do “row‑level incrementals” based on a lastUpdatedDate that lives in the target dataset. The connector only filters which files to ingest (by last‑modified timestamp in storage), not which rows inside a file.

Someone else might chime in here but altering the process that writes the CSV file so the system only ingests “delta only” files might be an option if feasible at your end.

View in thread

4 replies

Tof_Jossic
Adobe Employee
Tof_JossicAdobe EmployeeAccepted solution
Adobe Employee
Jan 30, 2026 10:23 AM UTC

​@KishorPh1 

I believe there is no built‑in way in ADLS 2.0 CSV sources to do “row‑level incrementals” based on a lastUpdatedDate that lives in the target dataset. The connector only filters which files to ingest (by last‑modified timestamp in storage), not which rows inside a file.

Someone else might chime in here but altering the process that writes the CSV file so the system only ingests “delta only” files might be an option if feasible at your end.

Translate
    SG14_97
    SG14_97
    Level 3
    Feb 2, 2026 7:26 AM UTC

    ​@KishorPh1

    I think the incremental logic needs to be setup in your source system (not sure if you have a datamart or warehouse CRM system) you can calculate the logic there. The other option is to make use of Query service to compute the latest version for each record and populate it towards UPS (profiling) post initial ingestion in the datalake but it will be compute heavy since ADSL 2.0 probably will not be able to check for the incremental updates

    Translate
      Sukrity_Wadhwa
      Sukrity_Wadhwa
      Level 4
      Feb 13, 2026 9:08 AM UTC

      Hi ​@KishorPh1,

      Were you able to resolve this query with the help of the provided solutions, or do you still need further assistance? Please let us know. If any of the answers were helpful in moving you closer to a resolution, even partially, we encourage you to mark the one that helped the most as the 'Best Answer'.

      Thank you!

      Translate
      Sukrity Wadhwa
      itsMeTechy
      itsMeTechy
      Level 4
      Feb 17, 2026 6:38 AM UTC

      are you ingesting profile data or event data? if profile data, you can see if it is possible option to ingest it weekly. i have seen profile data with millions of data is been ingested as full file on a weekly basis, and that should not be a problem. even after ingesting the full file weekly or daily, you can plan to delete the older batches using APIs, and that will also help in cleaning up the old files and keep the data lake not with huge amount of data. profile store will always have the latest data from the latest file as it is a full file.

      Translate
        Terms & ConditionsAccessibility statement

        Loading footer...