Incremental Load Issue with BigQuery Connection in Adobe Experience Platform Sources

[raphaell3678617]

I'm experiencing an issue with the incremental load feature in Adobe Experience Platform (AEP) Sources when connecting to a BigQuery table.

Details:

• The BigQuery table contains a date field called "data_date" of type TIMESTAMP.
• I've configured the delta field (i.e., "data_date") for incremental load in AEP.
• Despite this setup, the incremental load does not function as expected.

I have verified the delta field values and ensured that the TIMESTAMP field is appropriately populated in the BigQuery table. However, the data ingestion seems to process all records the first time the dataflow runs but never process the incremental data for the new rows in the table for the subsequent daily runs.

Any idea what can be going on?

Thanks,

[Parvesh_Parmar]

Hello @raphaell3678617 , 

Can you check your date time field form BigQuery table is mapped correctly with the datetime filed in AEP during the mapping? 

Kr,

Parvesh

[raphaell3678617]

Hi Parvesh,

Thank you for replying, and I'm sorry I didn't reply sooner.

The example you gave above appears to use an XDM Individual Profile. However, I'm trying to use an XDM ExperienceEvent. If Adobe mandates the inclusion of specific fields like _repo.createDate or any other required fields for incremental processing, I believe that might explain the issue. However, if these fields are not a deciding factor, I can confirm that the BigQuery table is correctly mapped, including the datetime field in AEP.

Please let me know if additional prerequisites or specific fields need to be included when working with an XDM ExperienceEvent. And thank you very much for taking the time to reply!

Cheers,

Raphael