Incremental Load Issue with BigQuery Connection in Adobe Experience Platform Sources

[raphaell3678617]

I'm experiencing an issue with the incremental load feature in Adobe Experience Platform (AEP) Sources when connecting to a BigQuery table.

Details:

• The BigQuery table contains a date field called "data_date" of type TIMESTAMP.
• I've configured the delta field (i.e., "data_date") for incremental load in AEP.
• Despite this setup, the incremental load does not function as expected.

I have verified the delta field values and ensured that the TIMESTAMP field is appropriately populated in the BigQuery table. However, the data ingestion seems to process all records the first time the dataflow runs but never process the incremental data for the new rows in the table for the subsequent daily runs.

Any idea what can be going on?

Thanks,

[Parvesh_Parmar]

Hello @raphaell3678617 , 

Can you check your date time field form BigQuery table is mapped correctly with the datetime filed in AEP during the mapping? 

Kr,

Parvesh

[raphaell3678617]

Hi Parvesh,

Thank you for replying, and I'm sorry I didn't reply sooner.

The example you gave above appears to use an XDM Individual Profile. However, I'm trying to use an XDM ExperienceEvent. If Adobe mandates the inclusion of specific fields like _repo.createDate or any other required fields for incremental processing, I believe that might explain the issue. However, if these fields are not a deciding factor, I can confirm that the BigQuery table is correctly mapped, including the datetime field in AEP.

Please let me know if additional prerequisites or specific fields need to be included when working with an XDM ExperienceEvent. And thank you very much for taking the time to reply!

Cheers,

Raphael

[kautuk_sahni]

@raphaell3678617 Did you find the suggestion helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!

[raphaell3678617]

Hi Kautuk Sahni,

The suggestion did not answer my question. I provided more detail to see if that helps.

Regards,

Raphael

[brekrut]

Hello @raphaell3678617 

What is the format of your timestamp field in GBQ.  Please ensure the timestamp field is SO date format

 YYYY-MM-DDTHH:mm:ssZ