Expand my Community achievements bar.

Help shape the future of AI assistance by participating in this quick card sorting activity. Your input will help create a more effective system that better serves your needs and those of your colleagues.

Incremental Load Issue with BigQuery Connection in Adobe Experience Platform Sources

Avatar

Level 1

I'm experiencing an issue with the incremental load feature in Adobe Experience Platform (AEP) Sources when connecting to a BigQuery table.

Details:

  • The BigQuery table contains a date field called "data_date" of type TIMESTAMP.
  • I've configured the delta field (i.e., "data_date") for incremental load in AEP.
  • Despite this setup, the incremental load does not function as expected.

I have verified the delta field values and ensured that the TIMESTAMP field is appropriately populated in the BigQuery table. However, the data ingestion seems to process all records the first time the dataflow runs but never process the incremental data for the new rows in the table for the subsequent daily runs.

 

Any idea what can be going on?

 

Thanks,

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

5 Replies

Avatar

Community Advisor

Hello @raphaell3678617 , 

 

Can you check your date time field form BigQuery table is mapped correctly with the datetime filed in AEP during the mapping? 

 

 

Parvesh_Parmar_1-1731535347209.png

 

 

Parvesh_Parmar_2-1731535372581.png

 

 

 

Kr,

Parvesh

 

Avatar

Level 1

Hi Parvesh,

 

Thank you for replying, and I'm sorry I didn't reply sooner.

 

The example you gave above appears to use an XDM Individual Profile. However, I'm trying to use an XDM ExperienceEvent. If Adobe mandates the inclusion of specific fields like _repo.createDate or any other required fields for incremental processing, I believe that might explain the issue. However, if these fields are not a deciding factor, I can confirm that the BigQuery table is correctly mapped, including the datetime field in AEP.

 

Please let me know if additional prerequisites or specific fields need to be included when working with an XDM ExperienceEvent. And thank you very much for taking the time to reply!

 

Cheers,

Raphael

Avatar

Administrator

@raphaell3678617 Did you find the suggestion helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!



Kautuk Sahni

Avatar

Level 1

Hi Kautuk Sahni,

 

The suggestion did not answer my question. I provided more detail to see if that helps.

 

Regards,

Raphael

Avatar

Employee

Hello @raphaell3678617 

 

What is the format of your timestamp field in GBQ.  Please ensure the timestamp field is SO date format

 

 YYYY-MM-DDTHH:mm:ssZ