I'm experiencing an issue with the incremental load feature in Adobe Experience Platform (AEP) Sources when connecting to a BigQuery table.Details:The BigQuery table contains a date field called "data_date" of type TIMESTAMP.I've configured the delta field (i.e., "data_date") for incremental load in AEP.Despite this setup, the incremental load does not function as expected.I have verified the delta field values and ensured that the TIMESTAMP field is appropriately populated in the BigQuery table. However, the data ingestion seems to process all records the first time the dataflow runs but never process the incremental data for the new rows in the table for the subsequent daily runs. Any idea what can be going on? Thanks,

Question

Incremental Load Issue with BigQuery Connection in Adobe Experience Platform Sources

Forum|Forum|1 year ago
November 13, 2024
2 replies
1049 views

I'm experiencing an issue with the incremental load feature in Adobe Experience Platform (AEP) Sources when connecting to a BigQuery table.

Details:

The BigQuery table contains a date field called "data_date" of type TIMESTAMP.
I've configured the delta field (i.e., "data_date") for incremental load in AEP.
Despite this setup, the incremental load does not function as expected.

I have verified the delta field values and ensured that the TIMESTAMP field is appropriately populated in the BigQuery table. However, the data ingestion seems to process all records the first time the dataflow runs but never process the incremental data for the new rows in the table for the subsequent daily runs.

Any idea what can be going on?

Thanks,

Experience Platform

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Parvesh_Parmar

Community Advisor

Hello @raphaell3678617 ,

Can you check your date time field form BigQuery table is mapped correctly with the datetime filed in AEP during the mapping?

Kr,

Parvesh

Parvesh Parmar – Adobe Community Advisor https://www.linkedin.com/in/parvesh-parmar/

R

raphaell3678617Author

Hi Parvesh,

Thank you for replying, and I'm sorry I didn't reply sooner.

The example you gave above appears to use an XDM Individual Profile. However, I'm trying to use an XDM ExperienceEvent. If Adobe mandates the inclusion of specific fields like _repo.createDate or any other required fields for incremental processing, I believe that might explain the issue. However, if these fields are not a deciding factor, I can confirm that the BigQuery table is correctly mapped, including the datetime field in AEP.

Please let me know if additional prerequisites or specific fields need to be included when working with an XDM ExperienceEvent. And thank you very much for taking the time to reply!

Cheers,

Raphael

kautuk_sahni

Community Manager

@raphaell3678617 Did you find the suggestion helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!

Kautuk Sahni