Exportation of Historical Data via Destinations

[TylerKrause]

Hey all! Running into an issue I'm wondering if anyone else has faced before. We're working on a new integration that we need to export 2 years of historical transaction data to support. The first year was super easy, we have it in a profile enabled dataset, so we were able to export that to our S3 Integration directly. The second year, has been removed off of profile via TTL but still lives within the datalake. I wrote a query to pull the data and added it to a new historical dataset, but when attempting to export to our S3 Connection, no records are being successfully activated. I noticed in the documentation, that there is a guard rail in place to only allow data with a timestamp of less than 365 days to be activated. 

Has anyone else run into this issue before, and how to you get around it?

[vinodkomiri · Accepted solution]

Hi @TylerKrause ,

I believe you are using "Dataset Export to cloud storage" option? The limitation for this is it cannot export data older than 365 days. I would recommend you to trick the AEP platform by creating a new dataset without enabling it for profile. This will acting as temp dataset for exporting your data. Now you have to write a query to insert all the second year data from your old dataset to this dataset by modifying the Timestamp as Timestamp + 1 year. This will revise all the timestamp to have dates within 365 days and the export dataset will allow you to export. 

The only additional thing you need to do is while importing at your system you need to revert your timestamp to Timestamp - 1year. Which return your original timestamp of the record. 

Sometimes if the system does not allow you to perform any actions, it is good to trick the system with our actions and get the work done. 

Hope this helps you to extract the data. 

Regards,

Vinod

[TylerKrause]

This is almost exactly what I ended up doing. I just set the timestamp date as a dateTime field for right now and added my 'correct' timestamp field to another empty field I had on the dataset and just mapped that to timestamp for my external integration. 

Glad we had the same thought Vinod!  

Thanks for the reply!

Tyler