Exportation of Historical Data via Destinations

Question

Hey all! Running into an issue I'm wondering if anyone else has faced before. We're working on a new integration that we need to export 2 years of historical transaction data to support. The first year was super easy, we have it in a profile enabled dataset, so we were able to export that to our S3 Integration directly. The second year, has been removed off of profile via TTL but still lives within the datalake. I wrote a query to pull the data and added it to a new historical dataset, but when attempting to export to our S3 Connection, no records are being successfully activated. I noticed in the documentation, that there is a guard rail in place to only allow data with a timestamp of less than 365 days to be activated.

Has anyone else run into this issue before, and how to you get around it?

vinodkomiri · Accepted Answer

Hi @tylerkrause ,

I believe you are using "Dataset Export to cloud storage" option? The limitation for this is it cannot export data older than 365 days. I would recommend you to trick the AEP platform by creating a new dataset without enabling it for profile. This will acting as temp dataset for exporting your data. Now you have to write a query to insert all the second year data from your old dataset to this dataset by modifying the Timestamp as Timestamp + 1 year. This will revise all the timestamp to have dates within 365 days and the export dataset will allow you to export.

The only additional thing you need to do is while importing at your system you need to revert your timestamp to Timestamp - 1year. Which return your original timestamp of the record.

Sometimes if the system does not allow you to perform any actions, it is good to trick the system with our actions and get the work done.

Hope this helps you to extract the data.

Regards,

Vinod

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded