Exporting a large data set

[user77095]

My business partner wants a list of all of the customers by primary ID that received each treatment for an email so that they can analyze against other data sources to confirm client behavior. The total data set is around 380K.

When we execute using the query service we are maxed out at 50K records.

How can I export this full list to our business partners

[SatheeskannaK · Accepted solution]

@user77095 Since you have a query written take that and write to a dataset. Follow the below documentation to export the datasets and guardrails around this. There are different methods available to export the dataset choose the one that works for you.

https://experienceleague.adobe.com/en/docs/analytics-platform/using/cja-usecases/data-export/export-...

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/api/export-datasets

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/ui/activate/export-datas...

[FilipeFreitas]

Hello user77095.

I believe there are a few options here.

1. You can create a full dataset export to a destination that suits you - perhaps your data lake, or Data Lading Zone. This can be done by setting up a destination in AEP.

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/ui/activate/export-datas...

2. You can override the limit of 50k by adding LIMIT 0 in your query (let me admit I've never tried this). However, please note that there might be a 10 minutes timeout.

https://experienceleague.adobe.com/en/docs/experience-platform/query/best-practices/writing-queries

Please note that the 10 minutes timeout applies if you are running your query through Query Service. If you have Data Distiller, longer executions (up to 24 hours) are supported.

https://experienceleague.adobe.com/en/docs/experience-platform/query/troubleshooting-guide

3. If you are willing to add a considerate amount of manual work to quickly achieve your goal (depending on your needs), you can add a window function (PARTITION BY) to your query and filter out data in the WHERE clause. Something like WHERE rownumber BETWEEN 1 and 50.000; then run again with WHERE rownumber BETWEEN 50.001 and 100.000. Then run again until you export all 380k records.

I hope I was able to help you in some way.

Best Regards,

Filipe Freitas