Exporting a large data set | Community
Skip to main content
U
user77095
Level 2
June 2, 2025
Solved

Exporting a large data set

  • June 2, 2025
  • 2 replies
  • 410 views

My business partner wants a list of all of the customers by primary ID that received each treatment for an email so that they can analyze against other data sources to confirm client behavior. The total data set is around 380K.

When we execute using the query service we are maxed out at 50K records.

How can I export this full list to our business partners

Best answer by SatheeskannaK

@user77095 Since you have a query written take that and write to a dataset. Follow the below documentation to export the datasets and guardrails around this. There are different methods available to export the dataset choose the one that works for you.

https://experienceleague.adobe.com/en/docs/analytics-platform/using/cja-usecases/data-export/export-datasets

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/api/export-datasets

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/ui/activate/export-datasets#known-limitations

2 replies

SatheeskannaK
Community Advisor
SatheeskannaKCommunity AdvisorAccepted solution
Community Advisor
June 3, 2025

@user77095 Since you have a query written take that and write to a dataset. Follow the below documentation to export the datasets and guardrails around this. There are different methods available to export the dataset choose the one that works for you.

https://experienceleague.adobe.com/en/docs/analytics-platform/using/cja-usecases/data-export/export-datasets

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/api/export-datasets

https://experienceleague.adobe.com/en/docs/experience-platform/destinations/ui/activate/export-datasets#known-limitations

Thanks, Sathees
    F
    FilipeFreitas
    Level 3
    June 3, 2025

    Hello user77095.

     

    I believe there are a few options here.

     

    1. You can create a full dataset export to a destination that suits you - perhaps your data lake, or Data Lading Zone. This can be done by setting up a destination in AEP.

    https://experienceleague.adobe.com/en/docs/experience-platform/destinations/ui/activate/export-datasets

     

    2. You can override the limit of 50k by adding LIMIT 0 in your query (let me admit I've never tried this). However, please note that there might be a 10 minutes timeout.

    https://experienceleague.adobe.com/en/docs/experience-platform/query/best-practices/writing-queries

     

    Please note that the 10 minutes timeout applies if you are running your query through Query Service. If you have Data Distiller, longer executions (up to 24 hours) are supported.

    https://experienceleague.adobe.com/en/docs/experience-platform/query/troubleshooting-guide

     

    3. If you are willing to add a considerate amount of manual work to quickly achieve your goal (depending on your needs), you can add a window function (PARTITION BY) to your query and filter out data in the WHERE clause. Something like WHERE rownumber BETWEEN 1 and 50.000; then run again with WHERE rownumber BETWEEN 50.001 and 100.000. Then run again until you export all 380k records.

     

    I hope I was able to help you in some way.

     

    Best Regards,

    Filipe Freitas

    Terms & ConditionsAccessibility statement

    Loading footer...