Are there any Dataset size limitations? and is there any impact of huge data size

[Parm2]

We are working on single schema across multiple digital properties and have single dataset created for these properties. Wanted to know if all data in single dataset can lead to any concerns?

1. Are there any Dataset size limitations per dataset vs combined all datasets together?

2. Is there any impact of huge data size in these datasets on any other components like Querying the data, sending data to data streams, AEP CDP etc. or any performance concerns?

[arpan-garg]

Hi @Parm2 - Here you can find the default guardrails for real time profile data

https://experienceleague.adobe.com/docs/experience-platform/profile/guardrails.html%3Flang%3Dde

I hope this helps. 

[arijitg · Accepted solution]

@Parm2 There are certain profile guardrails that @arpan-garg shared above and there are certain ingestion guardrails (Guardrails for Data Ingestion | Adobe Experience Platform) that you can check but no such hard guardrail on dataset size as of now.

Though we are not aware about your data or use case but best practice is to split dataset based on sources (rather than dumping all data in a single dataset). This helps in query/analysis and troubleshooting, also in case you need to clean up or reload any dataset this design comes handy.