difference in datalake and profile store size

[Pradeep-Jaiswal]

i have seen majorly dataLake size is smaller than profile store, and in very few cases dataLake size is larger than profile store.

based on my understanding, one of the reason dataLake size is comparatively is smaller bcz dataLake uses compression technique to store the data, hence the size is smaller that profile store.

And if dataSet is enabled for profile, much after the data ingestion has been done already . this causes very less profile to get ingested into profile store, which causes dataLake size to be higher than profile store.

Apart from above two reason, what are the other explanation for these size variance between dataLake and profile store ?

I have attached screenshot for reference.

[Abie]

Hi @Pradeep-Jaiswal ,

One reason I can think of is - In the data lake, the data is stored in its raw format, containing no structural (schema) information, no metadata and possibly less indexes. Whereas in the Profile Store, the data is accompanied with schema details and metadata information, efficient indexing (for faster access), which makes it heavier.

Thanks!

[itsMeTechy]

data lake store data predominantly in parquet format , whereas profile store data in columnar database (could be cosmodb). also identity graph is stored in graph database. this could be the reason for higher storage size in profile store compared with data lake.