Joined the community
30-03-2020
7:03:08 AM
Offline
Re: Data Warehouse - requests gzip compression
piotrspring
Level 2
22 likes
5 posts
0 solutions
Top badges earned
View profile
piotrspring
- Adobe Analytics
4.3K
0
0
Likes
Total Posts
Correct Reply
500+ MB for Data Warehouse, for all reporting suites it's about 2-3 GB
hourly
Views
Likes
Replies
Betreff: Data Warehouse - requests gzip compression
piotrspring
Level 2
22 likes
5 posts
0 solutions
Top badges earned
View profile
piotrspring
- Adobe Analytics
4.3K
0
0
Likes
Total Posts
Correct Reply
submited as an idea
https://experienceleaguecommunities.adobe.com/t5/adobe-analytics-ideas/datawarehouse-gzip-compression/idi-p/357547
Views
Likes
Replies
datawarehouse gzip compression
piotrspring
Level 2
22 likes
5 posts
0 solutions
Top badges earned
View profile
piotrspring
- Adobe Analytics
2.8K
13
2
Likes
Total Posts
Correct Reply
Hi, we are using data warehouse exports.The Files that we are receiving
a quite big -> in avg. 1GB on hourly basis. Our db don't support ZIP
format (the same like Athena, Redshift, Snowflake etc.), so we need to
convert the files first in GZIP. Since unzip large files consumes a lot
of time & ressources, we would like to submit an idea:- add the gzip
compression format to the exports- split large files automatically (like
in Data Feeds) into smaller chunks of gzip files. It's actually the same
w...
Views
Likes
Replies
Re: Data Warehouse - requests gzip compression
piotrspring
Level 2
22 likes
5 posts
0 solutions
Top badges earned
View profile
piotrspring
- Adobe Analytics
4.3K
0
0
Likes
Total Posts
Correct Reply
Hi Adrey, the average size about 500MB, but 30% of the exports are
beyond 1GB -> up to 1.5GB in a single file for each of our reporting
suites.The exports are scheduled on hourly basis. Currently we divide
those with segments into smaller ones, but still it's a lot of
overhead:- to manage those exports (like add/remove columns)- unzip and
gzip Ok, I'll submit this as an idea. Thx!
Views
Likes
Replies
Data Warehouse - requests gzip compression
piotrspring
Level 2
22 likes
5 posts
0 solutions
Top badges earned
View profile
piotrspring
- Adobe Analytics
6.1K
9
5
Likes
Total Posts
Correct Reply
Hi, we are using data warehouse requests and everything works fine.
Unfortunately we put a lot of ressources to unzip the exports 😞 Is
there plan to change the compression type or add gzip compression in the
next future? Like in Data Feeds, where bigger files are automatically
splitted into smaller ones? ZIP it's not a supported format by
Snowflake, Athena, etc .... Instead of unzip -> gzip -> load, we would
like to load the exports directly. Regards,Piotr
Views
Likes
Replies
