Your achievements

Level 1

0% to

Level 2

Tip /
Sign in

Sign in to Community

to gain points, level up, and earn exciting badges like the new
BedrockMission!

Learn More

View all

Sign in to view all badges

datawarehouse gzip compression

Avatar

Avatar
Shape 1
Level 2
piotrspring
Level 2

Likes

22 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Shape 1
Boost 5
Boost 3
Boost 10
Boost 1
View profile

Avatar
Shape 1
Level 2
piotrspring
Level 2

Likes

22 likes

Total Posts

5 posts

Correct Reply

0 solutions
Top badges earned
Shape 1
Boost 5
Boost 3
Boost 10
Boost 1
View profile
piotrspring
Level 2

31-03-2020

Hi,

 

we are using data warehouse exports.

The Files that we are receiving a quite big -> in avg. 1GB on hourly basis.

 

Our db don't support ZIP format (the same like Athena, Redshift, Snowflake etc.), so we need to convert the files first in GZIP.

 

Since unzip large files consumes a lot of time & ressources,  we would like to submit an idea:

- add the gzip compression format to the exports

- split large files automatically (like in Data Feeds) into smaller chunks of gzip files.

 

It's actually the same way Data Feeds currently exports the data.

 

Cheers,

Piotr  

2 Comments

Avatar

Avatar
Level 1
DavidLippert
Level 1

Likes

0 likes

Total Posts

2 posts

Correct Reply

0 solutions
View profile

Avatar
Level 1
DavidLippert
Level 1

Likes

0 likes

Total Posts

2 posts

Correct Reply

0 solutions
View profile
DavidLippert
Level 1

21-04-2020

Implementing this idea would save a lot of costs. While receiving .gzip exports we could stop post-processing jobs that do "stupid" unzip-and-gzip.

Avatar

Avatar
Shape 1
Level 1
frankde1
Level 1

Like

1 like

Total Posts

2 posts

Correct Reply

0 solutions
Top badges earned
Shape 1
Boost 1
View profile

Avatar
Shape 1
Level 1
frankde1
Level 1

Like

1 like

Total Posts

2 posts

Correct Reply

0 solutions
Top badges earned
Shape 1
Boost 1
View profile
frankde1
Level 1

31-10-2020

please implement. We are way beyond the times when files were manually extracted and put into an Excel for Analysis.