datawarehouse gzip compression

Avatar

Avatar

piotrspring

Avatar

piotrspring

piotrspring

31-03-2020

Hi,

 

we are using data warehouse exports.

The Files that we are receiving a quite big -> in avg. 1GB on hourly basis.

 

Our db don't support ZIP format (the same like Athena, Redshift, Snowflake etc.), so we need to convert the files first in GZIP.

 

Since unzip large files consumes a lot of time & ressources,  we would like to submit an idea:

- add the gzip compression format to the exports

- split large files automatically (like in Data Feeds) into smaller chunks of gzip files.

 

It's actually the same way Data Feeds currently exports the data.

 

Cheers,

Piotr  

2 Comments

Avatar

Avatar

DavidLippert

Avatar

DavidLippert

DavidLippert

21-04-2020

Implementing this idea would save a lot of costs. While receiving .gzip exports we could stop post-processing jobs that do "stupid" unzip-and-gzip.

Avatar

Avatar

frankde1

Avatar

frankde1

frankde1

31-10-2020

please implement. We are way beyond the times when files were manually extracted and put into an Excel for Analysis.