Compression on Bulk Export APIs? | Community
Skip to main content
Kurt_Koller
Kurt_Koller
Level 4
June 22, 2017
Question

Compression on Bulk Export APIs?

  • June 22, 2017
  • 3 replies
  • 6116 views

For Bulk Activity Export - exporting a file it is both stored as plain text and transmitted without gzip compression.

This data is repetitive and compresses very well. In my analysis of files with all activity types, ~90%+ with gzip -6 settings. It's a little better if you're asking for the same activity type.

If we ask for 205M worth of activities, and if the data were stored compressed, it would be 20.5M versus 205M. Even if just gzip were turned on in nginx, at least that would save 90% of transmission time. Ideally this data would be stored compressed to not count toward the quota.

At a minimum can compression be turned on for the file download so that part is quicker? The server currently doesn't seem to send gzipped data even when asked.

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

3 replies

Kurt_Koller
Kurt_KollerAuthor
Level 4
July 19, 2017

Would love to see compression on the file level more than the http transfer level in order to extend the quota.

Would also love to see CSVs not contain the word null for null values between commas:

id,field,field,field,field,field

Right now:

id,null,null,field,null,field

Should be:

id,,,field,,field

E
Adobe Employee
everlyAdobe Employee
Adobe Employee
August 1, 2017

Hello Kurt,

HTTP compression (gzip) is supported by bulk extract leads and activities.  When calling either Get Export Lead File, or Get Export Activity File endpoints, simply add the HTTP header "Accept-Ecoding: gzip" and the response body will be gzipped.

curl -H 'Accept-Encoding: gzip' '123-XYZ-456.mktorest.com/bulk/v1/activities/export/<exportId>/file.json'

-Dave

Kurt_Koller
Kurt_KollerAuthor
Level 4
August 3, 2017

thanks, the accept header wasn't working for us at launch, I will try again. Appreciate all your responses over the last few days.

Frederic_Pinch1
Frederic_Pinch1
Level 2
November 22, 2017

I tried yesterday, and even though the Header Accept-Encoding: gzip gets accepted in the request, it still downloads a regular CSV, not a GZIP file...

What am I missing?

SanfordWhiteman
SanfordWhiteman
Level 10
November 22, 2017

Encoding means the text in the payload is compressed and automatically decompressed by your client. It's not a gzip file (otherwise requesting gzip for JS files, like browsers do, could never work since ungzip isn't a native JS capability).

Frederic_Pinch1
Frederic_Pinch1
Level 2
November 22, 2017

Oh OK, so that still means that the volume downloaded out of the Marketo instance will be compressed, consuming less of the 500MB a day quota?

Terms & ConditionsAccessibility statement

Loading footer...