Write and store Bulk Export files gzipped | Community
Skip to main content
March 2, 2018
New

Write and store Bulk Export files gzipped

  • March 2, 2018
  • 2 replies
  • 2354 views

Currently Marketo Leads and Activities Bulk Export API writes and stores Bulk Export files in plain text which in some cases result in producing huge files.

In our case a month worth of activity data produces a ~50Gb files or daily leads export files up to 2Gb in size.

It's wasteful and unnecessarily requires additional storage on Marketo side and forces end users like us pay for increasing daily volume API limits.

Instead export files can be written to storage in a gzipped format in a streaming fashion.

All major technology stacks provide a way to implement it fairly easily.

2 replies

Kenny_Elkington
Adobe Employee
Adobe Employee
March 5, 2018

>forces end users like us pay for increasing daily volume API limits.

This is not true, Peter.  Bulk Export is metered by size-on-disk, and not by data transfer size, so this does not affect your quota utilization..

March 5, 2018

It is in fact true, Kenny.

If you write a gzipped file on disk instead of plain text you end up having a much smaller file. Therefore metered size-on-disk would be less and as a result quota utilization would be less.

February 4, 2020
No text available