Bulk Lead Extract API questions
Working with the Bulk API, have a few questions about it.
1. What's the easiest way to just get all leads for an initial sync via the bulk API? You need to have a filter. Say I have an instance with 150k leads created over 3 years. Do I need to do 30 day bulk API requests starting at some arbitrary old date to get all leads? This needs to be done through the API. It seems that in this scenario using the non-bulk Rest API with a paging token would be faster (see #3) even though it would result in more calls. This is what we are currently doing but we'd hoped to replace it with the Bulk API. (If this is how this must be would LOVE to see the 30 limit upped to 31 days so we can logically pull by calendar month)
2. (minor) Why does the Bulk API return the word "null" in a missing filed in a CSV? In a CSV a missing/null value is just missing, not the word "null". Here's an example:
id,email,company,sfdctype,postalcode,inferredpostalcode,inferredmetropolitanarea,country,inferredcountry,persontype
408,me@example.com,MyCompany,Contact,null,null,null,United States,null,contact
In every other CSV scenario I've worked in, this would be represented as:
id,email,company,sfdctype,postalcode,inferredpostalcode,inferredmetropolitanarea,country,inferredcountry,persontype
408,me@example.com,MyCompany,Contact,,,,United States,,contact
3. Why does the startedAt date change/why the time weirdness here? I created a lead export of 1 month of leads filtered by createdAt. Here are the stages:
Create:
{
"fields": ["id", "email", "company", "createdAt", "updatedAt","unsubscribed", "emailInvalid", "originalSourceType", "sfdcType","postalCode", "inferredPostalCode",
"inferredMetropolitanArea","country", "inferredCountry", "personType"],
"format": "CSV", "filter": { "createdAt": {
"startAt": "2017-06-01T00:00:00Z", "endAt": "2017-07-01T00:00:00Z" } }
}
Response:
{
"requestId": "bcfe#15d53739e6b",
"result": [
{
"exportId": "b70642a3-8a89-4bf1-a441-24b214096b78",
"format": "CSV",
"status": "Created",
"createdAt": "2017-07-18T02:07:52Z"
}
],
"success": true
}
Status after enqueue:
{
"requestId": "1796#15d53747d57",
"result": [
{
"exportId": "b70642a3-8a89-4bf1-a441-24b214096b78",
"format": "CSV",
"status": "Queued",
"createdAt": "2017-07-18T02:07:52Z",
"queuedAt": "2017-07-18T02:08:49Z"
}
],
"success": true
}
Status after change from "Queued" to "Processing":
{
"requestId": "16721#15d5374f1fe",
"result": [
{
"exportId": "b70642a3-8a89-4bf1-a441-24b214096b78",
"format": "CSV",
"status": "Processing",
"createdAt": "2017-07-18T02:07:52Z",
"queuedAt": "2017-07-18T02:08:49Z",
"startedAt": "2017-07-18T02:09:00Z"
}
],
"success": true
}
Status after complete:
{
"requestId": "17c12#15d53bbac87",
"result": [
{
"exportId": "b70642a3-8a89-4bf1-a441-24b214096b78",
"format": "CSV",
"status": "Completed",
"createdAt": "2017-07-18T02:07:52Z",
"queuedAt": "2017-07-18T02:08:49Z",
"startedAt": "2017-07-18T03:26:09Z",
"finishedAt": "2017-07-18T03:26:25Z",
"numberOfRecords": 43,
"fileSize": 7389
}
],
"success": true
}
Why does the startedAt date change? I polled status.json every minute or so and it was the same as the status above until it finished at which time the startedAt jumped forward.
3a. Also under what circumstances would an export of 1 month of lead records out of an instance that has 1600 leads take more than 1 hour 15 minutes? The final result looks like it took 16 seconds but it entered processing way before that. There is no other API usage, no other Bulk API jobs are queued, etc. I've repeated this 3 times in a row.
4. There's a note in the API that the filter type updatedAt*
* Filter type is unavailable for some subscriptions. Marketo Support can provide you with this information
How can we find out via the API whether this is available or not? Just call it and process a filter error? Or will it just return an empty set? I need to know what to do when we run against an instance and we have no idea whether they have this feature or not, how would we find/extract leads that are updated since a date through the bulk API? So far I haven't seen this on instances I've tested, but I'd like to know what to do.
Any insight is appreciated.