How to speed it up when getting multiple Leads by SOAP API?

Forum|Forum|12 years ago
October 29, 2013
5 replies
1403 views

Hi all,

Now getMultipleLeads() costs me nearly 4 minutes per 1000 records. And I have more than 500k records updated per a week. I need to get them out in 24 hours at most.
Is there any way to speed it up?

Best Regards,
Biao

Marketo

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

Best answer by

Biao,
Here are our recommendations -

1. If you know which lead attributes you need to execute your business logic, you can specifying only those attributes you wish to be returned with each lead. This helps in two very big ways. The performance is much faster and you will also reduce the response payload. This will give you the biggest performance boost with the least implementation change.

                Sample Input XML:
                                <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
       <ns2:paramsGetMultipleLeads xmlns:ns2="marketo.com/mktows/">
       <leadSelector xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:LastUpdateAtSelector">
      <latestUpdatedAt>2013-09-08T16:13:52.977-07:00</latestUpdatedAt>
       <oldestUpdatedAt>2013-09-05T16:13:52.978-07:00</oldestUpdatedAt>
       </leadSelector>
       <batchSize>200</batchSize>
       <includeAttributes>
       <stringItem>FirstName</stringItem>
      <stringItem>AnonymousIP</stringItem>
       <stringItem>Company</stringItem>
       </includeAttributes>
       </ns2:paramsGetMultipleLeads>

2. Use the <leadSelector> field to specify dates instead of <lastUpdatedAt>. (This alone won't help performance – however <lastUpdatedAt> is deprecated)
                                Sample Input XML:
                                <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
       <ns2:paramsGetMultipleLeads xmlns:ns2="marketo.com/mktows/">
       <leadSelector xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:LastUpdateAtSelector">
      <latestUpdatedAt>2013-09-08T16:13:52.977-07:00</latestUpdatedAt>
       <oldestUpdatedAt>2013-09-05T16:13:52.978-07:00</oldestUpdatedAt> 
       </leadSelector>
         <batchSize>200</batchSize>
       </ns2:paramsGetMultipleLeads>

3. If you want to really speed things up now that you are using leadSelector is to specify a date range. <oldestUpdatedAt/> is the since time stamp and <latestUpdatedAt/> is the until time stamp. With this approach you could thread multiple calls for different time ranges. If you choose this approach I highly recommend you implement your SOAP client using persistent SOAP connections. This is a bit more complex, but the rewards can be significant for processing larger data sets.

A

Anonymous

What is the environment running API calls? I am used to have faster execution times for getMultipleLeads.

A

Anonymous

VS2010 with C#

A

Anonymous

I am not in position to compare any results since I am Solaris, Linux and Mac OS X user therefore far away from any Microsoft products. However I can comfortably state there is noticeable performance penalty using Eclipse IDE (Helios and Kepler), in some ways similar to Visual Studio. I would suspect the slowness would be a mix of network bandwitdh and Visual Studio itself.

I may be totally wrong and will gladly accept any comments from someone familair with Microsoft environments.

A

Anonymous

Accepted solution

Biao,
Here are our recommendations -

1. If you know which lead attributes you need to execute your business logic, you can specifying only those attributes you wish to be returned with each lead. This helps in two very big ways. The performance is much faster and you will also reduce the response payload. This will give you the biggest performance boost with the least implementation change.

                Sample Input XML:
                                <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
       <ns2:paramsGetMultipleLeads xmlns:ns2="marketo.com/mktows/">
       <leadSelector xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:LastUpdateAtSelector">
      <latestUpdatedAt>2013-09-08T16:13:52.977-07:00</latestUpdatedAt>
       <oldestUpdatedAt>2013-09-05T16:13:52.978-07:00</oldestUpdatedAt>
       </leadSelector>
       <batchSize>200</batchSize>
       <includeAttributes>
       <stringItem>FirstName</stringItem>
      <stringItem>AnonymousIP</stringItem>
       <stringItem>Company</stringItem>
       </includeAttributes>
       </ns2:paramsGetMultipleLeads>

2. Use the <leadSelector> field to specify dates instead of <lastUpdatedAt>. (This alone won't help performance – however <lastUpdatedAt> is deprecated)
                                Sample Input XML:
                                <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
       <ns2:paramsGetMultipleLeads xmlns:ns2="marketo.com/mktows/">
       <leadSelector xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:LastUpdateAtSelector">
      <latestUpdatedAt>2013-09-08T16:13:52.977-07:00</latestUpdatedAt>
       <oldestUpdatedAt>2013-09-05T16:13:52.978-07:00</oldestUpdatedAt> 
       </leadSelector>
         <batchSize>200</batchSize>
       </ns2:paramsGetMultipleLeads>

3. If you want to really speed things up now that you are using leadSelector is to specify a date range. <oldestUpdatedAt/> is the since time stamp and <latestUpdatedAt/> is the until time stamp. With this approach you could thread multiple calls for different time ranges. If you choose this approach I highly recommend you implement your SOAP client using persistent SOAP connections. This is a bit more complex, but the rewards can be significant for processing larger data sets.

A

Anonymous

Thanks, bgomes.

And hi Raj,
I've tried your first way, and when taking 1/10 of the attributes, about half time would be saved. So I guess something else also costs large time. Thank you very much.

Best Regards,
Biao

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded