Fastest way to update a value | Community
Skip to main content
Gareth_Spencer
Gareth_Spencer
Level 2
May 28, 2018
Question

Fastest way to update a value

  • May 28, 2018
  • 6 replies
  • 7385 views

Can someone please advise whether it is faster to update the same value in one data field for a large quantity of leads (100K+) by:

a) Smart Campaign

b) List Import

Thanks!

    This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

    6 replies

    Charles_THIERY
    Charles_THIERY
    Level 2
    May 28, 2018

    Hello,

    I know that is a tricky question and I'm adding the best experts known in Marketo world to help us out?

    @Grégoire Michel​

    @Sanford Whiteman​

    @Josh Hill [RingCentral]​

    We have seen that marketo takes 7 hours for updating 1 field with same value for 170K persons through a smart campaign "data value change" command.

    Quite surprising in a database update world, we don't know if it's due to GDPR's effect that the marketo server is so slow (maybe everybody in marketo is making update right now).

    We wonder if there is way to perform this kind of action faster, maybe through a excel import?

    BR,

    Charles

    Grégoire_Miche2
    Grégoire_Miche2
    Level 10
    May 28, 2018

    Bonjour Charles,

    We are talking about 24k records /hours which is not that bad. First look at you campaign queue to see whether some other, higher priority activity is slowing down this one. If you are running a smart campaign which first flow step is a data value change, it makes it a medium priority one and that alone could explain the duration of the process.

    Alternatives:

    • I am not sure an import might have been faster, as AFAIK, it usually processes about 20k records / hour.
    • I am not sure the bulk API performance would be better, as to me it's similar to an import.
    • The standard REST API Lead controller has it's own limitation (100 calls / 20 seconds, 1MB body size, Nb of rows per calls...) it would take some computation to know if it can be faster and also some look at the API consumption from other services.

    -Greg

    @Kenny Elkington​

    Charles_THIERY
    Charles_THIERY
    Level 2
    May 28, 2018

    Hello Grégoire,

    Thanks for your reply.

    Is there any documentation I can refer to or it's based on your knowledge?

    I'm looking a marketo documentation about this limitations and how priority are defined in order to prepare my fix.

    Thanks for your help.

    BR,

    Charles

    Kevin_Delgado1
    Kevin_Delgado1
    Level 2
    May 29, 2018

    This is from my personal experience. I like list import as its faster to make mass updates and doesn't queue up my campaigns and cause a bottleneck. The setback with this that I've noticed is that I have a program if an email address changes, update invalid email to false if it was true. If you have something similar, turn it off temporarily so it doesn't cause invalid email addresses to revive by accident just because of a capital letter difference. kdelgado vs Kdelgado.

    One other thing is that list imports will only grab one "person" that matches the email address. If you have duplicates, it may not update the correct one you wanted. You may want to do the list import and then run a smart campaign to cover anything that may have been missed(field is blank or not [value]), which should be relatively low. Or export and cross-check, I do this sometimes with excel so I can verify.

    Also when doing list imports, its not suppose to create duplicates with "persons" with the same email address but I see it happen still(very few though). I use lead source on my side that doesn't get overwritten. I make lead source, "LI_persona-update_20180528". If anything has that value and was created in the past 2 days, just to be safe, I review and merge/remove if necessary.

    Another also to note, if you're updating a field that is synced to your CRM, you may want to turn off the sync temporarily so it doesn't use up too many api calls. Once the update is complete, then go ahead and turn on again. I prefer it this way as it bulk updates instead of one at a time or a few at a time. It uses less api calls from what I've seen. Hope the above helps!

    A
    Anonymous
    May 29, 2018

    I personally have found that list import is significantly faster than using a campaign for updating data.

    A
    Anonymous
    May 29, 2018

    For what it's worth, I tried updating one field on 600k+ records with a smart campaign. I killed the campaign when it had not completed after 4 days and we did a list import which was done the same day.

      A
      Anonymous
      May 29, 2018

      For simple, large-volume changes, Bulk Import is your best bet - it works in 1k chunks and does direct writes to the database.  Smart Campaigns are overkill for that amount of change. Performance can vary based on a few factors, mainly standard attributes versus custom attributes.

      I actually talked about both in my Under the Hood session at Summit.  And by total coincidence, I just recorded the audio to the slides yesterday.  Enjoy!

      Scott

      Video Link : 2381

        Gareth_Spencer
        Gareth_SpencerAuthor
        Level 2
        May 31, 2018

        hi Scott

        Thanks for this.

        Does this slide:
        suggest that importing a list and "bulk import" via API in fact go through the same process exactly, once loaded?  And therefore does this take the same total time to process?

        Also, what reporting is available after bulk import via API, in case of errors that require amendment (v the error reporting available in list import)?

        i.e. What are the pros and cons of each method?

        Thanks.

        Charles_THIERY
        Charles_THIERY
        Level 2
        May 31, 2018

        ok, after testing both way, here are the figures within a same instance:

        - 170 000 persons one field updated = 420 minutes // Smart campaign

        - 170 000 persons one field updated =   30 minutes // List import

        I'm going to compare with bulk API too.

        BR,

        Charles

          Terms & ConditionsAccessibility statement

          Loading footer...