Random sample flow step not very random | Community
Skip to main content
P
PhillipWi1
Level 9
May 9, 2018
Question

Random sample flow step not very random

  • May 9, 2018
  • 1 reply
  • 2842 views

Hi everyone

A question about using Random Sample as a flow step. I had a flow step that looks like this:

Add to List

Choice 1: Random Sample is 50 - add to List A

Default: Add to List B.

We then used those lists to report on bookings, as an A/B test.

So here's the weird thing. We have a custom field in Salesforce called "Agency Secondary". I grouped the people in each list by this, and here's a sample of what I see:

​

Agency SecondaryList AList B
Agency 10650
Agency 210550
Agency 32460
Agency 40116

This does not look very random to me! My guess is that Marketo is taking large chunks of my total list and then allocating then to lists in, say,  1,000 person chunks. Not cool.

What should I be seeing here, and is there any way I can make this closer to random? I know that "random" is an interesting topic in computing so it's never going to be truly random, but the above seems a bit ridiculous to me. That's fine if you have consumers where different people in a given chunk of 1,000 shouldn't show similar behaviour, but at a particular travel agency (which is what I'm dealing with) there would be specific booking demographics of clients that could easily sway any A/B test I'm looking to run.

Any ideas?

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.

1 reply

SanfordWhiteman
SanfordWhiteman
Level 10
May 10, 2018

The random sample method is (at least this is the received wisdom) some kind of hash-bucket algorithm. So it's not using a PRNG, that's true. The trend you're seeing here indicates that the hash bucket keys are dependent in some way on that field. But the relationship may be more complex, for example if people were added with those agency values in a particular order?

In any case, you can get a truly random distribution by using a webhook.

P
PhillipWi1Author
Level 9
May 16, 2018

Thanks Sanford....I don't think they would have been added in a particular order, but it's certainly possible.

How would you do it using a webhook? Push out to a service which allocates a value of 0 or 1, randomly?

SanfordWhiteman
SanfordWhiteman
Level 10
May 16, 2018

One of two ways:

  • Random allocation like you said. No problem with this except it tends toward a 50/50 split but is not guaranteed to be exact.
  • Round-robin allocation, which guarantees a strict 50/50 split (for an even number of leads!) but if in some strange reason the input is already interleaved in the same way, would not change that distribution (i.e. if leads, or widgets in general, are already being round-robined between two stacks before being sent in, then the output would simply mirror that distribution, if that makes sense).
Terms & ConditionsAccessibility statement

Loading footer...