I have a currently running test where a winning treatment was recognized/achieved. On 6/21/16 12AM, I was asked to update the test Traffic allocation to 100% to the new experience that was declared the winner. I altered the Activity Traffic Allocation to 0% Control/100% Treatment B. However, as 6/21 day progressed the metrics showed traffic still going to Control when we wanted all traffic to go to new treatment.
QUESTION: Is there a transition process over time for traffic to actually go to a full 100% or should it happen relatively instantly?
It is recommended that, if you want to change percentages or greatly affect the flow of people into each experience, you should create or copy a new campaign.If you change the percentages on different experiences, it will take a few days for the data to normalize again
Ex- if your A/B test is split 50/50, and then you change the split to 80/20, for the first few days after that change, the results might look skewed. If the average time to conversion is high, meaning it takes someone several hours or even days to make a purchase, then these delayed conversions can affect the reports. So, in that first experience where the number went from 50 to 80, and the average time to conversion is two days, only people from the 50% of the population are converting on the first day of the test, although today 80% of the population is entering the experience. This makes it look like the conversion rate plummeted, but it will normalize again after these 80% of people have the two days to convert.