Skip to main content

Random Chance for AB Tests

4 minute read                                                                                                                         Advanced​

Split by random chance refers to a technique used to divide a group of users randomly into different segments or subsets for testing or targeting purposes. It involves assigning users to different segments based on a random selection process, ensuring that each user has an equal chance of being placed in any segment. Split By Random Chance is used in flows when we want to test different user journeys/experiences against each other. This could be a registration process, a video, an informative image, a quiz, etc.

Video Tutorial​



In the below example, we are testing the effectiveness of different "Introduction to Glific" videos against each other and checking which one leads to the most positive responses from users.

Lets say there are 3 videos we want to test against each other:

  1. Create the flow with the Split by random chance node and add a separate video in each bucket/branch, label each video sent.
  2. Ask the user if they want to learn more after sending them the video.
  3. Add labels for the positive responses. (and for negative responses, if relevant for your analysis)
  4. Analyse the data to see which video is performing best and make changes to your default flow.

So here, at the end of the test, we will measure how many users got each video, and then how many users wanted to know more after watching a given video. The video with the highest success rate (#Wanted to know more/#Recieved video) is the video we will choose to use in the future.

We may additionally choose to interview some of our sampled users, to learn whether they watched the video or not and their reasons for giving a positive response before making our final decision.

Steps for using Split by Random Chance​

i) Select Split by Random Chance question type.

Img1_AB

ii) Select the collections you want to split on

Img2_AB

iii) The node will look like this.

Img3_AB

Sample Flow​

Img4_AB

Execution Results​

Different users will randomly be sent into different flow branches and see different videos in this flow. Each image represents a different user journey as per the shared example.

Img5 1_ABImg5 2_ABImg5 3_AB

Additional Info​

  1. Ensure that you add users who have enter/ pass through the split randomly node to a collection and use the split by collection node at the beginning of your flow to ensure that these users cannot re-enter the test as this will skew you data results. Refer to the sample flow image to see an illustraation of this.
  2. Read more about how to conduct A/B tests on this blog and read about how Antarang has used this functionality to improve their registration process here

Advanced Feature Webinar​