AB testing for multiple deliveries in single workflow

[Sujith_02kumar]

Hello Everyone, 

I am trying to configure A/B testing for multiple deliveries within a single workflow. Currently, the method mentioned in the official documentation works for identifying a single delivery winner based on my workflow ID. However, I want to determine the winner for each individual delivery within the same workflow.

Could anyone help me with this?

Thank you!

Regards,

Sujith kumar

[SushantTrimukheD]

Hi @Sujith_02kumar,

It’s not considered best practice to keep the workflow on hold for 1-5 days to identify the winning delivery. However, you can still determine the winning delivery by using the following JavaScript code:

// Fetch KPIs for each delivery manually entered or from workflow
var deliveryIds = ["Delivery_A_ID", "Delivery_B_ID", "Delivery_C_ID"]; // Replace with real IDs

for each (var deliveryId in deliveryIds) {
    var query = xtk.queryDef.create(
        <queryDef schema="nms:deliveryLog" operation="select">
            <select>
                <node expr="count(@id)" alias="total"/>
                <node expr="sum(@open)" alias="opens"/>
            </select>
            <where>
                <condition expr={"@deliveryId = '" + deliveryId + "'"}/>
            </where>
        </queryDef>
    );

    var res = query.ExecuteQuery();
    var total = res[0].total;
    var opens = res[0].opens;
    var openRate = total > 0 ? (opens / total) * 100 : 0;

    logInfo("Delivery " + deliveryId + ": Open Rate = " + openRate.toFixed(2) + "%");
}

• Replace deliveryIds with your actual delivery IDs (find them in the delivery properties or dynamically fetch them).
• Run this in a JavaScript code activity after the Wait.

OR

• Set Up the Workflow
  • Query: Target your full audience (e.g., 200,000 recipients).
  • Split: Divide the audience into equal or custom-sized groups for each delivery variant:
    • Subset 1: Delivery A (e.g., 33% or 10%).
    • Subset 2: Delivery B (e.g., 33% or 10%).
    • Subset 3: Delivery C (e.g., 33% or 10%).
    • Optional: Enable "Generate complement" for the remaining population (e.g., 70%) if you plan a follow-up.
  • Delivery Activities: Attach a unique Email Delivery to each subset. Ensure each delivery has a clear label (e.g., "A/B Test - Variant A").
• Send Deliveries and Wait
  • Add a Wait activity after each delivery (e.g., 1-3 days) to allow recipients time to interact (open, click, etc.).
• Analyze with Enrichment and Test Activities
  • Enrichment: After the Wait, add an Enrichment activity for each delivery to pull key performance indicators (KPIs) from the delivery logs:
    • Link to nms:deliveryLog schema.
    • Add fields like @open (opens) and @click (clicks) aggregated as counts or rates.
  • Query for KPIs: Use a Query activity to calculate totals for each delivery:
    • Schema: nms:deliveryLog.
    • Filter: @deliveryId = [specific delivery ID].
    • Aggregates: Count(@id) for total sent, Sum(@open) for opens, Sum(@click) for clicks.
  • Test Activity: Use a Test activity to define "winner" conditions based on simple thresholds (e.g., open rate > 20%).
• Log Results
  • Use a File Export activity to save KPIs (e.g., CSV file) for each delivery, or display them in the workflow log with a basic JS snippet.
• Optional Follow-Up
  • If you want to send the "winner" to the complement, manually pick the best delivery based on the exported KPIs and configure a final Delivery activity.

Thanks

Sushant Trimukhe

[Sujith_02kumar]

@SushantTrimukheD ,

Thank you for your response. Our requirement is to conduct A/B testing for all promotional campaigns. Some campaigns may contain only one segment, while others may have multiple segments. I want to automate the A/B testing process for each individual segment and determine the winner automatically within the same workflow.

Example:

• Male Segment: A Delivery, B Delivery, Script to determine the winner for the male segment.

• Female Segment: A Delivery, B Delivery, Script to determine the winner for the female segment.

This process should be handled within the same workflow. Do you have any ideas on how to achieve this?

Regards,

Sujith kumar

[DavidKangni]

@Sujith_02kumar 

In this approach, Do not use recurring deliveries as that fails to generate the delivery Id for each delivery and uses only single delivery Id for the execution time

The JavaScript activity is responsible for doing the calculation on the delivery log table to find the winner template. NOTE: The code has been modified to be used on multiple segments. Please replace the internal names with delivery internal names that you have got:

// query the database to find the winner (best open rate)
var winner = xtk.queryDef.create(
<queryDef schema=”nms:delivery” operation=”get”>
<select>
<node expr=”@id”/>
<node expr=”@label”/>
<node expr=”[@operation-id]”/>
<node expr=”[@workflow-id]”/>
</select>
<where>
<condition boolOperator=”AND” expr={“@FCP=0 and [@workflow-id]= “ + instance.id}/>
<condition boolOperator=”OR” expr=’@internalName = “Split3_TypeA”’ />
<condition expr=’@internalName = “Split3_TypeB”’ />
<condition />
</where>
<orderBy>
<node expr=”[indicators/@estimatedRecipientOpenRatio]” sortDesc=”true”/>
</orderBy>
</queryDef>).ExecuteQuery()
// create a new delivery object and initialize it by doing a copy of
// the winner delivery
var delivery = nms.delivery.create()
delivery.Duplicate(“nms:delivery|” + winner.@id)
// append ‘final’ to the delivery label
delivery.label = winner.@label + “ _Winner”
// link the delivery to the operation to make sure it will be displayed in
// the campaign dashboard. This attribute needs to be set manually here since
// the Duplicate() method has reset it to its default value => 0
delivery.operation_id = winner.@[“operation-id”]
delivery.workflow_id = winner.@[“workflow-id”]
logInfo(“Winner: “ + winner.@id)
// adjust some delivery parameters to make it compatible with the
// “Prepare and start” option selected in the Delivery tab of this activity
delivery.scheduling.validationMode = “manual”
delivery.scheduling.delayed = 0
// save the delivery in database
delivery.save()
// store the new delivery Id in event variables
vars.deliveryId = delivery.id

In case you're planning to use recurring deliveries, you'll need to associte the delivery ID with the workflow ID using the below script 

xtk.session.Write ( <delivery xtkschema="nms:delivery" _operation="update" _key="@id" id={vars.deliveryId} workflow-id={instance.id}/>);

Thanks,

David

[Sujith_02kumar · Accepted solution]

Thanks for your response. I wanted to do it dynamically, so I created a segment variable to retrieve both delivery IDs, passed it into the function, and called it in the workflow. This solved my problem.

Regards,

Sujith kumar