Tuesday Tech Bytes - Customer Journey Analytics - Week 5 - Experimentation Use Case

[#1]

CJA Experimentation Reporting

Over the last four weeks, we have looked at CJA Golden Nuggets, Tips & Tricks, Best Practices, and Integrations.  Today we'll enter the homestretch as we look at a specific use case - experimentation.  Adobe has recently launched an experimentation panel that many teams may find very useful.  A similar panel is available for some Analytics users, but in CJA it has greater availability and is easy to set up.

CJA unlocks several new use cases.  Today we’ll focus on experimentation and the ease of beginning an analysis.  We’ll walk through ensuring the experimentation components are set up correctly, the process of building the analysis, and we’ll close with some ways this feature could be expanded to bring more value.

The new experimentation panel makes it easy to generate accurate reporting.  This allows the experimentation team to focus more on ideating and implementing testing.  This panel simplifies the process and eliminates the need for creating custom segments and potential errors.

Admin setup

Before the panel can be used, the administrator must set up the experimentation dimensions.  In the appropriate data view, the experiment ID and variant ID need to be identified.

Each dimension will need to be a single value.  Depending upon how the data is set, this may require splitting multiple values into arrays and parsing the IDs into separate dimensions.  See Adobe’s documentation linked at the end of this post as needed.

Within the appropriate dimension, open the context labels dropdown (1).  This will allow you to select either experiment or variant (2).  Once these dimensions have been identified, you’ll no longer see these as options in the dropdown.  If you later need to change the selections, you’ll need to remove the context label from the existing dimension first.

 The dimension search screen allows you to show and sort by the context label field.

Building the project

Once the experimentation dimensions have been set up, you’re ready to begin your project.  In the panel section of the left rail, pull in the experimentation panel.

Begin by selecting the desired experiment ID.  After the variants load, you’ll select the control variant.  Identify one or more success metrics for the test.  Finally, select the normalizing metric for your analysis.  You can analyze per person, per session, or per event.  The selected metric is used as the denominator for the calculations.

The analysis will now be built based upon the selected criteria.  By default, the resulting analysis will add a segment that excludes those who were exposed to more than one variant. 

Analysis overview

The analysis overview shows several key insights.  The summary will provide the results of the experiment. 

The lift and confidence for the first selected success metric are featured.  Hovering above the lift and confidence metrics in the freeform table, we can view the calculation and the provided description.

Figure 1 - example analysis

Figure 2 - example from Adobe's documentation

Detailed results

Freeform tables and trending charts show the performance for each success metric.  Hovering above lift and confidence metrics in the table show the calculation and provided description.

If desired, the granularity of the trending can be changed by clicking the gear for the line graph.

Freeform tables and trending charts show the performance for other selected success metrics.

Customizing project

Now that you have an analysis of your experiment, you can add additional tables or visualizations to enhance your project. 

Summary and future enhancements

This new panel is great and simplifies the experimentation analysis process for analysts.  However, we’re limited to just one experiment dimension and one variant dimension.  There are expanded use cases that currently aren’t possible due to these limitations.

If CJA would allow multiple sets of experiment/variant dimensions, we could easily unlock new insights.  There are several different use cases marketing teams would love!  Imagine if they were able to quickly analyze the lift of various versions of creatives within the same campaign?  Or amongst various campaigns within the same marketing channel?  There are endless ways how teams could leverage these capabilities.

Further information

Adobe’s documentation provides further details including discussion of the statistical methodologies utilized.