A novel approach to a web engagement metric

[#1]

Introduction

This is the first of two blog posts about my experience with customer engagement and customer experience metrics on global web properties at 3M.  This first post is an overview of how we defined a custom 3M.com Engagement Index in Adobe Analytics.  In the second post, I will share some surprising insights about the relationship between engagement and customer experience metrics.

The theory behind our engagement index is straightforward.  We defined specific behaviors that our marketing partners felt were indicators of highly engaged (and assumed to be valuable) visitors.  The idea was to use this as a KPI, understand the components of engagement, drive improved engagement over time, understand how different businesses or geographies vary in their engagement, and so on.

We were also striving for some mathematical elegance.  More specifically, we wanted each individual behavioral dimension to have a comparable metric and be able to combine them into an overall composite metric.  To achieve this we created an index measure for each behavioral dimension that was normalized to a [0,1] scale.  This allowed us to further average together each dimensional index to create the overall composite metric.  In summary, the 3M.com Engagement Index is a composite of six dimensions of visitor engagement normalized to a [0,1] scale. 

The 3M.com Engagement Index combines six observable behaviors that represent different signals of an engaged visitor.  We think about engagement as a concept related to, but not the same as, conversion.  Engagement, conversion, and satisfaction are interrelated but also independent.

Retail Analogy

One way to think about these concepts is to consider a typical retail store environment.  Imagine watching people in a store as they are shopping.  There are many different behaviors that you might observe which might indicate a more or less engaged shopper.  These shopping behaviors are different from them actually buying something (conversion) and from how happy they are with the shopping experience (satisfaction).

For the 3M.com Engagement Index, we extended this retail analogy into specific priority behaviors that we measure in Adobe Analytics.  The table below is an outline of the behaviors, the digital equivalent, and the index we defined to measure the engagement behavior.

​​Less Engaged (-)	​More Engaged (+)	​Digital Equivalent	Behavior ​​Index
​Went to one place in the store	​Walked all over the store	​# Pages Viewed	​Page View Index (P)
​Took a quick glance	​Studied things carefully	​Page Scroll %	​Scroll Depth Index (S)
​Quickly entered and le​ft	​Spent a long time in the store	​Time on Site	​Duration Index (D)
​Stopped in one time	​Returned many times	​Repeat Visit	​Repeat Visit Index (R)
​Looked only at marketing and way-finding signs	​Studied products, picked up products, read the packaging	​Product Detail Page View	​Catalog Detail Index (C)
​Did not talk to anyone	​Asked questions, had conversation with a sales person	​Form Submit, Online Chat, Mobile Call	​Interaction Index (I)

Behavior Index Calculations

In order to allow the engagement index to be both normalized and "tuned", each element of the index is defined by using a threshold value.  For example, if we determined that any visit with more than 5 page views represents an engaged visit, we set the threshold for the Page View Index at 5 pages.  For each behavioral element of the index, we defined the behavioral index as the % of visits that exceed the threshold:

[Behavior] Index = % Visits where the index measurement > [Behavior] Threshold

Engagement Score of a Single Visit

At a single visit, we defined a behavior score as "1" or "0" if the visit exceeds the threshold for the behavior.  With this definition, an Engagement Score can be defined for a single visit as the simple sum of the engagement behaviors.  The Engagement Score is an integer value from 0 to 6:

Engagement Score =  ( P + S + D + R + C + I )

Overall Engagement Index Calculation

The 3M.com Engagement Index is simply the average Engagement Score for all visits that is normalized to a [0,1] scale:

Engagement Index = Average ( Engagement Score | All Visits ) / ( 6 )

An equivalent description is the sum of all normalized Engagement Scores divided by the number of visits:

Engagement Index = [ Sum ( Engagement Score | All Visits ) / ( 6 ) ] / [ Number of Visits ]

Setting Thresholds

The threshold values for each behavior index is important to the usefulness of the composite index.  We defined initial threshold values using historical data.  To illustrate the impact of a threshold, if the historical baseline for number of page views is 4.5 with very few visits having more than 10, a threshold value of 20 would make that element of the total index irrelevant to any insight or improvement.

The Engagement Index is tuned by setting the threshold values of each behavioral measure.  Our goal in tuning each index was for all of the individual values to have a meaningful and somewhat comparable contribution in the composite index.  I thought of this like adjusting the input dials on an oscilloscope!  One of the (many) beautiful things about Adobe Analytics is that we were able to adjust these segment and metric definitions "on the fly".

Engagement Index Calculated Metrics in Adobe Analytics

In Adobe Analytics, each individual index is created by using a segment that includes all visits where the threshold of the index is achieved.  The index calculation is simply the number of visits in this index segment divided by the total number of visits.  We created an index metric for each of the six individual engagement dimensions as well as the overall Engagement Index.

What's next?

It took some time for us to define this theory and figure out how to create these engagement metrics in Adobe.  In the end it all worked brilliantly, however…

In the next blog, I will share how I was totally wrong about the Engagement Index!

Please share any thoughts or questions about this post!

Russ Vander Wiel (3M Digital Marketing Analytics Director & Adobe Analytics Champion)