A (surprising) difference between Customer Experience and Engagement (part 2)

[#1]

Introduction

This is the second of two blog posts about customer engagement and customer experience metrics on global web properties at 3M.  In the first entry (A Novel Approach to a Web Engagement Metric), I provided an overview of a 3M.com Engagement Index which is a composite measurement of six dimensions of visitor engagement.  In this post, I will share some surprising insight about the relationship between engagement and customer experience metrics.

Measuring Customer Experience

We have been gathering customer experience feedback using surveys on 3M.com for more than two decades.  I was even featured in a Fast Company article about this way back in the year 2000 (Fast Company: Here’s Our Site – Are You Satisfied?).  The form and function of our visitor survey has changed many times, but the fundamental concepts have been consistent.  This customer experience data is a nice compliment to the rich behavioral data in Adobe Analytics.

We trigger feedback surveys at several key points in the visitor experience.  The surveys include structured questions with response values in the range [0,10] and some open text fields.  For the purpose of this Blog, I will focus on a few of the metrics in our survey.  These are not exactly the questions, but the spirit of the metrics are:

• Satisfaction (0 = not at all, 10 = extremely satisfied)
• Likely to Recommend (0 = not at all, 10 = extremely likely) - note this is the basis of Net Promotor Score
• Ease of Use (0 = not easy at all, 10 = extremely easy)

In the past two years we have collected 50,000 data points for these measures.  We directly populate Adobe Analytics dimensions with the response values when a survey is completed.  This allows us to analyze visits using customer experience feedback along with the full digital journey.

More Engagement = Better Experience?

It seems pretty obvious, in retrospect, that we would analyze relationship between our 3M.com Engagement Index and the customer experience metrics.  The reality is that for a long time we just assumed that more engagement is better.  I don't recall what motivated me, but eventually I decided to take a look at the relationship with the data.

The analysis was simple using Adobe Analytics.  The first step was to segment visits that completed a survey into eleven (11) buckets for each of these three survey measures.  Every visit where we get a "1" score in Satisfaction is put in the "Satisfaction=1" segment, and so on.  I created a total of 33 segments across the three measures.  I was then able to create a simple table of the Customer Experience segments from zero (0) to eleven (11) with the average 3M.com Engagement Index of all visits in that segment.

With this data, it was an easy step to do a linear regression by treating the segments from 0-11 as a continuous variable.  Well, I say it was easy but that is because I happen to have Microsoft Excel on my computer.  A side note to my friends at Adobe - I would love to see some basic statistical analysis incorporated into Analysis Workspace :-).  Back to the story, the graph below shows the surprising (but not really) summary of this analysis.

The graph shows a clear negative correlation of the 3M Engagement Index with the Customer Experience metrics.  These correlations have a very high level of statistical significance as noted.  In the messy world of digital marketing, I did not expect to find an R^2 of 0.96.  This pattern was consistent across many different slices of the data including global geographies, time frames, and business segments.

Now what?

After the initial shock, denial, checking data validity (standard analyst steps) we came to terms with this reality.  One of my colleagues summarized this result by saying that "our site is like going to the DMV -- you go there occasionally to get something done, not regularly to chat with all the nice people, stand in lines, and engage with the paperwork."  The data clearly shows that our 3M.com Engagement Index is a reliable predictor of the customer experience.  The more engaged the visitor the more likely they had a bad experience!  Yikes.

We are not at all discouraged by this.  As a global manufacturer, our web content is very important and our visitors arrive with a specific purpose.  They just want to accomplish their purpose quickly and efficiently.  This data has provided valuable insight and will be used to inform additional CX/UX research in 2023.

Please share any thoughts or questions about this post!

Russ Vander Wiel (3M Digital Marketing Analytics Director & Adobe Analytics Champion)