Real-Time Scoring to Improve Personalization with Adobe Experience Platform

[#1]

Authors: Wouter Van Geluwe, Krishna Punyakoti, Kalaiyarasan Venkatesan, Althaf Kandi, and Douglas Paton

This blog details a single implementation or customization on Adobe Experience Platform. Not all aspects are guaranteed as general availability. If you need professional guidance on how to proceed, then please reach out to Adobe Consulting Services on this topic.

In this post, we look at an early stage POC workflow on how to generate real-time sales predictions for anonymous, unknown website visitors using Adobe Experience Platform. We discuss how we used Adobe Experience Platform and Data Science Workspace services to provide brands with the ability to use ML Models to score customer behavior in real-time to personalize and contextualize personalization in order to increase conversions.

When anonymous prospects interact with your website, knowing whether or not they’re likely to become a customer gives you the ability to make informed decisions about how you should engage with them.

This is exactly where Machine Learning comes into play. Machine Learning makes it possible to learn from all customer behavior and then evaluate this specific new anonymous prospect’s behavior on the fly. Predicting the next-best-action or content based on customer behavior calls for Real-time ML (RTML), which provides a way of listening to f.i. an anonymous prospect’s online behavior, during this session but also from past sessions, and have an ML Model calculate a score that expresses how likely this prospect is to actually become a paying customer of your product.

Knowing this score helps inform the way you can and should approach a person. Instead of the usual bombardment of generic messages, popups and calls-to-action on their first visit, you sit back and listen first. After having truly listened to your customers, you can then respond intelligently with better, more appropriate messaging while they’re on your site and relevant advertisements on other channels once they’ve left. Ultimately, while many brands these days still apply a mass-marketing approach, accurate scoring gives you a stronger edge over the competition.

Today, many brands are already applying ML using the data inside their own data lakes. Some of the ML capabilities in those data lakes already have capabilities of evaluating and scoring behavior in near real-time. However, those environments are typically very disconnected from the channels used by marketers to speak with customers.

Many of the brands we work with report delays of hours up to days between their ML environment calculating a new score and their downstream channels actually being able to use that score in a consistent and orchestrated way across the various channels of activation.

As with everything related to communication, it’s always better if it’s immediate and interactive.

So brands need a way to get their machine learning-based insights activated towards their customers faster than ever before. Activating these ML-based insights faster than before helps brands to understand who their ideal prospects are, and stop wasting their advertising efforts on the percentage of customers who are either already buying or who’ll never buy. Instead, brands can now understand that golden zone in the middle, customers who are on the fence, better than ever and with that, they have the power to deliver a more personalized and contextual experience that speaks to that specific prospect’s direct needs.

Better scoring with Adobe Experience Platform

With Adobe Experience Platform Data Science Workspace (DSW) and the concept of Real-time Scoring, we were able to reduce the amount of time it took to score prospects from hours or days, down to milliseconds. The path of collecting customer behavior on a website that leads to calculating an ML-based propensity score starts with what is the foundation of Adobe Experience Platform: the Experience Data Model (XDM). XDM is a standardized language to describe everything that’s related to experiences.

In the step-by-step breakdown of this use-case, let’s start with the customer behavior on the website. If you want to score a customer’s behavior in real-time, the logical starting point is to collect data in real-time. Collecting that data in real-time is done by making use of Adobe Experience Platform Launch, Adobe’s tag management solution.

Adobe Experience Platform Launch makes it possible to use the brand-new Adobe Experience Platform Web SDK Extension to collect data straight from a website’s data layer, map it against an XDM-based schema and then stream it into Adobe Experience Platform. By mapping data against an XDM schema immediately upon collection, a lot of time and effort is saved downstream.

Every type of customer behavior that is collected client-side and sent to Adobe Experience Platform is from now on defined as an Experience Event. Experience Events describe events like Page Views, Product Views, Add To Carts, Purchases, Offline POS Transactions, Call Center Interactions and much, much more. All of these Experience Events are then linked to a customer’s Real-time Customer Profile in Adobe Experience Platform.

Figure 1: Adobe Experience Platform Web SDK in Adobe Experience Platform Launch.

As data is streamed into Adobe Experience Platform as an Experience Event, it hits our Adobe Pipeline. The Adobe Pipeline is one of the largest Apache Kafka clusters globally and it guarantees real-time processing of Adobe Experience Platform data. From Pipeline, data is then made available to Adobe Experience Platform Real-time Customer Profile (running on Microsoft Azure Cosmos DB), and eventually, the Experience Data Lake (running on Microsoft Azure Data Lake Services) where it is stored in Datasets (of which the structure is defined by an XDM Schema).

All the data stored in datasets can be accessed directly from Adobe Experience Platform Data Science Workspace, without any delays or complex data wrangling processes.

Typically, data scientists these days are highly trained and qualified individuals and when they finally come out of school and score an awesome job as a Data Scientist for a leading brand, what they do is spend 80% of their time on trying to understand all the data that lives inside their company and trying to map that data so that it can be used for Machine Learning purposes. There’s a better way for these highly qualified individuals to spend their time. Thanks to Adobe Experience Platform and XDM, they can now finally stop spending 80% of their time on data wrangling and instead, can spend that time on things that matter: to build, train and optimize ML Models that actually impact the bottom line of the brand they work for.

Figure 2: Adobe Experience Platform Data Science Workspace and Jupyter Notebooks.

Adobe Experience Platform Data Science Workspace provides data scientists with an embedded version of Jupyter Notebooks directly inside Adobe Experience Platform and as such, they can run notebooks using Python, R, Spark, PySpark, and Scala natively on top of the Experience Data Lake of Adobe Experience Platform. We’ve built two SDKs for Data Scientists to use:

• Data Read SDK that allows data scientists to easily read data from Adobe Experience Platform (Remember: all data is mapped against a XDM Schema which means it’s already standardized and understandable).
• Data Write SDK which allows data scientists to write back things like propensity scores, recommendations, next-best offers, etc to Adobe Experience Platform’s Real-Time Customer Profile for activation.

These two SDKs solve the two biggest hurdles Data Scientists face on a daily basis: understanding data and writing back ML-based output to other systems for activation.

Data Scientists will build and train their ML Model using those SDKs inside Jupyter Notebooks in Adobe Experience Platform and once that’s done, their ML Model can be packaged and operationalized as a Real-time ML (RTML)-service that can be scheduled to train on a predetermined frequency, which is then published on the “Edge” for Real-time Scoring.

A practical example: Luma Insurance

Luma Insurance is a typical Insurance Company with a big offline dealer-network and a website with lots of traffic. Luma Insurance has been trying to personalize their digital communication for years, but it’s very hard to personalize anything if you don’t know who you’re speaking to. As most of the traffic on the website is from unknown, anonymous users who only visit the website once or twice, it’s difficult for Luma Insurance to build out a deep customer profile.

Most of these unknown, anonymous visitors only come to the website with one goal: get a quick indication of the cost of insurance, by filling out a form which then gives them an idea of the insurance cost.

Because of this limited amount of information, Luma Insurance has a hard time optimizing any kind of digital campaign which leads to prospects being bombarded with irrelevant and inconsistent messaging.

As a consequence, Luma Insurance doesn’t understand which customers are interesting targets and which customers are not, is unable to optimize their ad spend and is unable to understand which channels are best used to interact with any given customer.

If Luma Insurance wants to speak in an intelligent and consistent way, with a single voice, to any prospect, they need a central brain for communication and orchestration. This central brain for communication and orchestration needs to be intelligent and needs to apply that intelligence in real-time.

To Luma Insurance, that central brain for communication and orchestration is called Adobe Experience Platform.

Figure 3: A high-level look at how real-time works.

As you can see in Figure 3, when the anonymous prospect clicks the “Get Quote”-button on the Luma Insurance demo website, all the data that is collected in the Insurance Application Form is streamed into Pipeline and made available to Adobe Experience Platform as an Experience Event. Adobe Experience Pipeline also makes that data available to the RTML-endpoint, which then extracts the information of the prospect’s application (fields like Car Brand, Car Type, and Driver Age) as features needed by the ML Model that was previously trained in Adobe Experience Platform Data Science Workspace.

The RTML-endpoint now evaluates the anonymous prospect’s application based on the defined features and assigns a propensity score to that anonymous prospect’s profile in real-time. This score is then attached to that anonymous prospect’s Real-time Customer Profile inside Adobe Experience Platform.

From the Real-time Customer Profile, every new attribute or event leads to real-time evaluation of all segments that are configured in Adobe Experience Platform. In this case, based on the propensity score of f.i. 67, the anonymous prospect now qualifies for a segment “Car Insurance Propensity 60–70”. This segment qualifies in real-time and is made available to all destinations that are connected to Adobe Experience Platform.

This means that applications like Adobe Target, Adobe Audience Manager, Real-Time Customer Data Platform Application Service, Google DV360, AppNexus, Facebook, and many more will be informed of this anonymous prospect’s segment qualification and they will each be able to reach that customer on their respective channels with contextual and more relevant advertising.

This gives you a level of personalization that cuts through the typical generic, mass-marketing messaging and tells the prospects, “Hey. These folks know where I’m coming from.”

Creating a framework

The exciting thing about developing this real-time scoring process is that we ended up creating something that wasn’t just a single-use solution. We created a framework that can be used with any ML Model and for any kind of use-case. There’s no limit to what we can do here.

Adobe Experience Platform Data Science Workspace gave us the ability to create something that sounds incredibly complex, in a way that was actually straight-forward and efficient. And by having gone through this configuration once, we can change the use case and everything still works exactly like it was supposed to.

We effectively gave ourselves an edge over our competition, by providing our customers a way to gain an edge on their competition.

Examples of use cases for real-time scoring

With the RTML-framework in place, it’s easy to come up with several use case examples of how real-time scoring can work in various industries:

• Insurance: Companies can identify a person’s propensity for different insurance types and adapt their communication and orchestration strategies in real-time.
• Airlines: Airlines can now dynamically and in real-time assess how likely someone who just booked a flight is to purchase other services, like travel insurance, accommodations, or rental vehicles.
• Telecommunications or Utility: Real-time Machine Learning can, and should, be used to calculate Churn Propensity scores in real-time and take action on them in real-time, in an omnichannel world.

Next steps

Now that we’ve got a framework that can be applied to any number of industries and use cases, the next step is for us to formally productize this and make it available to customers. We can take this one step further by operationalizing this process and even provide pre-built “ML Recipes” for specific industries.

By offering up “pre-built” ML Recipes we enable brands to jumpstart their evolution to intelligent and real-time personalization and get started with real-time scoring quickly and easily, helping them gain an edge over their competitors.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetups.

Resources

1. Adobe Experience Platform — https://www.adobe.com/experience-platform.html
2. Adobe Experience Platform Experience Data Models (XDM) — https://www.adobe.io/open/standards/xdm.html
3. Adobe Experience Platform Data Science Workspace (DSW) — https://www.adobe.com/ca/experience-platform/data-science-workspace.html
4. Real-time Customer Data Platform— https://www.adobe.com/experience-platform/real-time-customer-d ata-platform.html
5. Adobe Experience Platform Launch — https://www.adobe.com/experience-platform/launch.html
6. Apache Kafka — https://kafka.apache.org/
7. Adobe Target — https://business.adobe.com/ca/products/target/adobe-target.html
8. Adobe Audience Manager — https://www.adobe.com/analytics/audience-manager.html
9. Google DV360 — https://marketingplatform.google.com/about/display-video-360/
10. AppNexus — https://www.appnexus.com/
11. Python — https://www.python.org
12. R — https://www.r-project.org
13. Spark — https://spark.apache.org/
14. PySpark — https://spark.apache.org/docs/latest/api/python/index.html
15. Scala — https://www.scala-lang.org
16. Microsoft Azure Cosmos DB — https://azure.microsoft.com/en-ca/services/cosmos-db/
17. Microsoft Azure Data Lake Services — https://azure.microsoft.com/en-ca/solutions/data-lake/
18. Jupyter Notebooks — https://jupyter.org/

Originally published: Mar 31, 2020