Segmentation.ai: Automated Audience-Clustering-as-a-Service in Adobe Experience Platform

[#1]

Authors: Charles Menguy, Vidit Bhatia, and Jenny Medeiros

This post describes our innovative audience segmentation service and how it empowers Adobe Experience Platform customers to easily explore and understand their audiences for effective targeting.

With the exponential influx of data and the growing customer expectation for tailored experiences, marketers can no longer heap an entire audience into a single bucket. Segmenting these audiences provides some guidance, but traditional segmentation based on age, gender, and income fails to take into account customers moving from device to device and channel to channel.

Adobe Audience Manager is a leading data management platform that reimagines audience segmentation. It enables modern marketers to seamlessly collect data from multiple sources, build focused audience segments (or clusters), and follow their customers’ journey to deliver relevant messaging in real-time.

It still, however, requires manual input to create each segment, along with some expertise to understand the data. With Segmentation.ai, we aim to simplify this process by automatically creating audience clusters and condensing insights into interactive graphics for easy exploration and comprehension.

Audience segmentation in Adobe Audience Manager

In Adobe Experience Platform, marketers rely on Adobe Audience Manager to segment their users and draw actionable data to better inform their sales choices. To achieve this, Adobe Audience Manager captures billions of online user interactions, each of which is tied to a trait that represents a given user’s behavior. These traits can then be combined into meaningful segments for cross-channel targeting through Adobe Experience Platform.

The problem with the current audience segmentation is that the overwhelming amount of collected data makes it difficult for users to create representational segments and explore important insights. Another speed bump is the need for users to manually define rules in Adobe Audience Manager for segment activation. A rule being, for example, “users aged between 25–30 and studying engineering.”.

To remedy this overbearing process, Segmentation.ai uses customer traits and segments to achieve the following:

• Create an accurate timeline of user interactions.
• Shrink lengthy user journeys – where each has differing numbers of traits and segments – into equal, 16-digit vectors for easier analysis.
• Convert audience clusters into interactive, graphical visualizations that non-domain experts can swiftly explore and understand.
• Create a streamlined, end-to-end solution for segment activation.

Additionally, this innovative service relieves customers from bloated, overly-complex dashboards; instead of allowing them to interact with their audience segments through a clean, focused, and user-friendly interface.

Segmentation.ai architecture

Segmentation.ai is built entirely upon Adobe Experience Platform to leverage its services, such as Real-Time Customer Profile and Data Science Workspace.

As shown in the figure below, the Segmentation.ai service is comprised of five stages: pre-processing, modeling, clustering, discovery, and ingestion.

Figure 1: Diagram of Segmentation.AI architecture.

To better illustrate each stage, we will explain using an example.

Pre-processing

Consider a user searching online for UX design tools who lands on the Adobe XD website. As they browse and eventually purchase the product, Adobe Audience Manager and Adobe Analytics capture the user’s interactions and combine them to form traits and segments.

Segmentation.AI then embeds these traits and segments into a user journey – building a timeline that accurately represents the user’s behavior.

For this, we run a PySpark job on Databricks for compute, along with Data Science Workspace recipes. The lookback window is currently seven days, and we automatically determine which traits and segments are the most meaningful for the customer.

Modeling

The service now has the UX design customer’s user journey, along with the user journeys of millions more who have interacted with Adobe products. Although some users have shorter journeys than others, and longer journeys can contain thousands of traits and segments. This results in huge amounts of uneven data that is understandably difficult to work with.

To solve this challenge, we used Tensorflow and a deep learning auto-encoder to reduce each user journey to just 16-dimensions – while preserving the user’s behavior. Now, Segmentation.AI can comfortably digest these smaller user journeys to train and score the embeddings model.

Clustering

Since every user is on the website for different reasons, the service shows high-level clusters for the Adobe products the users interacted with.

Here, we run clustering on the sampled embeddings dataset.

We also simplified the user data even further since 16-dimensional data isn’t easy to explain. So, we partnered with ShapeVis for topological data analysis. This allowed us to group user data and generate a summarized, two-dimensional graphical representation, all while maintaining its inherent structure.

Discovery

Now, the service is ready for the marketer to explore the clusters and determine meaningful audience insights. They can navigate to the Segment.AI module within Adobe Experience Platform and view the high-level clusters that represent the users who interacted with Adobe products. (See image below.)

Figure 2: Sample of clusters.

The clusters with darker colors represent the highest concentration of user traits.

Ingestion

Once the marketer has found clusters of interest, they can click to expand them and view meaningful insights about the audience, such as the main users for a specific product and what device they’re using. Then, to activate an audience segment, they can simply ingest it into the Real-Time Customer Profile.

For this final stage, we have two parts:

• Segment expansion: Since the clusters showed are of sampled data, we allow the customer to expand a cluster and view all the users in the complete dataset. This is done by running a PySpark job on Databricks.
• Segment ingestion: In the same job, we create a new batch in a dataset marked for Real-Time Customer Profile ingestion, along with segment metadata for the new audience. Currently, it can take a few hours for the segment to be ingested.

Implementation challenges

Since Adobe Audience Manager traits are defined by the user and not all traits are accurate or up-to-date, we faced the challenge of determining the right set of traits to run our model. As of now, we have solved the problem to an extent (using what?), but continue to scout for a better solution.

Another challenge was explaining the smaller audience clusters, as some traits captured much more information about the user than others. For example, a large cluster could easily reveal that most users were using iPhones; but a small cluster discerning the specific iPhone models would be much more difficult to explain.

What’s next for Segmentation.AI

We are proud to further the business goals of Adobe Experience Platform users with this technology and are delighted to announce that this innovative service is already showing positive results in customer trials. Even so, we continue to improve, test and adapt it to the needs of our customers. Such improvements include the following:

• Enhance the user interface: Currently, our customers rely on Adobe support for model configurations. We aim to build a simple and accessible user interface that will enable customers to take charge of this process.
• Add unified data support: To extend the power of Segmentation.AI, we are adding support for different Experience Event datasets, such as Adobe Analytics data. Today, this technology only works with data from Adobe Audience Manager.
• Add real-time segment description: With the addition of segment descriptions that function as queries, customers will be able to further refine segment creation and streaming.
• Build objective-driven segment discovery: Using machine learning models, this type of segment discovery will swiftly create segments that optimize specific marketing goals.

With these improvements underway and many more innovations on the horizon, we continue to help Adobe Experience Platform users fine-tune their marketing efforts and drive powerful, customer-focused experiences in real-time.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetups.

Originally published: Jan 30, 2020