Segmentation in Seconds: How Adobe Experience Platform Made Real-time Customer Profiles a Reality

[#1]

Authors: Aaron Shields and Jody Arthur

Adobe Experience Platform has flipped the traditional relational database model upside down with a new model for segmentation that eliminates the trade-off between small, lightweight profiles that are easy to activate and the data-rich, 360-degree view of the customer to make highly accurate decisions even faster.

When it comes to making the key decisions to power customer experience, enterprise marketers have struggled for years with the trade-off between speed and accuracy. With big data continually increasing what we know about consumers, the gap between a small, lightweight profile and a holistic, 360-degree view of that consumer grows ever wider. Simply put, the bigger the data, the longer it takes to read through that data for actioning. Now, by turning the traditional relational database model upside down, Adobe Experience Platform has found and patented a new way to eliminate the trade-off allowing real-time decisioning on highly accurate profiles.

The problem with using the traditional relational database model for segmentation

Big data offers incredible opportunities for delivering better customer experiences. However, the bigger our big data becomes, the longer our queries take. Optimizing (i.e. shrinking) our data to keep our queries performant remains a significant challenge.

Segmentation, dividing up our customers into groups that share similar characteristics, means asking questions of the data we have about those customers. In order to do segmentation on big data with the traditional relational database model, we start with a giant data lake in which all our profile fragments from multiple channels reside. This is big data at rest. To query this data, we have to write a huge SQL statement to join the tables for each type of profile fragment. Based on the way SQL works, our query is going to have to sift through every record, reading everything in the data set in order to execute. With no easy way to eliminate the data we don’t need from the data lake, the bigger the data set, the longer it takes for the query to run and the more resource-intensive segmentation becomes.

A common method for optimizing this process is to make the data set easier to query by pre-joining all of the tables for all our different profile fragments and then index that to reduce the size of the data set so that the query can execute faster. However, this approach creates a couple of different problems. For example, the joins may inadvertently drop or overlook data we later realize we need for accurate segmentation. In addition, when new data comes in (e.g. a customer logs in on a device we previously didn’t know about), we have to wait until the next batch job and inner join in order to use that information effectively.

Let’s flip that model

Adobe Experience Platform Real-time Customer Profile is a purpose-built customer experience platform that offers a new and innovative approach to segmentation, one that allows for lightning-fast decisioning on streaming data in real-time.

Imagine a world where instead of querying our big data at rest, we maintain our queries at rest and allow the data to flow through them (Figure 1). In this scenario, the number of queries is very small in comparison to our data set. For example, picture an organization with 2,000 segments and 200 million profiles. Rather than spending a lot of time optimizing our data so we can run our queries more quickly, we will instead index and optimize the queries themselves.

Figure 1: Adobe Experience Platform turns the traditional relational database model upside down with its ability to hold queries at rest, steaming data flows through them for dynamic, real-time segmentation.

When we write our queries, the system keeps track of the fields that are needed for segmentation across all of those queries. So, when we talk about the queries being at rest and optimizing them, we are actually talking about keeping track of all of the fields (behaviors and other attributes) that we need for our segmentation. This provides a “superset” of fields that together provide exactly the profile information needed to evaluate our segmentation. With this information already prepared, as data flows into the system, we are able to attach to that piece of data all the extra information we need to run it through our queries. Because the query has everything it needs and only what it needs, when the profile comes out on the other side, our segmentation is complete.

Streaming segmentation: how we get to real-time

Figure 2 shows a simplified version of how we apply this new model to get real-time results. As streaming data comes into the system, the first step is to determine for that identity what other identities have been linked in our Identity Graph. Then we add the profile information from those linked identities (only that which is needed for segmentation) to form the real-time customer profile, which then passes through our segmentation layer. Because our queries already have all the fields they need to perform the segmentation, all segments can be evaluated simultaneously and new segment realizations are appended to the incoming data. That means by the time the data lands in the profile store, the segmentation is complete and ready for decisioning.

Figure 2: Simplified view of streaming segmentation with Adobe Experience Platform.

Identity graph: solving the consumer “identity crisis” for better segmentation

When a customer interacts with business across different channels, that profile information (e.g. a cookie from an anonymous website visit, a mobile device ID, or an email) and any identity information associated with it are stored as profile fragments (Figure 3).

Figure 3: Examples of different profile fragments that result from customer interactions across different channels.

Adobe Experience Platform developed the Identity Graph (Figure 4) to keep track of all the identities for profile fragments created by consumers as they interact with a business on these different channels. As a consumer logs into a webpage, a link is created between that web identity and the CRM identity for that consumer. A login from a mobile app generates additional links. As consumers continue to interact with a brand, the identity graph is continually enriched and updated.

Figure 4: Adobe Experience Platform’s Identity service solves the customer “identity crisis” by finding the connections between profile fragments.

Assembling profiles in real-time

With this information in the platform, we are able to use Adobe Experience Platform’s Real-time Customer Profile Service to stitch together any of the profile fragments that belong together; in other words, we dynamically create unified customer profiles right when they are needed (Figure 5). The process of stitching the data together is also informed by merge policies set up within Adobe Experience Platform Segmentation Service to deduplicate the data and resolve potential conflict data by indicating which datasets should be trusted the most.

Figure 5: Adobe Experience Platform’s Real-time Customer Profile dynamically assembles data across channels using the most up-to-date identity graph.

Segment evaluation

With Adobe Experience Platform, segments are created in the platform’s segment builder workspace or created via API. Segments are built using Adobe Experience Platform’s Experience Data Model (XDM) — a standardized data model to which all profile data is mapped. To optimize the segmentation process, we look through all of the segmentation queries and keep track of which XDM fields are required for these segments. That way, we know exactly what we need.

Once new profile data has passed into our system and we know the current identity graph, we assemble a lightweight profile based only on the required fields. This results in a much smaller and more accurate profile passing through the segment query layer, and it has exactly the data we need to evaluate all segments as the data flows through.

Once the updated profiles exit the segmentation layer, their segment membership has already been updated and they can be published to Adobe’s Real-Time Customer Data Platform (CDP) or other downstream systems via API for actioning in real-time. They also land back in the profile store where they are continually enriched as additional data continues to flow into Experience Platform.

The benefits of streaming segmentation

Adobe Experience Platform’s unique architecture obviates the need to sift through massive amounts of data to evaluate a segment. Instead we perform segmentation on the right data as it moves through the system. There is no need to wait for queries to complete because the results are streamed at the same rate the data comes in.

Remember those 2,000 segments and 200 million profiles? With a traditional relational database model, we will get results for everything up to the time we started our query in minutes or hours. In comparison, Adobe Experience Platform’s streaming segmentation model provides continuous person-to-person updates, which can be sent directly to downstream consuming systems, including Adobe’s Real-Time CDP.

There are benefits for batched data, too

Most enterprises today still have many systems that export data in daily or weekly dumps. For these sources of data, the key benefits of the robust cross-channel identity graph and the single customer view will continue to add huge business value. And remember, we can still evaluate all the segments for the organization as soon as the data lands in Experience Platform, and the time needed to execute them is about the same whether we are evaluating 25 or 2,500 segments.

The transition from traditional batching of data to a more continuous “streaming” model will be a journey for most companies. The best place to start is with data sources that lend themselves better to streaming, such as activity on the web, mobile devices, interactions with kiosks and other touchpoints that are already connected to the Internet. These are the perfect starting point for companies with the ultimate goal of converting all their behavioral and interaction data to streaming.

While some segments may be too complex to be evaluated in a streaming fashion, or the data they depend on is only available in batches, this model will still dramatically improve your ability to connect with the customer. Segments can also be composited so that even if we have a slowly-changing batch component and a reactive behavioral component, both can still be evaluated in real-time.

Adobe Experience Platform has made real-time customer profiles a reality. No matter where your enterprise is in terms of its readiness for real-time, whether you are just beginning to explore the possibilities or are already deep into big data, streaming segmentation in Adobe Experience Platform will help you deliver better customer experiences faster with on-the-fly segmentation on highly accurate cross-channel profiles.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetups.

Resources

1. 360-Degree View of the Customer — https://www.adobe.com/experience-cloud/use-cases/customer-intelligence.html
2. Adobe Experience Platform — https://www.adobe.com/experience-platform.html
3. Identity Graph — https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform-blogs/adobe-experience-pl...
4. Adobe Experience Platform’s Real-Time Customer Profile Service — https://www.adobe.com/experience-platform/real-time-customer-profile.html
5. Merge Policies — https://www.adobe.io/apis/experienceplatform/home/profile-identity-segmentation/profile-identity-seg...
6. Adobe Experience Platform Segmentation Service — https://www.adobe.io/apis/experienceplatform/home/profile-identity-segmentation/profile-identity-seg...
7. Segment Builder Workspace — https://www.adobe.io/apis/experienceplatform/home/profile-identity-segmentation/profile-identity-seg...
8. Adobe Experience Platform’s Experience Data Model — https://www.adobe.com/experience-platform/experience-data-model.html
9. XDM Schema — https://www.adobe.io/open/standards/xdm.html
10. Adobe Experience Platform Pipeline
11. Adobe Experience Platform’s Identity Service — https://www.adobe.com/experience-platform/identity-service.html
12. Adobe Real-Time Customer Data Platform — https://www.adobe.com/experience-platform/real-time-customer-data-platform.html
13. Adobe Experience Platform streaming segmentation — https://www.adobe.io/apis/experienceplatform/home/profile-identity-segmentation/profile-identity-seg...

Originally published: Mar 5, 2020