Implementing Adobe Experience Platform Real-Time Customer Profile through our “Customer Zero” Program

[#1]

Authors: Joel Huff, James Rennie, Nishith Kota, Kevin Davis, Gowtham Ranganath, Jayant Bhushan, and Jody Arthur

This blog is how Adobe Experience Platform was implemented at Adobe IT and Support organizations to help sales and support teams deliver better customer experiences as part of our "Customer Zero" program.

When it comes to using enterprise data to deliver extraordinary customer experiences, 1/10th of the data often represents 10x of the complexity. This complexity is the result of customer data fragmentation caused by many different types of data from numerous data silos.

Enterprises across all industries are struggling with data fragmentation as a roadblock to their ROI for deep investments in digital transformation intended to increase loyalty and retention by providing exceptional customer experiences.

The data fragmentation roadblock presents two fundamental challenges that must be overcome in order to deliver indelible personalized customer experiences in real-time.

• The first challenge is to create a 360-degree view of the customer, which is critical to delivering exceptional customer experiences. This requires the aggregation of all customer data from diverse sources into one platform so that multiple identities, behavioral data, and interactional data can be resolved. However, this alone does not generate business results.
• The second challenge enterprises face is activating the customer profile downstream in all the different interaction channels that impact the customer journey. The latter is necessary to provide the customer with the right experience at the right time. This requires the ability to create and manage audience segments and profiles that can be personalized across any channel in real-time.

Adobe Experience Platform is addressing both of these challenges with its ability to aggregate diverse customer data sets and stitch them together into one unified profile that can be served through APIs to any one of a number of downstream consuming systems.

Adobe has a program called "Customer Zero" for internally testing our innovations, to assure both marketing and experience experts that what we are releasing is "fit for use" by our customers. This blog post describes how Adobe Experience Platform was deployed with Adobe's internal marketing and IT organizations to share with the world what we learned in that process and our recommendations for Adobe customers developing their own implementations within Adobe Experience Platform.

High-level architecture: Working in an imperfect state

Adobe's first use case for Adobe Experience Platform was to develop and implement Adobe Experience Platform Real-Time Customer Profile for Adobe's Customer Activity Hub (CAH). CAH is the system we built to help our customer support and sales agents get the data they need to support Adobe customers. We started with CAH because as a customer-centric company, Adobe knows the critical role that customer support plays in the customer journey. We knew we had to go beyond our CRM system to gather and stitch together all the customer data available to deliver the best experiences to Adobe customers.

This data exists in many different silos in varying formats and often with high latency. CAH integrates data from many different sources allowing the IT team that maintains our CRM applications to make one call to get everything they need to have a 360-degree view of a customer. This enables Adobe sales and support teams to pull all the data they need into one screen so they have a complete understanding of a given customer's activity.

Figure 1: Overview of Customer Activity Hub on Adobe Experience Platform

The challenge of Adobe Experience Platform Real-Time Customer Profile lies in aggregating diverse customer data sets into a single profile that can be operationalized in real-time. The figure below illustrates the high-level architecture of Adobe's CAH. Prior to our Customer Zero implementation, Adobe's CAH relied on our Hadoop system, which aggregated and operationalized the data for profile modeling in a project-oriented fashion, which in most cases, involved daily batch processing into channel-specific silos. As the needs of our support and sales teams became more real-time in nature, it became clear that moving CAH onto Adobe Experience Platform was an important thing to do despite that fact that we were still at least a year away from a general release.

Figure 2: High-level architecture for Customer Activity Hub on Adobe Experience Platform

For this first implementation, we needed something with minimal dependencies that could go live on the platform long before making it generally available. We decided to work with Adobe's existing Hadoop system as the starting point because it already met the key data requirements our sales and support team needed. This allowed us to focus on ensuring the first release provided clear business value while keeping the initial new dependencies to a minimum.

Here's how our CAH implementation works. Data from multiple sources are ingested into Adobe's IDS Hadoop platform:

• Adobe Analytics data, including behavioral data, survey response data, churn risk, and insights around what might be causing it.
• Product data, including information about what Adobe Creative Cloud products our customers have downloaded, what products they are using, and which features within the products they are using.
• ERP data from Adobe's SAP implementation such as contract information, seat deployments, user entitlements, credit card expiration dates.

Once in our Hadoop platform, we have various ETL workflows that aggregate and consolidate the data and orient it toward a single customer, whether that be a single individual or a team or an enterprise. We then used the Adobe Experience Platform APIs and various components to assemble the data and ship it into Adobe Experience Platform.

Once the data was expressed in Adobe Experience Platform XDM, we were able to leverage the data management features in Adobe Experience Platform to tag the CAH datasets for the Real-Time Customer Profile. This allowed each batch to be automatically ingested into the Real-Time Customer Profile service for on-demand activation access.

Our data ingestion process took advantage of the Adobe Experience Platform's batch ingestion capabilities to organize the data flowing in and to monitor the batch ingestion status. These processes ensure that we don't have any errors in the ingestion (e.g. checking to see that the number of records sent is the same as the number of records received) prior to shipping the data into the Adobe Experience Platform data lake for use by our Real-Time Customer Profile service.

Once the batch upload into the data lake was complete, the profile automatically ingests the data sets. The data stored in the data lake is persisted and partitioned in a way that is efficient for analytics and long term storage. The profile ingestion jobs then persist the data efficiently to facilitate real-time access in CAH. Using our APIs, we were able to make the unified profile data available to the three consuming applications within our CRM ecosystem.

It's worth noting that any Adobe customer can follow the same path to implement Adobe Experience Platform Customer Profile for any number of use cases. Our APIs are available to any Adobe Experience Platform customers to use inflowing their data into the platform from any source and any architecture. The main prerequisite is a willingness to think of the data as a product. This means designing a more generic schema and planning for the extensibility needed to allow the consuming systems to access the data in any way they need with a consistent set of APIs. The goal is to build a framework that allows different teams to append the profile attributes they need and still fit into the overall schema structure that organizes profile data.

Navigating breaking changes in XDM during a live implementation

Adobe's customer support and sales teams have two instances of Microsoft Dynamics: one for individual accounts and one for enterprise accounts. The figures below illustrate the kind of information they are now able to pull from CAH in real-time. The simplicity of the information provided in the CAH interface belies the complexity of the behind-the-scenes processes required to provide it and the challenges we encountered while developing those processes.

Figure 3: Example of how data are consolidated (shown in the red boxes) into the user interface for Dynamics Agent Experience for working with individual accounts in the Customer Activity Hub.

As Customer Zero, we were "eating our own dog food" going live while the Adobe Experience Platform (and the early version of XDM) was still in beta. Throughout the beta period in 2018, we were working to ensure all the needs of our sales and support teams were addressed and to identify any problems resulting from the implementation. Collaboration across teams allowed us to start testing, diagnosing, and resolving issues early in the staging environment and into the production environment.

Adobe Experience Platform team received useful feedback and suggestions during the beta period about the overall approach to XDM schemas. The most common theme that emerged from this feedback was that XDM's core schemas had become too bloated to include cases that weren't needed by all customers.

Adobe's Adobe Experience Platform XDM schemas were intended to cover the full breadth of use cases across our customer experience management system and the entire Adobe Experience Cloud set of solutions. As a result, over the XDM beta period, the core schemas for events and profile had grown to include all of these use cases. To resolve this, the Adobe Experience Platform team adjusted its approach to building schemas, starting with the minimal set of attributes necessary and a set of standard mixins that represented additional use cases. This allowed our customers to build just what they needed without having to accommodate the entire breadth of the full data experience model.

This new approach, starting lean with building blocks that can be added as needed, was a major improvement in Adobe Experience Platform XDM, which was rolled out in March 2019. However, it also introduced disruptive changes to the pre-existing implementation we built during the XDM beta period.

Rebuilding our XDM allowed us to test the model end-to-end, but it also meant we had a live implementation that was going through a number of breaking changes in the way XDM worked. This work delayed the release of XDM a bit. But in return, we (and Adobe Experience Platform customers) are now enjoying the benefits of a leaner schema composition model.

Results from our Customer Zero implementation

The initial load at the beginning of the CAH build was several GBs of data collected from nine different data sets. Adobe Experience Platform Customer Profile for CAH now maintains hundreds of millions of profile records across the profile and six custom schemas. CAH is now getting hundreds of thousands of API lookups per day with an average response time of less than one second for 99 percent of them. And, we are able to achieve a 99.99 percent uptime.

Our implementation of Adobe Experience Platform Customer Profile in CAH has benefited both our IT department and our customer support and sales teams. IT enjoys the benefit of the high-availability API and low-latency response time without having to manage that part of the stack. That low latency also allows our customer support and sales teams to access the data more easily and in real-time significantly accelerating their ability to close customer issues quickly. As a result, Adobe is now able to serve both its internal and external customers more personalized experiences faster than ever before.

Our Customer Zero implementation showed us that an incremental approach is not only possible but often preferable because it allows more flexibility in working with data across a broad range of different enterprise systems. Adobe Experience Platform users don't have to retool entire systems to benefit from XDM capabilities. Rather, they can tailor their use of the platform to fit their existing use cases. For example, even if an enterprise has offline processes to compute, running a high-availability, low-latency API to consume the results is still a lot of work. Adobe Experience Platform allows an incremental approach in which pre-computed attributes can be pushed to the platform to provide the unified profile and real-time access to clients. This is the API at scale.

The Adobe vision for Real-Time Customer Profile

While this implementation has provided Adobe the ability to give its customers a battle-tested platform in which to develop their own unified profiles with their own diverse data sets, we still have work to do.

Currently, our implementation for CAH is still a very batch-oriented solution where all the data are aggregated and assembled in IT systems before being batched and shipped into the Adobe Experience Platform. We are continuing our work to realize our vision for the CAH implementation of Adobe Experience Platform Customer Profile.

While our current implementation is a success, we have been testing aspects of all the different aspects of the streaming, real-time components of Adobe Experience Platform along the way. Our goal is to successfully bring all of these components together to fully realize our ability to deliver personalized customer experiences in real-time.

Figure 4: Adobe’s goals for the future architecture of the Customer Activity Hub’s implementation of Adobe Experience Platform Real-Time Customer Profile.

Once we inventoried the data sources and the client use cases, our IT team used Adobe Experience Platform's XDM system to develop and express the CAH profile data as a set of XDM-based schema. We leveraged the platform's existing XDM schemas, adding a few extended fields as well as custom schemas that reflect Adobe's specific approach to representing customer data.

Adobe is a Customer Zero organization. As a result of Adobe's internal implementation of Real-Time Customer Profile more than a year before its release, the Adobe Experience Platform is now hardened and has matured its capabilities and operational readiness. In the future, more direct data from streaming sources will be centralized as events flow into the Adobe Experience Platform first and are then selectively synchronized with Hadoop as needed.

Adobe Experience Platform Customer Profile is a trusted and governed data layer between source systems and execution endpoints managing customer interaction. At Adobe, we're removing the silos that are in our data lakes today and we're accelerating the data to the least latent state manageable for all of these customer experiences. Adobe Experience Platform can do the same for your enterprise. To learn more, visit us here.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetups.

Originally published: Aug 22, 2019

[#2]

Very insightful information. Thank you @NimashaJain