Analytical Workloads and Real-Time Customer Profile - Two Sides of the Same Brain with a Single-Minded Purpose

[#1]

Authors: Eric Knee and Jody Arthur

Learn how Adobe Experience Platform combines both streaming and batch data into one data workflow that supports both analytical and operational workloads, reducing complexity and eliminating latency to deliver real-time personalized experiences at scale.

Adobe Experience Platform is rapidly transforming the way brands interact with their customers with an open, intelligent, and real-time platform for customer experience. The ability to extensible, and holistic platform capable of combining data across the enterprise to power real-time profiles enhanced through experience intelligence and governance, and activates these profiles to deliver real-time personalized experiences at scale.

This has been very challenging for enterprises. Most maintain their customer data in separate analytical and operational systems (Figure 1). Their data is constantly moving between these systems incurring latency, inaccuracies, and not able to respond to the ever-increasing demands of consumers for better customer experiences.

Figure 1: Overview of how most legacy enterprise systems function to support their data workflows to support both analytical and operational (activation) needs.

Adobe Experience Platform has built architecture with two sides of a single “brain” to solve these problems (Figure 2). Both sides serve very different functions integral to the goal of providing better customer experiences. In Adobe Experience Platform, there is one data workflow that supports both operational and analytical use cases in one system, which increases ROI, reduces complexity, and eliminates latency issues.

Figure 2: Overview of how Adobe Experience Platform’s unique architecture brings both analytical workloads and activation together in one platform to improve ROI, reduce complexity, and eliminate latency.

The left brain is responsible for analytical workloads, bringing together data from Adobe Experience Cloud, Adobe Analytics, and Adobe Audience Manager and combining it with data coming from your domain (e.g. transactional data or operational data for the same consumers). Then in the right brain, what we call the Real-Time Customer Profile brain, we are able to craft a picture of your customer in real-time that can be used both Adobe applications or your own applications via our APIs to deliver better customer experiences in real-time.

The two brains serve very different but important purposes in Adobe Experience Platform. The Analytical side of the brain sits over in the Data Lake and shows all the history of what happens with a profile every time you load a new record into the system. Data Lake allows you to do a lot of useful analytics because you can see your values changing over time. In contrast, the Real-Time Customer Profile over in the right side of the brain doesn’t care about historical data or the patterns that exist in it. Real-time Customer Profile Profile is only concerned with the latest data because that is what we need to action on in the present to support our real-time use cases. For example, in Adobe Analytics, you might see that Profile 1 is increasing in value. So, that might become a net new attribute that we want to compute for all our profiles to change a marketing behavior to take advantage of the customer’s demonstrated propensity to buy more over time.

Building the ability to ingest batch and streaming data into a single platform to power new capabilities

In order to bring the customer profile, activation side of the brain, and the analytical workload side of the brain together within the platform, we first had to solve the problem of how to get very different types of data into the platform. We had to think about streaming data and how to ingest near real-time event-based data, customer attribute-based data, audience membership, as well as the identities associated with those records. With the batch methodology, there were analytical workloads required to get the data into the same system.

Once we were able to ingest these different types of data into the system (Figure 3), we wanted to next look at how we might be able to use the data now in Adobe Experience Platform’s Data Lake and begin to start leveraging tools, such as Query Service through a pSQL, Power BI, or Tableau connection to do some holistic reporting on that data. We also wanted to be able to leverage data science capabilities on top of that data to build and train models that can expose net new attributes that we can add to our profiles to help drive additional personalization use cases out in the web and ecosystem.

Figure 3: Adobe Experience Platform supports the ingestion of both batch and streaming data to provide enhanced capabilities.

Solving these challenges on the analytical side, bringing together the ability to ingest all types of data in a way that allows us to leverage existing tools to enhance its value for important use cases, provided the foundation for new capabilities on the operational side. In the Real-Time Customer Profile, we were now able to do streaming segmentation on the data coming in, qualify those elements and profile fragments in real-time for the audiences and segments they should be a part of. We can now start to leverage some of these segmentation features in batch and begin to batch activate segments.

Both sides of the brain, functioning together to support important use cases

Where you start seeing all of this come together is in the outbound pieces coming out of Adobe Experience Platform both from the Real-Time Customer Profile and the analytical brain, now functioning together to provide all-new capabilities (Figure 4).

Figure 4: Two sides of a single brain — Real-Time Customer Profile and analytical workloads, together within a single platform to deliver all-new capabilities.

With the ability to stream data into the system in real-time, we needed to provide a way for Adobe Experience Platform users to easily extract enriched customer profiles in real-time for activation through their various channels. So, we have built into Adobe Experience Platform the ability for users to stream their real-time customer profile and segment data to any Adobe or non-Adobe applications they may be used for activation.

We also have the Point Lookup (REST) endpoint where you can look up a single profile and view its attributes data and event data. One of our large telco customers recently leveraged this powerful capability to deliver real-time customized email content to its customers the moment they opened the company’s email message. With the lookup capability, the system was able to use the attributes coming back from the platform for each customer to inform the content that should be displayed in the email at time of opening.

The system also gives our users the ability to do batch exports of the data we are bringing in both on the analytical side of the brain as well as the events data we are streaming in on the real-time side of the brain. These batches can be delivered to other systems within Adobe Experience Platform or to external systems to support reporting and modeling needs. For example, an enterprise can use batch data in Adobe Analytics to do Customer Journey Analytics and then export those results to a tool like Power BI for reporting.

With the ability to combine both streaming and batch data into one data workflow, Adobe Experience Platform eliminates the complexity in many enterprise systems that prevent their ability to fully support multiple needs. By bringing these two worlds together, enterprises can actually activate the insights faster for marketing because the latency of “FedExing” the data between various IT infrastructure is now removed. Likewise gathering insights on the marketing actions a customer takes feeds directly back into the analytical brain eliminating the need to FedEx that data back into the system. As a result, profiles are continually being enriched as this data seamlessly flows into the system.

In Adobe Experience Platform, two brains — analytical workloads, such as machine learning and deriving deep insights from the data through business intelligence applications, and operational workloads to support real-time use cases — think as one to allow enterprises to deliver real-time personalized experiences at scale.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up here for future Adobe Experience Platform Meetups.

Resources

1. Adobe Experience Platform — https://www.adobe.com/experience-platform.html
2. Adobe Experience Cloud — https://www.adobe.com/experience-cloud.html
3. Adobe Analytics — https://www.adobe.com/analytics/adobe-analytics.html
4. Adobe Audience Manager — https://www.adobe.com/analytics/audience-manager.html
5. Real-Time Customer Profile — https://www.adobe.com/experience-platform/real-time-customer-profile.html
6. Adobe Experience Platform Data Ingestion Overview — https://docs.adobe.com/content/help/en/experience-platform/ingestion/home.html
7. Adobe Experience Platform APIs — https://www.adobe.io/apis/experienceplatform.html
8. Query Service — https://www.adobe.com/experience-platform/query-service.html
9. pSQL — http://postgresguide.com/utilities/psql.html
10. Microsoft Power BI — https://powerbi.microsoft.com/en-us/
11. Tableau — https://www.tableau.com/
12. Data Science Workspace — https://www.adobe.com/experience-platform/data-science-workspace.html
13. Data Science Workspace Overview — https://docs.adobe.com/content/help/en/experience-platform/data-science-workspace/home.html
14. How to Train and Evaluate a Model in Adobe Experience Platform Data Science Workspace — https://docs.adobe.com/content/help/en/experience-platform/data-science-workspace/models-recipes/tra...
15. Adobe Sensei for Business Intelligence — https://www.adobe.com/sensei.html
16. Real-Time Customer Profile Entities (Profile Access) — https://docs.adobe.com/content/help/en/experience-platform/profile/api/entities.html
17. Customer Journey Analytics — https://www.adobe.com/analytics/customer-journey-analytics.html

Originally published: Jul 23, 2020