Using Identity Graph Viewer to Validate and Debug Adobe Experience Platform’s Identity Services
Authors: Shota Ido (@shotaido), Jileesh Sathyan Mandody (@jileeshms), Rich Beaver, and Scott Christofferson (@schristo)
This blog explores the capabilities of Adobe Experience Platform Identity Graph Viewer to visualize, validate, and debug consumer identities constructed by Adobe Experience Platform’s Identity Service.
Identity graphs link multiple user and device identifiers into a single, unified profile of an individual. To address our customer needs, Adobe Experience Platform team built an Identity Graph Viewer to make it easier for data engineers to validate and debug their organization’s identity graphs and explore the underlying data.
Introducing Adobe Identity Graph Viewer
Adobe Experience Platform Identity Service collects the information necessary to assemble these graphs. Our new feature, Identity Graph Viewer, extends these capabilities to provide greater transparency of the underlying process. With this tool you can now more readily answer some key questions:
- Are identities being connected as expected by the underlying algorithms?
- Are there profile collapse issues, when one graph represents two or more individuals due to bad links?
From these questions, we architected our solution. Here is some basic terminology before diving into the details.
- Identity Graph: Data structure that contains all identifiers correlating with an individual consumer, including logins, email addresses, device IDs, cookies, and other digital footprints.
- Identity Service: Part of Adobe Experience Platform that stitches together consumer identities to create identity graphs based on deterministic matching.
An individual identity graph contains three primary components:
- Identity (Node): A unique identity value associated with an individual.
- Link (Edge): A connection between identity values.
- Graph (Cluster): The collection of identities and links that represents an individual.
Note: All identity values shown in this blog post are sample IDs for illustrative purposes.
Identity Graph Viewer gives data engineers and other authorized users the ability to:
- Search: Visualize all of the identities and links related to a given namespace and identity value, via Experience Platform user interface.
- Explore: Examine the data sources used to generate the graph, the related values, and the time when the link was last updated.
- Filter: Narrow results of the graph output based on selected criteria which are useful when reviewing larger graphs.
Technical details
Identity Graph Viewer can be found in the Identities section of Adobe Experience Platform UI, under the Graph Viewer tab.
Note: All identity values shown in this blog post are sample IDs for illustrative purposes.
To build identity graph visualizations we start with the raw graph data, which is retrieved via Identity Graph REST API. A query results in a JSON response with the relevant graph data, containing raw information on the nodes, links, and values associated with the queried identity.
Here is a sample JSON response from Identity Graph API:
{
"version": "2.0",
"graphs": [
{
"composite_xid": {
"nsid": 6,
"id": "RW@yahoo.com"
},
"vertices": [
{
"composite_xid": {
"nsid": 6,
"id": "RW@yahoo.com"
},
"last_updated_ts": 1597879530383
},
{
"composite_xid": {
"nsid": 20914,
"id": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22"
},
"last_updated_ts": 1597879789137
},
{
"composite_xid": {
"nsid": 4,
"id": "48679369652223806281268583030921471748"
},
"last_updated_ts": 1597878709100
},
{
"composite_xid": {
"nsid": 7,
"id": "(147) 555-5555"
},
"last_updated_ts": 1597878709100
},
{
"composite_xid": {
"nsid": 6,
"id": "VH@yahoo.com"
},
"last_updated_ts": 1597879789137
}
],
"edges": [
{
"composite_xid_pair": [
{
"nsid": 6,
"id": "RW@yahoo.com"
},
{
"nsid": 4,
"id": "48679369652223806281268583030921471748"
}
],
"first_established_ts": 1597878709100,
"last_established_ts": 1597878709100,
"dataset_ids": [
"5f3c2a4bc479ad194f065d4d"
],
"batch_ids": [
"01EG4FVDCKFXK550XNXNN6F5MT"
]
},
{
"composite_xid_pair": [
{
"nsid": 6,
"id": "VH@yahoo.com"
},
{
"nsid": 6,
"id": "RW@yahoo.com"
}
],
"first_established_ts": 1597879530383,
"last_established_ts": 1597879530383,
"dataset_ids": [
"5f3c2a4bc479ad194f065d4d"
],
"batch_ids": [
"01EG4GMJKTA0ZYX6MVGJCYQH4Y"
]
},
{
"composite_xid_pair": [
{
"nsid": 7,
"id": "(147) 555-5555"
},
{
"nsid": 6,
"id": "RW@yahoo.com"
}
],
"first_established_ts": 1597878709100,
"last_established_ts": 1597878709100,
"dataset_ids": [
"5f3c2a4bc479ad194f065d4d"
],
"batch_ids": [
"01EG4FVDCKFXK550XNXNN6F5MT"
]
},
{
"composite_xid_pair": [
{
"nsid": 6,
"id": "VH@yahoo.com"
},
{
"nsid": 20914,
"id": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22"
}
],
"first_established_ts": 1597879789137,
"last_established_ts": 1597879789137,
"dataset_ids": [
"5f3c2a4bc479ad194f065d4d"
],
"batch_ids": [
"01EG4GWH058FM77T8MA4YM6MZ4"
]
},
{
"composite_xid_pair": [
{
"nsid": 7,
"id": "(147) 555-5555"
},
{
"nsid": 4,
"id": "48679369652223806281268583030921471748"
}
],
"first_established_ts": 1597878709100,
"last_established_ts": 1597878709100,
"dataset_ids": [
"5f3c2a4bc479ad194f065d4d"
],
"batch_ids": [
"01EG4FVDCKFXK550XNXNN6F5MT"
]
}
]
}
],
"unprocessed_composite_xids": []
}While this response has all of the information necessary to build a graph of the identity relationships, it does not contain more granular data that is helpful for validating and debugging entries. To get this additional information we use GraphQL.
Getting additional details with GraphQL
Making Identity Graph Viewer useful means querying additional APIs, to get information such as the identity namespace, schema definition, and dataset names related to the graph values. We are using GraphQL to orchestrate the various REST API calls associated with these elements and merge them into a single response to the client.
As illustrated in Figure 3, the client makes a single request to obtain the graph data through GraphQL. The client makes a single request to GraphQL. Then GraphQL makes multiple API requests. Coordinating all of the API calls behind a single gateway helps to reduce the total latency of the request, and also enables us to cache responses to reduce duplicate requests. One of the primary caching mechanisms we use is based on the DataLoader concept.
Here is a sample JSON response from GraphQL query:
{
"data": {
"identityGraph": {
"id": "RW@yahoo.com",
"namespace": {
"code": "Email",
"__typename": "IdentityNamespace"
},
"nodes": [
{
"id": "RW@yahoo.com",
"namespace": {
"code": "Email",
"__typename": "IdentityNamespace"
},
"lastUpdated": "2020-08-19T23:25:30.383Z",
"__typename": "IdentityGraphNode"
},
{
"id": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22",
"namespace": {
"code": "GAID",
"__typename": "IdentityNamespace"
},
"lastUpdated": "2020-08-19T23:29:49.137Z",
"__typename": "IdentityGraphNode"
},
{
"id": "48679369652223806281268583030921471748",
"namespace": {
"code": "ECID",
"__typename": "IdentityNamespace"
},
"lastUpdated": "2020-08-19T23:11:49.100Z",
"__typename": "IdentityGraphNode"
},
{
"id": "(147) 555-5555",
"namespace": {
"code": "Phone",
"__typename": "IdentityNamespace"
},
"lastUpdated": "2020-08-19T23:11:49.100Z",
"__typename": "IdentityGraphNode"
},
{
"id": "VH@yahoo.com",
"namespace": {
"code": "Email",
"__typename": "IdentityNamespace"
},
"lastUpdated": "2020-08-19T23:29:49.137Z",
"__typename": "IdentityGraphNode"
}
],
"edges": [
{
"source": "RW@yahoo.com",
"target": "48679369652223806281268583030921471748",
"firstEstablished": "2020-08-19T23:11:49.100Z",
"lastEstablished": "2020-08-19T23:11:49.100Z",
"datasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"xdmFullSchema": {
"id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
"title": "Identity Profile Test Schema for Test",
"isAdhoc": false,
"__typename": "XDMFullSchema"
},
"connectorId": null,
"connection": null,
"__typename": "Dataset"
}
],
"batches": [
{
"id": "01EG4FVDCKFXK550XNXNN6F5MT",
"completed": "2020-08-19T23:11:30.644Z",
"created": "2020-08-19T23:11:19.101Z",
"relatedDatasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"__typename": "Dataset"
}
],
"__typename": "Batch"
}
],
"__typename": "IdentityGraphEdge"
},
{
"source": "VH@yahoo.com",
"target": "RW@yahoo.com",
"firstEstablished": "2020-08-19T23:25:30.383Z",
"lastEstablished": "2020-08-19T23:25:30.383Z",
"datasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"xdmFullSchema": {
"id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
"title": "Identity Profile Test Schema for Test",
"isAdhoc": false,
"__typename": "XDMFullSchema"
},
"connectorId": null,
"connection": null,
"__typename": "Dataset"
}
],
"batches": [
{
"id": "01EG4GMJKTA0ZYX6MVGJCYQH4Y",
"completed": "2020-08-19T23:25:14.237Z",
"created": "2020-08-19T23:25:03.628Z",
"relatedDatasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"__typename": "Dataset"
}
],
"__typename": "Batch"
}
],
"__typename": "IdentityGraphEdge"
},
{
"source": "(147) 555-5555",
"target": "RW@yahoo.com",
"firstEstablished": "2020-08-19T23:11:49.100Z",
"lastEstablished": "2020-08-19T23:11:49.100Z",
"datasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"xdmFullSchema": {
"id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
"title": "Identity Profile Test Schema for Test",
"isAdhoc": false,
"__typename": "XDMFullSchema"
},
"connectorId": null,
"connection": null,
"__typename": "Dataset"
}
],
"batches": [
{
"id": "01EG4FVDCKFXK550XNXNN6F5MT",
"completed": "2020-08-19T23:11:30.644Z",
"created": "2020-08-19T23:11:19.101Z",
"relatedDatasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"__typename": "Dataset"
}
],
"__typename": "Batch"
}
],
"__typename": "IdentityGraphEdge"
},
{
"source": "VH@yahoo.com",
"target": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22",
"firstEstablished": "2020-08-19T23:29:49.137Z",
"lastEstablished": "2020-08-19T23:29:49.137Z",
"datasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"xdmFullSchema": {
"id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
"title": "Identity Profile Test Schema for Test",
"isAdhoc": false,
"__typename": "XDMFullSchema"
},
"connectorId": null,
"connection": null,
"__typename": "Dataset"
}
],
"batches": [
{
"id": "01EG4GWH058FM77T8MA4YM6MZ4",
"completed": "2020-08-19T23:29:32.692Z",
"created": "2020-08-19T23:29:24.031Z",
"relatedDatasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"__typename": "Dataset"
}
],
"__typename": "Batch"
}
],
"__typename": "IdentityGraphEdge"
},
{
"source": "(147) 555-5555",
"target": "48679369652223806281268583030921471748",
"firstEstablished": "2020-08-19T23:11:49.100Z",
"lastEstablished": "2020-08-19T23:11:49.100Z",
"datasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"xdmFullSchema": {
"id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
"title": "Identity Profile Test Schema for Test",
"isAdhoc": false,
"__typename": "XDMFullSchema"
},
"connectorId": null,
"connection": null,
"__typename": "Dataset"
}
],
"batches": [
{
"id": "01EG4FVDCKFXK550XNXNN6F5MT",
"completed": "2020-08-19T23:11:30.644Z",
"created": "2020-08-19T23:11:19.101Z",
"relatedDatasets": [
{
"id": "5f3c2a4bc479ad194f065d4d",
"name": "Identity Profile Test Schema for Test",
"__typename": "Dataset"
}
],
"__typename": "Batch"
}
],
"__typename": "IdentityGraphEdge"
}
],
"__typename": "IdentityGraph"
}
},
"extensions": {}
}Using GraphQL helps us build an object model for the graph and create a useful abstraction between the viewer and all of the APIs involved. In our previous blog, the pros and cons of GraphQL are covered.
IdentityGraphViewer Component
This component acts as a container for both parts of the process: one that renders the graph, and the other that renders the table, as shown in Figure 2. The GraphQL query returns two arrays, with the node and edge details separated. Two methods are defined in this component, called onNodeClick and onEdgeClick. These methods are passed on IdentityGraph to update the graph viewer’s DataTable component when the user clicks on a node or edge on the graph. As a result, the UI shows the details corresponding to the node or the edge clicked on the graph in the table.
Here is a simplified example of IdentityGraphViewer:
import React from 'react';
const IdentityGraphViewer = ({ nodes, edges }) => {
const [selectedNodeId, setSelectedNodeId] = useState();
const [selectedEdgeId, setSelectedEdgeId] = useState();
const handleEdgeClick = edge => {
setSelectedEdgeId(edge.id);
}
const handleNodeClick = node => {
setSelectedNodeId(node?.id);
}
return (
<div>
<IdentityGraph
nodes={nodes}
links={edges}
onNodeClick={handleNodeClick}
onEdgeClick={handleEdgeClick}
/>
<DataTable
rows={nodes}
selectedRows={selectedNodeId ? [selectedNodeId] : []}
>
<DataColumn dataKey="type"/>
<DataColumn dataKey="id" label="values" />
</DataTable>
<DataTable
rows={edges}
selectedRows={selectedEdgeId ? [selectedEdgeId] : []}
>
<DataColumn dataKey="type"/>
<DataColumn dataKey="details" label="details" />
</DataTable>
</div>
)
}IdentityGraph Component
This component receives node details, link details, and methods to call when a node or edge is clicked from IdentityGraphViewer component. Force-directed graphs represent these connections well and we chose D3JS as the best implementation method. D3 is used for rendering the graph and all math around calculating the position of nodes and edges. The render lifecycle method defines a reference to an SVG element and the graph is plotted onto this reference inside the componentDidMount hook.
Here is the render method of IdentityGraph component:
render() {
return (
<div className={styles.container}>
<svg
className={styles.graph}
ref={ref => {
this.svgRef = ref;
}}
width="100%"
height="100%"
>
<g/>
</svg>
</div>
);
}We defined this component as a React.PureComponent so that the graph is updated only when a shallow comparison of props is different from the previous time. This helps us avoid deep comparisons of props from the previous state, which could be very expensive when the data is large. We use d3.forceSimulation to create a new simulation with the nodes, and apply a link force on this simulation to create a force-directed graph.
Here is a sample code for IdentityGraphComponent:
import React from 'react';
const IdentityGraphViewer = ({ nodes, edges }) => {
const [selectedNodeId, setSelectedNodeId] = useState();
const [selectedEdgeId, setSelectedEdgeId] = useState();
const handleEdgeClick = edge => {
setSelectedEdgeId(edge.id);
}
const handleNodeClick = node => {
setSelectedNodeId(node?.id);
}
return (
<div>
<IdentityGraph
nodes={nodes}
links={edges}
onNodeClick={handleNodeClick}
onEdgeClick={handleEdgeClick}
/>
<DataTable
rows={nodes}
selectedRows={selectedNodeId ? [selectedNodeId] : []}
>
<DataColumn dataKey="type"/>
<DataColumn dataKey="id" label="values" />
</DataTable>
<DataTable
rows={edges}
selectedRows={selectedEdgeId ? [selectedEdgeId] : []}
>
<DataColumn dataKey="type"/>
<DataColumn dataKey="details" label="details" />
</DataTable>
</div>
)
}What’s Next
We plan to enhance capabilities on the Identity Graph Viewer in several ways. First, based on customer feedback, we plan to improve the user experience of navigating through this UI. Second, we plan to support visualizing other types of graphs, such as household graphs. Finally, we plan to add functions other than the view, such as deleting and editing graphs.
Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up for future Adobe Experience Platform Meetups.
References
- Adobe Experience Platform
- Adobe Experience Platform Identity Service overview
- Adobe Experience Platform Identity Graph Viewer overview
- Identity Graph REST API
- JSON response from Identity Graph API
- GraphQL
Related Blogs
- Adobe Experience Platform Identity Graph is the Foundation for the Unified Profile
- Adobe Experience Platform’s Identity Service — How to Solve the Customer Identity Conundrum
- GraphQL: Making Sense of Enterprise Microservices for the UI
- Introducing React Spectrum
Originally published: Apr 29, 2021
Latest Articles
Unlocking the Power of Data with 'View SQL' in Dashboards
63
2
0
Views
Likes
Replies
Unlocking the Power of Views for Insightful Dashboards
77
0
0
Views
Likes
Replies
Mastering Consent Analysis in Adobe Real-Time CDP: A Guide to Building Your Customized Insights using Data Distiller
287
1
0
Views
Like
Replies
A Guide to Building Custom Insights Data Models in the Accelerated Store with Data Distiller
199
0
0
Views
Likes
Replies
Introduction to the AEP Edge Network
1.5K
9
2
Views
Likes
Replies
Integrating Power BI with Adobe Real-Time Customer Data Platform: A Game-Changer for Business Intelligence
410
1
0
Views
Like
Replies
Unleashing the Power of Query Service in Adobe Experience Platform for Complex Data Calculations
265
4
0
Views
Likes
Replies
Tuesday's Tech Bytes - Best practices to follow while migrating Adobe Web/Mobile SDK (Part 02)
422
0
1
Views
Likes
Replies
Tuesday's Tech Bytes - Best practices to follow while migrating Adobe Web/Mobile SDK (Part 01)
480
1
0
Views
Like
Replies
Tuesday's Tech Bytes - Tips & tricks in Adobe Web/Mobile SDK Migration (Part 02)
469
1
0
Views
Like
Replies
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.