Expand my Community achievements bar.

Using Identity Graph Viewer to Validate and Debug Adobe Experience Platform’s Identity Services

Avatar

Administrator

9/25/21

Authors: Shota Ido (@shotaido), Jileesh Sathyan Mandody (@jileeshms), Rich Beaver, and Scott Christofferson (@schristo)

banner image.jpeg

This blog explores the capabilities of Adobe Experience Platform Identity Graph Viewer to visualize, validate, and debug consumer identities constructed by Adobe Experience Platform’s Identity Service.

Identity graphs link multiple user and device identifiers into a single, unified profile of an individual. To address our customer needs, Adobe Experience Platform team built an Identity Graph Viewer to make it easier for data engineers to validate and debug their organization’s identity graphs and explore the underlying data.

Introducing Adobe Identity Graph Viewer

Adobe Experience Platform Identity Service collects the information necessary to assemble these graphs. Our new feature, Identity Graph Viewer, extends these capabilities to provide greater transparency of the underlying process. With this tool you can now more readily answer some key questions:

  • Are identities being connected as expected by the underlying algorithms?
  • Are there profile collapse issues, when one graph represents two or more individuals due to bad links?

From these questions, we architected our solution. Here is some basic terminology before diving into the details.

  • Identity Graph: Data structure that contains all identifiers correlating with an individual consumer, including logins, email addresses, device IDs, cookies, and other digital footprints.
  • Identity Service: Part of Adobe Experience Platform that stitches together consumer identities to create identity graphs based on deterministic matching.

An individual identity graph contains three primary components:

  • Identity (Node): A unique identity value associated with an individual.
  • Link (Edge): A connection between identity values.
  • Graph (Cluster): The collection of identities and links that represents an individual.

Figure 1: Sample identity graph displayed in Identity Graph Viewer, including identities and links.Figure 1: Sample identity graph displayed in Identity Graph Viewer, including identities and links.

Note: All identity values shown in this blog post are sample IDs for illustrative purposes.

Identity Graph Viewer gives data engineers and other authorized users the ability to:

  • Search: Visualize all of the identities and links related to a given namespace and identity value, via Experience Platform user interface.
  • Explore: Examine the data sources used to generate the graph, the related values, and the time when the link was last updated.
  • Filter: Narrow results of the graph output based on selected criteria which are useful when reviewing larger graphs.

Technical details

Identity Graph Viewer can be found in the Identities section of Adobe Experience Platform UI, under the Graph Viewer tab.

Figure 2: Identity Graph display in Adobe Experience Platform UI.Figure 2: Identity Graph display in Adobe Experience Platform UI.

Note: All identity values shown in this blog post are sample IDs for illustrative purposes.

To build identity graph visualizations we start with the raw graph data, which is retrieved via Identity Graph REST API. A query results in a JSON response with the relevant graph data, containing raw information on the nodes, links, and values associated with the queried identity.

Here is a sample JSON response from Identity Graph API:

{
    "version": "2.0",
    "graphs": [
        {
            "composite_xid": {
                "nsid": 6,
                "id": "RW@yahoo.com"
            },
            "vertices": [
                {
                    "composite_xid": {
                        "nsid": 6,
                        "id": "RW@yahoo.com"
                    },
                    "last_updated_ts": 1597879530383
                },
                {
                    "composite_xid": {
                        "nsid": 20914,
                        "id": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22"
                    },
                    "last_updated_ts": 1597879789137
                },
                {
                    "composite_xid": {
                        "nsid": 4,
                        "id": "48679369652223806281268583030921471748"
                    },
                    "last_updated_ts": 1597878709100
                },
                {
                    "composite_xid": {
                        "nsid": 7,
                        "id": "(147) 555-5555"
                    },
                    "last_updated_ts": 1597878709100
                },
                {
                    "composite_xid": {
                        "nsid": 6,
                        "id": "VH@yahoo.com"
                    },
                    "last_updated_ts": 1597879789137
                }
            ],
            "edges": [
                {
                    "composite_xid_pair": [
                        {
                            "nsid": 6,
                            "id": "RW@yahoo.com"
                        },
                        {
                            "nsid": 4,
                            "id": "48679369652223806281268583030921471748"
                        }
                    ],
                    "first_established_ts": 1597878709100,
                    "last_established_ts": 1597878709100,
                    "dataset_ids": [
                        "5f3c2a4bc479ad194f065d4d"
                    ],
                    "batch_ids": [
                        "01EG4FVDCKFXK550XNXNN6F5MT"
                    ]
                },
                {
                    "composite_xid_pair": [
                        {
                            "nsid": 6,
                            "id": "VH@yahoo.com"
                        },
                        {
                            "nsid": 6,
                            "id": "RW@yahoo.com"
                        }
                    ],
                    "first_established_ts": 1597879530383,
                    "last_established_ts": 1597879530383,
                    "dataset_ids": [
                        "5f3c2a4bc479ad194f065d4d"
                    ],
                    "batch_ids": [
                        "01EG4GMJKTA0ZYX6MVGJCYQH4Y"
                    ]
                },
                {
                    "composite_xid_pair": [
                        {
                            "nsid": 7,
                            "id": "(147) 555-5555"
                        },
                        {
                            "nsid": 6,
                            "id": "RW@yahoo.com"
                        }
                    ],
                    "first_established_ts": 1597878709100,
                    "last_established_ts": 1597878709100,
                    "dataset_ids": [
                        "5f3c2a4bc479ad194f065d4d"
                    ],
                    "batch_ids": [
                        "01EG4FVDCKFXK550XNXNN6F5MT"
                    ]
                },
                {
                    "composite_xid_pair": [
                        {
                            "nsid": 6,
                            "id": "VH@yahoo.com"
                        },
                        {
                            "nsid": 20914,
                            "id": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22"
                        }
                    ],
                    "first_established_ts": 1597879789137,
                    "last_established_ts": 1597879789137,
                    "dataset_ids": [
                        "5f3c2a4bc479ad194f065d4d"
                    ],
                    "batch_ids": [
                        "01EG4GWH058FM77T8MA4YM6MZ4"
                    ]
                },
                {
                    "composite_xid_pair": [
                        {
                            "nsid": 7,
                            "id": "(147) 555-5555"
                        },
                        {
                            "nsid": 4,
                            "id": "48679369652223806281268583030921471748"
                        }
                    ],
                    "first_established_ts": 1597878709100,
                    "last_established_ts": 1597878709100,
                    "dataset_ids": [
                        "5f3c2a4bc479ad194f065d4d"
                    ],
                    "batch_ids": [
                        "01EG4FVDCKFXK550XNXNN6F5MT"
                    ]
                }
            ]
        }
    ],
    "unprocessed_composite_xids": []
}

While this response has all of the information necessary to build a graph of the identity relationships, it does not contain more granular data that is helpful for validating and debugging entries. To get this additional information we use GraphQL.

Getting additional details with GraphQL

Making Identity Graph Viewer useful means querying additional APIs, to get information such as the identity namespace, schema definition, and dataset names related to the graph values. We are using GraphQL to orchestrate the various REST API calls associated with these elements and merge them into a single response to the client.

Figure 3: Orchestrating API calls with GraphQLFigure 3: Orchestrating API calls with GraphQL

As illustrated in Figure 3, the client makes a single request to obtain the graph data through GraphQL. The client makes a single request to GraphQL. Then GraphQL makes multiple API requests. Coordinating all of the API calls behind a single gateway helps to reduce the total latency of the request, and also enables us to cache responses to reduce duplicate requests. One of the primary caching mechanisms we use is based on the DataLoader concept.

Here is a sample JSON response from GraphQL query:

{
  "data": {
    "identityGraph": {
      "id": "RW@yahoo.com",
      "namespace": {
        "code": "Email",
        "__typename": "IdentityNamespace"
      },
      "nodes": [
        {
          "id": "RW@yahoo.com",
          "namespace": {
            "code": "Email",
            "__typename": "IdentityNamespace"
          },
          "lastUpdated": "2020-08-19T23:25:30.383Z",
          "__typename": "IdentityGraphNode"
        },
        {
          "id": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22",
          "namespace": {
            "code": "GAID",
            "__typename": "IdentityNamespace"
          },
          "lastUpdated": "2020-08-19T23:29:49.137Z",
          "__typename": "IdentityGraphNode"
        },
        {
          "id": "48679369652223806281268583030921471748",
          "namespace": {
            "code": "ECID",
            "__typename": "IdentityNamespace"
          },
          "lastUpdated": "2020-08-19T23:11:49.100Z",
          "__typename": "IdentityGraphNode"
        },
        {
          "id": "(147) 555-5555",
          "namespace": {
            "code": "Phone",
            "__typename": "IdentityNamespace"
          },
          "lastUpdated": "2020-08-19T23:11:49.100Z",
          "__typename": "IdentityGraphNode"
        },
        {
          "id": "VH@yahoo.com",
          "namespace": {
            "code": "Email",
            "__typename": "IdentityNamespace"
          },
          "lastUpdated": "2020-08-19T23:29:49.137Z",
          "__typename": "IdentityGraphNode"
        }
      ],
      "edges": [
        {
          "source": "RW@yahoo.com",
          "target": "48679369652223806281268583030921471748",
          "firstEstablished": "2020-08-19T23:11:49.100Z",
          "lastEstablished": "2020-08-19T23:11:49.100Z",
          "datasets": [
            {
              "id": "5f3c2a4bc479ad194f065d4d",
              "name": "Identity Profile Test Schema for Test",
              "xdmFullSchema": {
                "id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
                "title": "Identity Profile Test Schema for Test",
                "isAdhoc": false,
                "__typename": "XDMFullSchema"
              },
              "connectorId": null,
              "connection": null,
              "__typename": "Dataset"
            }
          ],
          "batches": [
            {
              "id": "01EG4FVDCKFXK550XNXNN6F5MT",
              "completed": "2020-08-19T23:11:30.644Z",
              "created": "2020-08-19T23:11:19.101Z",
              "relatedDatasets": [
                {
                  "id": "5f3c2a4bc479ad194f065d4d",
                  "name": "Identity Profile Test Schema for Test",
                  "__typename": "Dataset"
                }
              ],
              "__typename": "Batch"
            }
          ],
          "__typename": "IdentityGraphEdge"
        },
        {
          "source": "VH@yahoo.com",
          "target": "RW@yahoo.com",
          "firstEstablished": "2020-08-19T23:25:30.383Z",
          "lastEstablished": "2020-08-19T23:25:30.383Z",
          "datasets": [
            {
              "id": "5f3c2a4bc479ad194f065d4d",
              "name": "Identity Profile Test Schema for Test",
              "xdmFullSchema": {
                "id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
                "title": "Identity Profile Test Schema for Test",
                "isAdhoc": false,
                "__typename": "XDMFullSchema"
              },
              "connectorId": null,
              "connection": null,
              "__typename": "Dataset"
            }
          ],
          "batches": [
            {
              "id": "01EG4GMJKTA0ZYX6MVGJCYQH4Y",
              "completed": "2020-08-19T23:25:14.237Z",
              "created": "2020-08-19T23:25:03.628Z",
              "relatedDatasets": [
                {
                  "id": "5f3c2a4bc479ad194f065d4d",
                  "name": "Identity Profile Test Schema for Test",
                  "__typename": "Dataset"
                }
              ],
              "__typename": "Batch"
            }
          ],
          "__typename": "IdentityGraphEdge"
        },
        {
          "source": "(147) 555-5555",
          "target": "RW@yahoo.com",
          "firstEstablished": "2020-08-19T23:11:49.100Z",
          "lastEstablished": "2020-08-19T23:11:49.100Z",
          "datasets": [
            {
              "id": "5f3c2a4bc479ad194f065d4d",
              "name": "Identity Profile Test Schema for Test",
              "xdmFullSchema": {
                "id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
                "title": "Identity Profile Test Schema for Test",
                "isAdhoc": false,
                "__typename": "XDMFullSchema"
              },
              "connectorId": null,
              "connection": null,
              "__typename": "Dataset"
            }
          ],
          "batches": [
            {
              "id": "01EG4FVDCKFXK550XNXNN6F5MT",
              "completed": "2020-08-19T23:11:30.644Z",
              "created": "2020-08-19T23:11:19.101Z",
              "relatedDatasets": [
                {
                  "id": "5f3c2a4bc479ad194f065d4d",
                  "name": "Identity Profile Test Schema for Test",
                  "__typename": "Dataset"
                }
              ],
              "__typename": "Batch"
            }
          ],
          "__typename": "IdentityGraphEdge"
        },
        {
          "source": "VH@yahoo.com",
          "target": "27aeb5e0-2f34-4632-8ab6-16a2b1818f22",
          "firstEstablished": "2020-08-19T23:29:49.137Z",
          "lastEstablished": "2020-08-19T23:29:49.137Z",
          "datasets": [
            {
              "id": "5f3c2a4bc479ad194f065d4d",
              "name": "Identity Profile Test Schema for Test",
              "xdmFullSchema": {
                "id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
                "title": "Identity Profile Test Schema for Test",
                "isAdhoc": false,
                "__typename": "XDMFullSchema"
              },
              "connectorId": null,
              "connection": null,
              "__typename": "Dataset"
            }
          ],
          "batches": [
            {
              "id": "01EG4GWH058FM77T8MA4YM6MZ4",
              "completed": "2020-08-19T23:29:32.692Z",
              "created": "2020-08-19T23:29:24.031Z",
              "relatedDatasets": [
                {
                  "id": "5f3c2a4bc479ad194f065d4d",
                  "name": "Identity Profile Test Schema for Test",
                  "__typename": "Dataset"
                }
              ],
              "__typename": "Batch"
            }
          ],
          "__typename": "IdentityGraphEdge"
        },
        {
          "source": "(147) 555-5555",
          "target": "48679369652223806281268583030921471748",
          "firstEstablished": "2020-08-19T23:11:49.100Z",
          "lastEstablished": "2020-08-19T23:11:49.100Z",
          "datasets": [
            {
              "id": "5f3c2a4bc479ad194f065d4d",
              "name": "Identity Profile Test Schema for Test",
              "xdmFullSchema": {
                "id": "https://ns.adobe.com/platformuiintenv/schemas/6f49c5f9ef14d7874c95f173d5eac7b84a9d6e3d96ab91be",
                "title": "Identity Profile Test Schema for Test",
                "isAdhoc": false,
                "__typename": "XDMFullSchema"
              },
              "connectorId": null,
              "connection": null,
              "__typename": "Dataset"
            }
          ],
          "batches": [
            {
              "id": "01EG4FVDCKFXK550XNXNN6F5MT",
              "completed": "2020-08-19T23:11:30.644Z",
              "created": "2020-08-19T23:11:19.101Z",
              "relatedDatasets": [
                {
                  "id": "5f3c2a4bc479ad194f065d4d",
                  "name": "Identity Profile Test Schema for Test",
                  "__typename": "Dataset"
                }
              ],
              "__typename": "Batch"
            }
          ],
          "__typename": "IdentityGraphEdge"
        }
      ],
      "__typename": "IdentityGraph"
    }
  },
  "extensions": {}
}

Using GraphQL helps us build an object model for the graph and create a useful abstraction between the viewer and all of the APIs involved. In our previous blog, the pros and cons of GraphQL are covered.

IdentityGraphViewer Component

This component acts as a container for both parts of the process: one that renders the graph, and the other that renders the table, as shown in Figure 2. The GraphQL query returns two arrays, with the node and edge details separated. Two methods are defined in this component, called onNodeClick and onEdgeClick. These methods are passed on IdentityGraph to update the graph viewer’s DataTable component when the user clicks on a node or edge on the graph. As a result, the UI shows the details corresponding to the node or the edge clicked on the graph in the table.

Here is a simplified example of IdentityGraphViewer:

import React from 'react';

const IdentityGraphViewer = ({ nodes, edges }) => {
  const [selectedNodeId, setSelectedNodeId] = useState();
  const [selectedEdgeId, setSelectedEdgeId] = useState();

  const handleEdgeClick = edge => {
    setSelectedEdgeId(edge.id);
  }
  const handleNodeClick = node => {
    setSelectedNodeId(node?.id);

  }
  return (
  <div>
    <IdentityGraph
      nodes={nodes}
      links={edges}
      onNodeClick={handleNodeClick}
      onEdgeClick={handleEdgeClick}
    />
    <DataTable
      rows={nodes}
      selectedRows={selectedNodeId ? [selectedNodeId] : []}
    >
      <DataColumn dataKey="type"/>
      <DataColumn dataKey="id" label="values" />
    </DataTable>
    <DataTable
      rows={edges}
      selectedRows={selectedEdgeId ? [selectedEdgeId] : []}
    >
      <DataColumn dataKey="type"/>
      <DataColumn dataKey="details" label="details" />
    </DataTable>
  </div>
  )
}

IdentityGraph Component

This component receives node details, link details, and methods to call when a node or edge is clicked from IdentityGraphViewer component. Force-directed graphs represent these connections well and we chose D3JS as the best implementation method. D3 is used for rendering the graph and all math around calculating the position of nodes and edges. The render lifecycle method defines a reference to an SVG element and the graph is plotted onto this reference inside the componentDidMount hook.

Here is the render method of IdentityGraph component:

render() {
  return (
    <div className={styles.container}>
      <svg
        className={styles.graph}
        ref={ref => {
          this.svgRef = ref;
        }}
        width="100%"
        height="100%"
      >
        <g/>
      </svg>
    </div>
  );
}

We defined this component as a React.PureComponent so that the graph is updated only when a shallow comparison of props is different from the previous time. This helps us avoid deep comparisons of props from the previous state, which could be very expensive when the data is large. We use d3.forceSimulation to create a new simulation with the nodes, and apply a link force on this simulation to create a force-directed graph.

Here is a sample code for IdentityGraphComponent:

import React from 'react';

const IdentityGraphViewer = ({ nodes, edges }) => {
  const [selectedNodeId, setSelectedNodeId] = useState();
  const [selectedEdgeId, setSelectedEdgeId] = useState();

  const handleEdgeClick = edge => {
    setSelectedEdgeId(edge.id);
  }
  const handleNodeClick = node => {
    setSelectedNodeId(node?.id);

  }
  return (
  <div>
    <IdentityGraph
      nodes={nodes}
      links={edges}
      onNodeClick={handleNodeClick}
      onEdgeClick={handleEdgeClick}
    />
    <DataTable
      rows={nodes}
      selectedRows={selectedNodeId ? [selectedNodeId] : []}
    >
      <DataColumn dataKey="type"/>
      <DataColumn dataKey="id" label="values" />
    </DataTable>
    <DataTable
      rows={edges}
      selectedRows={selectedEdgeId ? [selectedEdgeId] : []}
    >
      <DataColumn dataKey="type"/>
      <DataColumn dataKey="details" label="details" />
    </DataTable>
  </div>
  )
}

What’s Next

We plan to enhance capabilities on the Identity Graph Viewer in several ways. First, based on customer feedback, we plan to improve the user experience of navigating through this UI. Second, we plan to support visualizing other types of graphs, such as household graphs. Finally, we plan to add functions other than the view, such as deleting and editing graphs.

Follow the Adobe Experience Platform Community Blog for more developer stories and resources, and check out Adobe Developers on Twitter for the latest news and developer products. Sign up for future Adobe Experience Platform Meetups.

References

  1. Adobe Experience Platform
  2. Adobe Experience Platform Identity Service overview
  3. Adobe Experience Platform Identity Graph Viewer overview
  4. Identity Graph REST API
  5. JSON response from Identity Graph API
  6. GraphQL

Related Blogs

Originally published: Apr 29, 2021