Server-Side Data Collection: Several options and API

[#1]

By @Jacob-DDdev  & @kopeti 

In the previous part of this 2-part series, we considered some of the reasons for choosing server-side data collection. In this second part, we will explore various APIs available, including the Edge Network Server API, Media Edge API, and Data Ingestion API, along with their benefits, technical aspects, and interesting fun facts. 

Aside from product-specific API, the following API are available for collecting data: 

• Edge Network Server API 

• Media Edge API 

• Data Ingestion API 

Edge Network Server API 

The Edge Network Server API allows data to first hit Adobe’s Edge Network before being routed to an array of different products and services.  

Benefit summary 

• A single request can send data to multiple Adobe products and services
• Can return data used for personalization and marketing 
• Ability to enrich data with Adobe’s lookup tables (e.g. Geo, Network, Device lookups) 
• Manipulate data in friendly interface with some advanced functionality 
• Supports both individual and batch ingestion 
• Supports authentication requirements 

Fun facts 

• Both the Adobe Web SDK and AEP Edge Network mobile SDK extension are API-First tools which use the Edge Network Server API behind the scenes along with additional tooling for automatic data capturing, boilerplate setup, and advanced response handling 
• Collect endpoints are preferred method for capturing web exit links, since there is never a personalization or marketing case where a response is needed on pages as they are being left 
• The collect endpoint can receive higher volumes of data—supporting 6000 requests/s compared to a max of 4000 requests/s from the interact endpoint 

Media Edge API 

Those familiar with Adobe’s Streaming Media Collection may already know that data has traditionally been sent to its own endpoint distinct from other products that may be using the data. Adobe offers custom analytical processing specialized for reporting on media collections. Though media collection is made easier when using an Adobe SDK, Media Edge API may be the best or only option when developing on a platform without a supported SDK or if designing an in-house, cross-platform video player with media collection.  

Benefit summary 

• Enables Adobe Media Collection from cross-platform codebases 
• Enables Adobe Media Collection from platforms not supported by Adobe SDKs 
• Allows metadata to be captured at an individual event-level 
• Allows for some additional datastream tooling such as Data Prep 

Fun facts 

• This is the underlying API used by the different Adobe SDKs for Media Collection 
• Transitioning from traditional media collection, unlike clickstream data, will require a new Edge-specific endpoint to handle XDM and media session requests 

Data Ingestion API (Streaming/Batch Ingestion API) 

Unlike the two API options available above which send data through Adobe’s Edge Network, this API sends data directly to AEP. Given that it bypasses the Edge Network, it is more limited in the products/services it can support but it is worth considering since it can be a valid option when collecting data for this subset of products, such as Adobe’s Real-Time CDP, Customer Journey Analytics, or Journey Optimizer. The main distinction that sets this API apart from the others is its ability to send Individual Profile class data instead of just ExperienceEvent class data. Despite its ability to provide data to multiple AEP products, it will not receive a response that can be used for personalization or marketing purposes—instead, product-specific API may be available to return data. 

For ExperienceEvent class data, it is often less preferable when compared to its Edge Network Server API counterpart since it offers less data routing/processing tooling, but for the same reason, it may be preferable for some to avoid unwanted validation or places for data to be manipulated. When avoiding the data processing of the Edge Network though, the payloads are more rigid in that they can only accept and successfully ingest the data that is already in XDM format completely aligned with both the schema defined in its payload and the schema configured on the dataset for which it is sent. This means it is often best that the system responsible for sending data with this API runs its own validation to minimize maintenance and data loss from failed ingestion. 

For Individual Profile class data, payloads will look very similar, with rigid XDM format, but it will reference a schema that was defined to extend from the Individual Profile class. Unlike some required fields for event data, like timestamp, profile data can be brought in as partial records. However, additional effort is needed to consider which field(s) will be used to capture profile identifiers, such as identityMap or a setting a field as a primary identity. 

Benefit summary 

• Enforces strict XDM format/validation requirements onto source system 
• Reduces ability to re-route/transform data after it is sent 
• Allows the ability to send data using any class of data, including Individual Profile class 
• Adds more automation capabilities to update data immediately compared to other AEP sources which are usually run manually or set on a schedule 

Fun facts 

• Some organizations are running hybrid API setups for AEP where Edge Network Server API is responsible for sending event data and Data Ingestion API is used to capture corresponding profile data 

Additional development considerations 

• Setting up infrastructure to handle API requests/responses 
• Handling volume guardrails leading to delays when ingesting large volumes of data (e.g., queues based on timestamps) 
• Validation may be required for accuracy of setup and/or integration 
• Troubleshooting server-side communications is often harder than monitoring network traffic from client devices 
• Implementing alerts for fluctuations, such as dips or spikes, in downstream Adobe products is advised 
• In the media vertical, exploring the advantages of incorporating Media Edge API alongside Edge Server API for media data collection could be beneficial. 
• Performance limitations should be considered, and it's crucial to comprehend the service constraints (e.g., maximum number of concurrently events, size of the data, and # of services receiving data) 

In summary, server-side data collection offers a compelling alternative for organizations facing challenges like managing client-side data on dynamic web/app pages, meeting strict privacy demands, and addressing evolving browser restrictions. It enhances data accuracy, navigates web/app development changes, reduces client load, and has the potential to provide a smoother user experience. However, organizations must be mindful of performance considerations and service limitations, choosing between authenticated and unauthenticated ingestion. Technical aspects involve infrastructure setup, batch data planning, validation efforts, and the potential benefits of combining Media Edge API with Edge Server API. Overall, server-side data collection is a robust solution for tackling data management, privacy, and user experience challenges, offering a strategic shift from client-side approaches.